Fix v4.15.0 changelog formatting and categorization

Add Step Audio EditX engine and universal inline edit tags

Technical implementation:
- Complete Step Audio EditX TTS and Audio Editor nodes
- Universal inline edit tag post-processing system
- Enhanced Multiline TTS Tag Editor with tabbed interface
- Fix segment grouping and chunk combination parameter bugs

.github/FUNDING.yml

@@ -0,0 +1,15 @@
+# These are supported funding model platforms
+
+github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: diogogo 
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: # Replace with a single Liberapay username
+issuehunt: # Replace with a single IssueHunt username
+lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
+polar: # Replace with a single Polar username
+buy_me_a_coffee: # Replace with a single Buy Me a Coffee username
+thanks_dev: # Replace with a single thanks.dev username
+custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,53 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: "[Bug]"
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**Init log**
+Provide your init log. PLEASE ERASE THE EXEMPLE BELOW, copy paste it yours (you can surround it with triple backticks ```) I NEED to know your system info + TTS Audio Suite version + other init dependency information like below:
+```
+Total VRAM 24564 MB, total RAM 65440 MB
+pytorch version: 2.8.0+cu129
+Set vram state to: NORMAL_VRAM
+Device: cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync
+Using pytorch attention
+Python version: 3.13.6 (tags/v3.13.6:4e66535, Aug  6 2025, 14:36:00) [MSC v.1944 64 bit (AMD64)]
+ComfyUI version: 0.3.57
+ComfyUI frontend version: 1.25.11
+[Prompt Server] web root: J:\stablediffusion1111s2\Data\Packages\ComfyUINEWCLEAN\py313New\ComfyUI_windows_portable\python_embeded\Lib\site-packages\comfyui_frontend_package\static
+✅ Numba JIT working properly - no workarounds needed
+ℹ️ Critical package versions: NumPy 2.2.6, Librosa 0.11.0, Numba 0.61.2, PyTorch 2.8.0+cu129, TorchAudio 2.8.0+cu129, Transformers 4.56.0, SoundFile 0.13.1
+✅ ChatterboxTTS loaded from bundled package
+✅ ChatterboxVC loaded from bundled package
+✅ F5-TTS loaded from system package
+APEX FusedRMSNorm not available, using native implementation
+🎭 Character voices: Found 32 characters, 10 aliases
+======================================================================
+🚀 TTS Audio Suite v4.9.6
+```
+
+**Provide your ERROR full log**
+copy paste it (you can surround it with triple backticks ```):
+```
+error...
+```
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**System information**
+ - OS: [e.g. iOS]
+ - ComfyUI type: [portable; direct venv; conda; desktop app]
+ - Python version: [check the init log]
+
+**Additional context**
+Add any other context about the problem here.
+
+_Note: Bug reports without the error logs, depending on the issue, might be ignored. It's very hard to provide support without minimal information._

.gitignore

@@ -1,3 +1,7 @@
+# Reference ComfyUI folders (not part of this project)
+ComfyUI/
+ComfyUI_frontend/
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]
@@ -137,6 +141,7 @@ venv/
 ENV/
 env.bak/
 venv.bak/
+comfyui_env/
 
 # Spyder project settings
 .spyderproject
@@ -205,6 +210,7 @@ CLAUDE.md
 *CLAUDE.md
 docs/Dev reports/CLAUDE_VERSION_MANAGEMENT_GUIDE.md
 .claude-code.json
+.claude/
 
 IgnoredForGitHubDocs/
 
@@ -218,3 +224,4 @@ IgnoredForGitHubDocs/
 
 .roo/
 .vscode/
+ignore/

PROJECT_INDEX.md

@@ -5,15 +5,19 @@
 ## Architecture Overview
 
 This extension features a **unified modular architecture** supporting multiple TTS engines:
-- **Unified Node Interface**: Single set of nodes (TTS Text, TTS SRT, Voice Changer) that work with any engine
-- **Engine Adapters**: Modular adapters for ChatterBox, F5-TTS, and RVC voice conversion
-- **Centralized Download System**: Unified downloader eliminates HuggingFace cache duplication with direct downloads to organized TTS/ folder structure
+- **Unified Node Interface**: Single set of nodes (TTS Text, TTS SRT, Voice Changer) that work with any engine via clean delegation
+- **ComfyUI Model Management Integration**: All TTS models now integrate with ComfyUI's native model management system, enabling "Clear VRAM" functionality and automatic memory management
+- **Universal Model Loading System**: Standardized model loading interface across all engines (ChatterBox, F5-TTS, Higgs Audio, VibeVoice, Step Audio EditX, RVC, Audio Separation) with fallback support
+- **Engine Processors**: Internal processing engines for each TTS system (ChatterBox, F5-TTS, Higgs Audio, VibeVoice, Step Audio EditX) handling engine-specific orchestration
+- **Engine Adapters**: Modular adapters for ChatterBox, F5-TTS, Higgs Audio 2, VibeVoice, Step Audio EditX, and RVC voice conversion
+- **Inline Edit Tag System**: Universal post-processing system applying Step Audio EditX effects (emotion, style, speed, paralinguistic) to any TTS engine's output via inline tags like `<Laughter>` or `<happy>`
+- **Centralized Download System**: Unified downloader eliminates HuggingFace cache duplication with direct downloads to organized TTS/ folder structure, with full extra_model_paths.yaml support for shared model directories
 - **Thread-Safe Architecture**: Stateless ChatterBox wrapper eliminates shared state corruption (Note: parallel processing is slower than sequential)
 - **Universal Streaming Infrastructure**: Unified streaming system with configurable workers (batch_size parameter) - sequential mode (batch_size=0) recommended for optimal performance
-- **Multilingual Support**: German and Norwegian models for both ChatterBox TTS and Voice Conversion
+- **Multilingual Support**: German and Norwegian models for ChatterBox, plus ChatterBox Official 23-Lang supporting 23 languages including Arabic, Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew, Hindi, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese, Russian, Spanish, Swedish, Swahili, and Turkish
 - **Smart Language Grouping**: SRT processing by language groups to minimize model switching
 - **Character Voice Management**: Centralized character voice system with flexible input types
-- **Comprehensive Audio Processing**: Interactive waveform analyzer, vocal separation, and audio mixing
+- **Comprehensive Audio Processing**: Interactive waveform analyzer, vocal separation, audio mixing, and specialized audio editing
 - **Professional Voice Conversion**: RVC implementation with UVR5 vocal separation and advanced pitch control
 
 ## Documentation Files
@@ -28,14 +32,16 @@ This extension features a **unified modular architecture** supporting multiple T
 
 **docs/CHARACTER_SWITCHING_GUIDE.md** - Guide for multi-character TTS using [CharacterName] tags, voice organization, and character voice management system
 
+**docs/CHATTERBOX_V2_SPECIAL_TOKENS.md** - Complete guide for ChatterBox Official 23-Lang v2 special emotion and sound tokens (<giggle>, <whisper>, <sigh>, etc.) with 30+ expressive tokens for enhanced TTS generation
+
+**docs/INLINE_EDIT_TAGS_USER_GUIDE.md** - Comprehensive guide for inline edit tag system enabling Step Audio EditX post-processing on any TTS engine output. Covers paralinguistic tags (<Laughter>, <Breathing>), emotion tags (<happy>, <sad>), style tags (<whisper>, <serious>), and speed control tags, with iteration control, language support notes, and best practices
+
 **docs/🌊_Audio_Wave_Analyzer-Complete_User_Guide.md** - Complete user guide for Audio Wave Analyzer with interactive waveform visualization and timing extraction
 
 **docs/VERSION_3.1_RELEASE_GUIDE.md** - Release documentation for character switching system and overlapping subtitles support
 
 ### Developer Documentation
 
-**docs/Dev reports/CLAUDE_VERSION_MANAGEMENT_GUIDE.md** - Automated version management using enhanced scripts with changelog generation
-
 **docs/Dev reports/F5TTS_INTEGRATION_SPECIFICATION.md** - Technical specification for F5-TTS integration architecture
 
 **docs/Dev reports/F5TTS_STREAMING_INTEGRATION_PLAN.md** - Universal streaming architecture plan to eliminate spaghetti code and enable F5-TTS streaming
@@ -46,9 +52,9 @@ This extension features a **unified modular architecture** supporting multiple T
 
 **docs/Dev reports/POST_V4.2.3_DEVELOPMENT_REVIEW.md** - Comprehensive analysis of 45 commits implementing universal streaming architecture and performance insights
 
-**docs/PAUSE_TAGS_IMPLEMENTATION_REPORT.md** - Implementation details for pause tag system with timing control syntax
+**docs/Dev reports/PAUSE_TAGS_IMPLEMENTATION_REPORT.md** - Implementation details for pause tag system with timing control syntax
 
-**docs/REFACTORING_PLAN.md** - Architectural refactoring plan and migration strategy documentation
+**docs/Dev reports/REFACTORING_PLAN.md** - Architectural refactoring plan and migration strategy documentation
 
 ## Core Architecture Files
 
@@ -61,7 +67,7 @@ This extension features a **unified modular architecture** supporting multiple T
 ### Engine Implementations
 
 **engines/chatterbox/** - Complete ChatterBox TTS and VC engine implementation
-- **tts.py** - ChatterBox TTS engine with multilingual support and perth watermarking
+- **tts.py** - ChatterBox TTS engine with multilingual support, perth watermarking, and ComfyUI model management integration
 - **vc.py** - ChatterBox Voice Conversion with multilingual model loading
 - **language_models.py** - Language model registry for English, German, and Norwegian models
 - **audio_timing.py** - Audio timing utilities and time-stretching functionality
@@ -71,11 +77,40 @@ This extension features a **unified modular architecture** supporting multiple T
 - **stateless_wrapper.py** - Thread-safe stateless wrapper eliminating shared state corruption in parallel generation
 - **models/** - Complete ChatterBox model architecture (S3Gen, T3, tokenizers, voice encoder)
 
+**engines/chatterbox_official_23lang/** - ChatterBox Official 23-Lang multilingual engine implementation
+- **tts.py** - Main ChatterBox Official 23-Lang TTS engine with 23-language support
+- **vc.py** - Voice conversion engine supporting all 23 languages with refinement passes
+- **language_models.py** - Language model configuration and multilingual support definitions
+- **stateless_wrapper.py** - Thread-safe wrapper for multilingual parallel processing
+- **batch_processor.py**, **adaptive_processor.py**, **gpu_aware_processor.py** - Advanced processing systems
+- **models/** - Complete multilingual model architecture (S3Gen, T3, tokenizers, voice encoder) optimized for 23-language generation
+
 **engines/f5tts/** - F5-TTS engine implementation and editing capabilities
 - **f5tts.py** - Main F5-TTS wrapper with ComfyUI integration
 - **f5tts_edit_engine.py** - Speech editing engine for targeted word replacement
 - **audio_compositing.py** - Audio compositing with crossfade and segment processing
 
+**engines/higgs_audio/** - Higgs Audio 2 engine implementation with voice cloning
+- **higgs_audio.py** - Main Higgs Audio 2 wrapper with voice cloning and chunking support
+- **higgs_audio_downloader.py** - Model auto-download system for generation and tokenizer models
+- **boson_multimodal/** - Complete boson_multimodal implementation from HiggsAudio team
+
+**engines/vibevoice_engine/** - VibeVoice engine implementation with multi-speaker and long-form capabilities
+- **vibevoice_engine.py** - Main VibeVoice wrapper with 90-minute generation and multi-speaker support
+- **vibevoice_downloader.py** - Model auto-download system for Microsoft VibeVoice models (1.5B and 7B)
+
+**engines/step_audio_editx/** - Step Audio EditX engine implementation with audio editing capabilities
+- **step_audio_editx.py** - Main Step Audio EditX wrapper with zero-shot voice cloning and emotion/style/speed editing support
+- **step_audio_editx_downloader.py** - Model auto-download system for Step Audio EditX models
+- **step_audio_editx_impl/** - Complete Step Audio EditX implementation with FunASR integration
+  - **config/** - Edit configuration and prompt definitions for emotion, style, speed, and paralinguistic editing
+  - **funasr_detach/** - Complete FunASR framework implementation for speech processing (models, frontends, tokenizers, metrics)
+  - **stepvocoder/** - CosyVoice2 BigVGAN vocoder implementation for high-quality audio synthesis
+  - **tts.py**, **tts_infer.py** - TTS generation engines with multi-modal processing
+  - **tokenizer.py** - Speech tokenization system
+  - **model_loader.py** - Model loading and management utilities
+  - **quantization/** - AWQ quantization support for VRAM reduction
+
 **engines/rvc/** - RVC (Real-time Voice Conversion) engine implementation
 - **__init__.py** - RVC engine initialization and ComfyUI integration
 - **hubert_downloader.py** - HuBERT model auto-download from Hugging Face with TTS/ folder organization
@@ -99,6 +134,12 @@ This extension features a **unified modular architecture** supporting multiple T
 
 **engines/adapters/f5tts_adapter.py** - F5-TTS engine adapter with parameter mapping and cache integration
 
+**engines/adapters/higgs_audio_adapter.py** - Higgs Audio 2 engine adapter with voice cloning and parameter validation
+
+**engines/adapters/vibevoice_adapter.py** - VibeVoice engine adapter with multi-speaker format conversion and parameter mapping
+
+**engines/adapters/step_audio_editx_adapter.py** - Step Audio EditX engine adapter with voice cloning and audio editing parameter mapping
+
 **engines/adapters/chatterbox_streaming_adapter.py** - ChatterBox streaming adapter bridging existing implementation to universal streaming system
 
 **engines/adapters/f5tts_streaming_adapter.py** - F5-TTS streaming adapter enabling parallel processing with language model switching
@@ -111,13 +152,23 @@ This extension features a **unified modular architecture** supporting multiple T
 
 **nodes/engines/f5tts_engine_node.py** - F5-TTS engine configuration node with model selection and generation settings
 
+**nodes/engines/higgs_audio_engine_node.py** - Higgs Audio 2 engine configuration node with voice cloning parameters and generation settings
+
+**nodes/engines/vibevoice_engine_node.py** - VibeVoice engine configuration node with multi-speaker modes and long-form generation settings
+
 **nodes/engines/rvc_engine_node.py** - RVC voice conversion node with pitch control and quality settings
 
+**nodes/engines/index_tts_engine_node.py** - IndexTTS-2 engine configuration node with character voice selection and generation parameters
+
+**nodes/engines/index_tts_emotion_options_node.py** - 🌈 IndexTTS-2 Emotion Vectors node with interactive radar chart for 8-emotion control
+
+**nodes/engines/step_audio_editx_engine_node.py** - ⚙️ Step Audio EditX Engine configuration node with zero-shot voice cloning, quantization options (int4/int8), and generation parameters for both clone and edit modes
+
 ### Unified Interface Nodes
 
 **nodes/unified/tts_text_node.py** - Universal TTS text generation node working with any configured engine
 
-**nodes/unified/tts_srt_node.py** - Universal SRT subtitle processing with smart language grouping and timing
+**nodes/unified/tts_srt_node.py** - Universal SRT subtitle processing node - clean delegation layer that routes to engine-specific processors
 
 **nodes/unified/voice_changer_node.py** - Universal voice conversion node with multilingual model support and flexible audio inputs
 
@@ -125,19 +176,44 @@ This extension features a **unified modular architecture** supporting multiple T
 
 **nodes/shared/character_voices_node.py** - Character voice management system providing NARRATOR_VOICE outputs for any TTS node
 
+### Text Processing Nodes
+
+**nodes/text/phoneme_text_normalizer_node.py** - 📝 Phoneme Text Normalizer with multilingual text preprocessing for improved TTS pronunciation. Features 4 processing methods: Pass-through, Unicode Decomposition (ą→a̧), IPA Phonemization (espeak backend), and Character Mapping (ASCII fallback). Supports auto-language detection and cross-platform phonemizer support.
+
 ### Engine-Specific Nodes
 
+*Note: These are internal processors/engines used by the Unified nodes, not direct ComfyUI interface nodes. They handle engine-specific orchestration while the Unified nodes provide the user interface.*
+
 **nodes/chatterbox/** - ChatterBox engine implementation nodes (called by Unified nodes)
 - **chatterbox_tts_node.py** - ChatterBox TTS engine node with streaming batch processing and character switching
 - **chatterbox_srt_node.py** - ChatterBox SRT engine node with streaming parallel processing and timing modes  
 - **chatterbox_vc_node.py** - ChatterBox voice conversion engine with iterative refinement
 
+**nodes/chatterbox_official_23lang/** - ChatterBox Official 23-Lang multilingual processors
+- **chatterbox_official_23lang_processor.py** - Main TTS processor supporting all 23 languages with character switching
+- **chatterbox_official_23lang_srt_processor.py** - SRT subtitle processor with multilingual timing and audio assembly
+- **chatterbox_official_23lang_vc_processor.py** - Voice conversion processor supporting all 23 languages with refinement passes
+
 **nodes/f5tts/** - F5-TTS specific nodes
 - **f5tts_node.py** - Direct F5-TTS generation node
 - **f5tts_srt_node.py** - F5-TTS SRT processing with language grouping
 - **f5tts_edit_node.py** - F5-TTS speech editor for word replacement
 - **f5tts_edit_options_node.py** - Advanced F5-TTS editing configuration
 
+**nodes/higgs_audio/** - Higgs Audio 2 internal processors
+- **higgs_audio_srt_processor.py** - Higgs Audio SRT orchestrator with multi-speaker support, character switching, and timing modes (internal processor used by Unified SRT node)
+
+**nodes/vibevoice/** - VibeVoice internal processors
+- **vibevoice_processor.py** - VibeVoice TTS orchestrator with multi-speaker support and long-form generation handling (internal processor used by Unified TTS node)
+
+**nodes/step_audio_editx/** - Step Audio EditX internal processors
+- **step_audio_editx_processor.py** - Step Audio EditX TTS orchestrator with zero-shot voice cloning and emotion/style/speed editing (internal processor used by Unified TTS node)
+- **step_audio_editx_srt_processor.py** - Step Audio EditX SRT processor with character switching and inline edit tag support for subtitle-based TTS workflows
+
+### Step Audio EditX Specialized Nodes
+
+**nodes/step_audio_editx_special/step_audio_editx_audio_editor_node.py** - 🎨 Step Audio EditX - Audio Editor node for post-processing audio with 14 emotions (happy, sad, angry, etc.), 32 styles (whisper, serious, child, etc.), speed control (faster/slower), 10 paralinguistic effects (<Laughter>, <Breathing>, <Sigh>), denoising, and VAD. Supports iterative editing (1-5 passes) and iteration caching for experimentation
+
 ### Audio Processing System
 
 **nodes/audio/analyzer_node.py** - Interactive Audio Wave Analyzer with web interface and timing extraction
@@ -172,26 +248,46 @@ This extension features a **unified modular architecture** supporting multiple T
 
 ### Model Management
 
-**utils/models/manager.py** - Intelligent model discovery and caching with multilingual support for both TTS and VC models
+**utils/models/manager.py** - Intelligent model discovery and caching with multilingual support for both TTS and VC models, integrated with ComfyUI model management and extra_model_paths.yaml for shared storage support
 
 **utils/models/smart_loader.py** - Universal smart model loader preventing duplicate model loading across all engines and modes
 
-**utils/models/f5tts_manager.py** - F5-TTS specific model management extending base manager functionality
+**utils/models/f5tts_manager.py** - F5-TTS specific model management extending base manager functionality, now with ComfyUI integration
 
 **utils/models/language_mapper.py** - Language-to-model mapping system with fallback support
 
 **utils/models/fallback_utils.py** - Model fallback and error recovery utilities
 
+**utils/models/comfyui_model_wrapper/** - Modular ComfyUI model wrapper system with engine-specific handlers (base_wrapper.py, model_manager.py, cache_utils.py, engine_handlers.py) enabling TTS models to integrate with ComfyUI's native model management
+
+**utils/models/unified_model_interface.py** - Universal model loading interface providing standardized factory pattern for all engines (ChatterBox, F5-TTS, Higgs Audio, RVC, Audio Separation) with ComfyUI integration
+
+**utils/models/factory_config.py** - Standardized `ModelLoadConfig` dataclass for consistent factory parameters across all TTS engines
+
+**utils/models/engine_registry.py** - Engine capability registry defining what each engine supports (voice conversion, multilingual switching, corruption recovery, etc.)
+
+**utils/models/fallback_handler.py** - Generic fallback chain utility for graceful model loading degradation (e.g., try local → download → fallback language)
+
+**utils/models/exceptions.py** - Standardized exception hierarchy for all model loading operations (TTSModelLoadingError, TTSModelNotFoundError, TTSDeviceError, etc.)
+
+**utils/models/extra_paths.py** - ComfyUI extra_model_paths.yaml integration system enabling shared model directories across multiple ComfyUI installations with intelligent path resolution and fallback support
+
 ### Audio Processing
 
 **utils/audio/processing.py** - Comprehensive audio utilities for tensor manipulation, normalization, and format conversion
 
+**utils/ffmpeg_utils.py** - Centralized FFmpeg dependency handling with graceful fallbacks for audio conversion and timing analysis
+
 **utils/audio/analysis.py** - Advanced audio analysis including waveform processing, silence detection, and timing extraction
 
 **utils/audio/audio_hash.py** - Centralized content-based hashing for consistent cache keys across all processing modes
 
 **utils/audio/cache.py** - Unified caching system for TTS engines with engine-specific cache management
 
+**utils/audio/chunk_combiner.py** - Modular chunk combination utility with smart auto-selection based on text analysis (sentence boundaries, commas, forced splits) and detailed timing information
+
+**utils/audio/chunk_timing.py** - Chunk combination timing information helper providing standardized timing info integration across all TTS engines
+
 ### Text Processing
 
 **utils/text/character_parser.py** - Universal character switching system with [CharacterName] tag parsing and language-aware processing
@@ -200,9 +296,15 @@ This extension features a **unified modular architecture** supporting multiple T
 
 **utils/text/pause_processor.py** - Pause tag parsing supporting [pause:xx] syntax for precise timing control
 
+**utils/text/segment_parameters.py** - Per-segment parameter system enabling inline control of seed, temperature, cfg, speed, and other engine-specific parameters via pipe-separated syntax like [Alice|seed:42|temperature:0.5]. Supports aliases (cfg_weight→cfg, temp→temperature) and case-insensitive input with universal parameter filtering and validation across all TTS engines
+
+**utils/text/phonemizer_utils.py** - F5-TTS multilingual phonemization system with IPA conversion, cross-platform backend support (espeak-phonemizer-windows/phonemizer), smart language detection, and model-specific exceptions for optimal quality
+
+**utils/text/step_audio_editx_special_tags.py** - Step Audio EditX inline edit tag system with tag conversion, stripping, detection, and UI option generation for paralinguistic effects, emotions, styles, and speed control. Enables universal post-processing of TTS output across all engines
+
 ### Voice Management
 
-**utils/voice/discovery.py** - Advanced voice file discovery with dual folder support and character mapping
+**utils/voice/discovery.py** - Enhanced voice file discovery with multi-path fallback system (models/voices/, models/TTS/voices/, extra_model_paths.yaml directories, voices_examples/), character mapping, and alias loading with priority support
 
 **utils/voice/multilingual_engine.py** - Central orchestrator for multilingual TTS with language switching optimization
 
@@ -212,9 +314,15 @@ This extension features a **unified modular architecture** supporting multiple T
 
 **utils/system/subprocess.py** - Safe subprocess execution with error handling and process isolation
 
+### Compatibility System
+
+**utils/compatibility/transformers_patches.py** - Centralized transformers compatibility patches (monkey patches) managing version compatibility across different transformers library versions including FlashAttentionKwargs, BaseStreamer, DynamicCache properties, and VibeVoice generation method signatures
+
+**utils/compatibility/numba_compat.py** - Centralized Numba/Librosa compatibility system with fast startup testing and intelligent JIT fallback management for Python 3.13+ compatibility
+
 ### Downloads
 
-**utils/downloads/unified_downloader.py** - Centralized download system for all engines eliminating HuggingFace cache duplication with direct HTTP downloads to organized TTS/ structure
+**utils/downloads/unified_downloader.py** - Centralized download system for all engines eliminating HuggingFace cache duplication with direct HTTP downloads to organized TTS/ structure, featuring full extra_model_paths.yaml support for shared model directories
 
 **utils/downloads/model_downloader.py** - RVC-specific model auto-download system with direct downloads (legacy, integrated with unified system)
 
@@ -242,6 +350,12 @@ This extension features a **unified modular architecture** supporting multiple T
 
 ## Web Interface Components
 
+### TTS Tag Editor (Modularized Architecture)
+
+**web/string_multiline_tag_editor.js**, **widget-event-handlers.js**, **widget-ui-builder.js**, **widget-parameter-section.js**, **widget-preset-system.js**, **editor-state.js**, **tag-utilities.js**, **syntax-highlighter.js**, **font-controls.js** - Modular TTS tag editor with event handling, UI sections, parameters, presets, state management, tag utilities, syntax highlighting, and font controls for character/parameter tag editing with undo/redo, formatting, and persistence
+
+### Audio Analysis
+
 **web/audio_analyzer_interface.js** - Main ComfyUI integration and node communication for Audio Wave Analyzer
 
 **web/audio_analyzer_core.js** - Core JavaScript functionality for waveform visualization
@@ -266,8 +380,14 @@ This extension features a **unified modular architecture** supporting multiple T
 
 **web/chatterbox_voice_capture.js** - Voice recording interface for microphone input
 
+**web/index_tts_emotion_radar.js** - ComfyUI integration for 🌈 IndexTTS-2 Emotion Vectors radar chart
+
+**web/emotion_radar_canvas_widget.js** - Interactive radar chart widget with dynamic color blending and click/drag controls
+
 **web/chatterbox_srt_showcontrol.js** - SRT subtitle display and timing controls
 
+**web/step_audio_editx_showcontrol.js** - Step Audio EditX SRT subtitle display and timing controls
+
 **web/audio_analyzer.css** - Styling for audio analyzer interface components
 
 ## Development Tools
@@ -297,6 +417,7 @@ This extension features a **unified modular architecture** supporting multiple T
 **voices_examples/** - Character voice reference files with both audio samples and text descriptions
 - **male/** - Male voice examples with reference audio and text
 - **female/** - Female voice examples with reference audio and text
+- **higgs_audio/** - Higgs Audio 2 voice presets (belinda, en_woman, en_man, chadwick, vex, etc.) with config.json
 - **#character_alias_map.txt** - Character alias mapping configuration
 - Individual character voice files (Clint Eastwood, David Attenborough, Morgan Freeman, Sophie Anderson)
 
@@ -311,10 +432,12 @@ This extension features a **unified modular architecture** supporting multiple T
 The TTS Audio Suite follows a **unified modular architecture** where:
 
 1. **Engine Nodes** (`nodes/engines/`) provide user configuration interfaces
-2. **Unified Nodes** (`nodes/unified/`) offer consistent user experience across all engines  
-3. **Engine Implementations** (`engines/`) handle the actual TTS/VC processing
-4. **Adapters** (`engines/adapters/`) bridge engines to the unified interface
-5. **Utilities** (`utils/`) provide shared functionality across all components
-6. **Web Interface** (`web/`) enables interactive features like audio analysis
+2. **Unified Nodes** (`nodes/unified/`) serve as clean delegation layers routing to appropriate engine processors
+3. **Engine Processors** (`nodes/chatterbox/`, `nodes/f5tts/`, `nodes/higgs_audio/`, `nodes/step_audio_editx/`) handle engine-specific orchestration and workflow logic
+4. **Engine Implementations** (`engines/`) handle the actual TTS/VC processing and model inference
+5. **Adapters** (`engines/adapters/`) bridge low-level engines to higher-level processors
+6. **Inline Edit System** (`utils/text/step_audio_editx_special_tags.py`) provides universal post-processing with Step Audio EditX across all engines
+7. **Utilities** (`utils/`) provide shared functionality across all components
+8. **Web Interface** (`web/`) enables interactive features like audio analysis
 
-This design allows easy addition of new engines while maintaining a consistent user experience and shared optimization features across all TTS engines.
+This layered design ensures consistent user experience while allowing engine-specific optimizations. The unified nodes act as thin delegation layers, eliminating code duplication and maintaining architectural consistency across all TTS engines. The inline edit tag system adds a universal post-processing layer, enabling Step Audio EditX capabilities (emotion, style, speed, paralinguistic effects) to be applied to any TTS engine's output.

__init__.py

@@ -2,7 +2,7 @@
 TTS Audio Suite - Universal multi-engine TTS extension for ComfyUI
 Unified architecture supporting ChatterBox, F5-TTS, and future engines like RVC:
 • 🎤 TTS Text (unified text-to-speech)
-• 📺 TTS SRT (unified SRT subtitle timing)  
+• 📺 TTS SRT (unified SRT subtitle timing)
 • 🔄 Voice Changer (unified voice conversion)
 • ⚙️ Engine nodes (ChatterBox, F5-TTS)
 • 🎭 Character Voices (voice reference management)
@@ -13,6 +13,139 @@ import importlib.util
 import os
 import sys
 
+# CRITICAL: Apply compatibility patches BEFORE any other imports
+#
+# Timing is critical: The monkey-patches must be applied BEFORE libraries are imported
+# by any other module. If we wait until after other imports, they will already
+# be loaded and our monkey-patches won't affect the already-imported module references.
+#
+# PyTorch patches solve TWO PyTorch 2.9 issues:
+# 1. TorchCodec DLL incompatibility on Windows - Global patch uses scipy instead
+# 2. PyTorch 2.9's changed torchaudio.load() returning raw int16 - safe_load_audio() normalizes
+#
+# Transformers patches solve:
+# 1. Step Audio EditX tokenization bug in transformers 4.54+ (audio tokens not recognized)
+# 2. Various model compatibility issues
+try:
+    # Load pytorch_patches directly by file path to avoid package import issues
+    pytorch_patches_path = os.path.join(os.path.dirname(__file__), "utils", "compatibility", "pytorch_patches.py")
+    spec = importlib.util.spec_from_file_location("pytorch_patches_module", pytorch_patches_path)
+    pytorch_patches_module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(pytorch_patches_module)
+
+    # Apply the patches (will only apply on PyTorch 2.9+, silently skip on older versions)
+    pytorch_patches_module.apply_pytorch_patches(verbose=True)
+except Exception as e:
+    print(f"⚠️ Warning: Could not apply PyTorch patches: {e}")
+
+# Apply transformers compatibility patches
+try:
+    # Load transformers_patches directly by file path
+    transformers_patches_path = os.path.join(os.path.dirname(__file__), "utils", "compatibility", "transformers_patches.py")
+    spec = importlib.util.spec_from_file_location("transformers_patches_module", transformers_patches_path)
+    transformers_patches_module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(transformers_patches_module)
+
+    # Apply the patches (will only apply on transformers 4.54+, silently skip on older versions)
+    transformers_patches_module.apply_transformers_patches(verbose=True)
+except Exception as e:
+    print(f"⚠️ Warning: Could not apply Transformers patches: {e}")
+
+# Smart Numba Compatibility System - tests and applies fixes only when needed
+try:
+    from utils.compatibility import setup_numba_compatibility
+    # Apply smart compatibility setup (fast startup test)
+    compatibility_results = setup_numba_compatibility(quick_startup=True, verbose=True)
+except ImportError:
+    # Fallback to simple approach if compatibility module not found
+    import sys
+    import os
+    if sys.version_info >= (3, 13):
+        # Basic test: try librosa.stft and apply workaround if it fails
+        try:
+            import numpy as np
+            import librosa
+            test_audio = np.random.randn(512).astype(np.float32)
+            _ = librosa.stft(test_audio, hop_length=256, n_fft=512)
+            # Only show when there's a problem, not success
+            # Mark that we've tested numba compatibility
+            import sys
+            sys.modules['__main__']._tts_numba_tested = True
+        except Exception as e:
+            if "'function' object has no attribute 'get_call_template'" in str(e):
+                os.environ['NUMBA_DISABLE_JIT'] = '1'
+                os.environ['NUMBA_ENABLE_CUDASIM'] = '1'
+                try:
+                    import numba
+                    numba.config.DISABLE_JIT = True
+                except ImportError:
+                    pass
+                print("🔧 Applied numba JIT workaround for Python 3.13 compatibility")
+            else:
+                print(f"⚠️ Librosa test failed with different error: {e}")
+    else:
+        # Only show warning when JIT is disabled (indicates a problem)
+        pass
+
+# TorchCodec note: Removed torchcodec dependency to eliminate FFmpeg system requirement
+# torchaudio.load() works fine with fallback backends (soundfile, scipy)
+import warnings
+import sys
+import os
+
+# Version disclosure for troubleshooting
+def print_critical_versions():
+    """Print versions of critical packages for troubleshooting"""
+    critical_packages = [
+        ('numpy', 'NumPy'),
+        ('librosa', 'Librosa'),
+        ('numba', 'Numba'),
+        ('torch', 'PyTorch'),
+        ('torchaudio', 'TorchAudio'),
+        ('transformers', 'Transformers'),
+        ('accelerate', 'Accelerate'),
+        ('soundfile', 'SoundFile'),
+    ]
+
+    version_info = []
+    for pkg_name, display_name in critical_packages:
+        try:
+            module = __import__(pkg_name)
+            version = getattr(module, '__version__', 'unknown')
+            version_info.append(f"{display_name} {version}")
+        except ImportError:
+            version_info.append(f"{display_name} not installed")
+
+    print(f"ℹ️ Critical package versions: {', '.join(version_info)}")
+
+def check_ffmpeg_availability():
+    """Check ffmpeg availability and log status"""
+    try:
+        from utils.ffmpeg_utils import FFmpegUtils
+        if FFmpegUtils.is_available():
+            # Only show when unavailable (problem)
+            pass
+        else:
+            print("⚠️ FFmpeg not found - using fallback audio processing (reduced quality)")
+            print("💡 Install FFmpeg for optimal performance: https://ffmpeg.org/download.html")
+    except ImportError:
+        # Fallback check if utils not available yet
+        try:
+            import subprocess
+            result = subprocess.run(['ffmpeg', '-version'], capture_output=True, timeout=5)
+            if result.returncode == 0:
+                # Only show when unavailable (problem)
+                pass
+            else:
+                print("⚠️ FFmpeg not found - using fallback audio processing (reduced quality)")
+        except Exception:
+            print("⚠️ FFmpeg not found - using fallback audio processing (reduced quality)")
+            print("💡 Install FFmpeg for optimal performance: https://ffmpeg.org/download.html")
+
+# Print versions and check dependencies immediately for troubleshooting
+print_critical_versions()
+check_ffmpeg_availability()
+
 # Check for old ChatterBox extension conflict
 def check_old_extension_conflict():
     """Check if the old ComfyUI_ChatterBox_SRT_Voice extension is installed"""
@@ -78,7 +211,68 @@ __description__ = "Universal multi-engine TTS extension for ComfyUI with unified
 
 __all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
 
-# Define web directory for JavaScript files
+# Define web directory for JavaScript files (settings UI)
 WEB_DIRECTORY = "./web"
 
+# Register API endpoint for widget data
+def setup_api_routes():
+    """Setup API routes for widget communication"""
+    try:
+        import json
+        from server import PromptServer
+        from aiohttp import web
+
+        @PromptServer.instance.routes.get("/api/tts-audio-suite/available-characters")
+        async def get_available_characters_endpoint(request):
+            """API endpoint to get available TTS character voices including aliases"""
+            try:
+                from utils.voice.discovery import get_available_characters, voice_discovery
+                characters = list(get_available_characters())
+                # Also get character aliases
+                aliases = list(voice_discovery._character_aliases.keys()) if hasattr(voice_discovery, '_character_aliases') else []
+                # Combine and deduplicate
+                all_chars = sorted(set(characters + aliases))
+                return web.json_response({"characters": all_chars})
+            except Exception as e:
+                print(f"⚠️ Error retrieving available characters: {e}")
+                return web.json_response({"characters": [], "error": str(e)})
+
+        @PromptServer.instance.routes.get("/api/tts-audio-suite/available-languages")
+        async def get_available_languages_endpoint(request):
+            """API endpoint to get available language codes from the canonical language mapper"""
+            try:
+                from utils.models.language_mapper import LANGUAGE_ALIASES
+                # Get all unique canonical language codes (the values in LANGUAGE_ALIASES)
+                languages = sorted(set(LANGUAGE_ALIASES.values()))
+                return web.json_response({"languages": languages})
+            except Exception as e:
+                print(f"⚠️ Error retrieving available languages: {e}")
+                # Fallback list
+                return web.json_response({"languages": ["en", "de", "fr", "ja", "es", "it", "pt", "th", "no"], "error": str(e)})
+
+        @PromptServer.instance.routes.post("/api/tts-audio-suite/settings")
+        async def set_inline_tag_settings_endpoint(request):
+            """API endpoint to receive settings from frontend for inline edit tags"""
+            print("🔧 Settings endpoint called")  # Immediate print to verify endpoint is reached
+            try:
+                data = await request.json()
+                precision = data.get("precision", "auto")
+                device = data.get("device", "auto")
+
+                print(f"🔧 Received settings: precision={precision}, device={device}")
+
+                # Store in global settings that edit_post_processor can access
+                from utils.audio.edit_post_processor import set_inline_tag_settings
+                set_inline_tag_settings(precision=precision, device=device)
+
+                return web.json_response({"status": "success", "precision": precision, "device": device})
+            except Exception as e:
+                print(f"⚠️ Error setting inline tag settings: {e}")
+                return web.json_response({"status": "error", "error": str(e)})
+    except Exception as e:
+        print(f"⚠️ Could not setup API routes: {e}")
+
+# Setup API routes when extension loads
+setup_api_routes()
+
 # nodes.py already handles all the startup output and status reporting

docs/ADVANCED_TTS_TAG_EDITOR_NODE.md

@@ -0,0 +1,418 @@
+# 🎨 Advanced TTS Tag Editor Node
+
+## Overview
+
+Advanced multiline string editor node that extends ComfyUI's standard multiline widget with a sidebar containing context-aware controls for TTS-specific tags (character switching, parameters, pauses). The node features real-time tag generation, preset management, and intelligent syntax support based on the CHARACTER_SWITCHING_GUIDE and PARAMETER_SWITCHING_GUIDE.
+
+---
+
+## Core Features
+
+### 1. Editor Interface
+
+- **Textarea**: Standard multiline editor (like ComfyUI's native string node)
+- **Left Sidebar**: Vertical control panel with buttons and dropdowns
+- **Smart Selection Detection**: Identify if selection is `[Character]text` or plain text
+- **Caret Position Tracking**: Insert tags at cursor position or wrap selection
+- **Mid-Sentence Tag Insertion**: Tags can be inserted anywhere in text, not just at the beginning (e.g., `"text text [alice|seed:2] more text"`)
+- **Undo/Redo Stack**: Full history of all edits with visual indicators, accessible via buttons and keyboard shortcuts (Ctrl+Z / Ctrl+Shift+Z)
+
+### 2. Sidebar Controls - Basic
+
+#### Character Management
+- Dropdown selector for available characters (auto-discovered from `voices_examples/`)
+- Text input to specify custom character name
+- Button: "Add Character" → wraps selection or inserts at caret as `[CharacterName]text`
+- Shows current available characters with icons/indicators
+
+#### Parameter Buttons
+(appear if character is selected)
+
+- `[Seed]` - Opens number input (0-4294967295) → adds `|seed:X`
+- `[Temp]` - Opens slider (0.1-2.0) → adds `|temperature:X`
+- `[CFG]` - Opens slider (engine-specific range) → adds `|cfg:X`
+- `[Speed]` - Slider (0.5-2.0, F5-TTS only) → adds `|speed:X`
+- `[Steps]` - Spinner (1-100, VibeVoice/IndexTTS) → adds `|inference_steps:X`
+- `[Top-P]` / `[Top-K]` - For advanced engines
+
+#### Pause Control
+- Time input field with duration selector (milliseconds/seconds)
+- Button: "Insert Pause" → adds `[pause:XXXms]` or `[pause:XXs]` at caret
+- Quick presets: "0.5s", "1s", "2s" buttons
+
+#### Language Control
+- Dropdown for supported languages (en, de, fr, ja, es, it, pt, th, no)
+- Works with character dropdown: `[language:Character]` format
+- Shows which models are available/loaded
+
+#### Undo/Redo Controls
+- **Undo Button** (`[↶]`) - Revert last change
+- **Redo Button** (`[↷]`) - Restore undone change
+- **History Indicator** - Shows current position in edit history (e.g., "Step 5/10")
+- Keyboard shortcuts: `Ctrl+Z` (Undo), `Ctrl+Shift+Z` (Redo)
+- Full persistence: Undo/redo history survives session reload
+
+### 3. Preset System
+
+#### Preset Management UI
+- Three preset slots (Preset 1, Preset 2, Preset 3)
+- Each slot has:
+  - Save button → captures configured tags
+  - Load button → applies preset to selection/caret
+  - Delete button → clears saved preset
+  - Label field → custom name for the preset
+
+#### Preset Content
+- Stores: character, language, and all parameters
+- Example: "Alice Calm" = `[de:Alice|seed:42|temp:0.3|cfg:0.7]`
+- Can be applied as wrapper around selection or at caret
+
+#### Preset Persistence
+- Store in ComfyUI widget state (node.serialize/deserialize)
+- Also optionally save to local config file for sharing
+- **Full Session Persistence**: All presets, settings, and metadata survive ComfyUI session reload
+
+### 4. Advanced Features
+
+#### Syntax Validation
+- Real-time visual indicators for tag syntax errors
+- Highlight brackets, detect mismatched `[` `]`
+- Warn on invalid parameter values (out of range)
+- Suggest corrections (e.g., `temp` → `temperature`)
+
+#### Multi-Selection Support
+- Apply same tag to multiple selected segments
+- Split by paragraph breaks or sentence punctuation
+- Batch apply parameters across multiple `[Character]text` blocks
+
+#### Tag Inspector
+- Show existing tags in selection/current line
+- Checkbox UI to toggle tags on/off temporarily
+- Quick-edit dialog for existing tag values
+
+#### Auto-Formatting
+- Button: "Auto-Format Tags" → organize tags consistently
+- Validates and cleans malformed tags
+- Groups parameters in standard order: `[char|seed|temp|cfg|speed]`
+
+#### Visual Feedback
+- Color-code different tag types:
+  - Character tags: **Blue** `[Character]`
+  - Parameters: **Green** `|seed:value`
+  - Pauses: **Orange** `[pause:1s]`
+  - Language tags: **Purple** `[de:Character]`
+- Hover tooltips explaining what each tag does
+
+---
+
+## SRT Support & Formatting
+
+### SRT-Aware Mode Detection
+
+The editor **automatically detects SRT format** and enables specialized visualization:
+
+```srt
+1
+00:00:01,000 --> 00:00:04,000
+[Alice] Hello! I'm Alice speaking with precise timing.
+
+2
+00:00:04,500 --> 00:00:09,500
+[Bob|seed:42|temperature:0.5] And I'm Bob with parameters!
+
+3
+00:00:10,000 --> 00:00:14,000
+[de:Narrator] Zurück zum Erzähler in Deutsch.
+```
+
+### SRT Color Coding
+
+When SRT format is detected, the editor applies specialized syntax highlighting:
+
+- **SRT Index**: `1`, `2`, `3` → **Gray** background (disabled/informational)
+- **Timestamp**: `00:00:01,000 --> 00:00:04,000` → **Cyan** (timing information)
+- **Text Content**: Plain text → Black
+  - **Character Tags**: `[Character]` → **Blue**
+  - **Language Tags**: `[de:Character]` → **Purple**
+  - **Parameters**: `|seed:42|temp:0.5` → **Green**
+  - **Pauses**: `[pause:1s]` → **Orange**
+- **Blank Lines**: Between entries → Gray
+
+### SRT-Specific Sidebar Options
+
+When editing SRT content, the sidebar includes:
+
+#### Subtitle Entry Navigation
+- Previous/Next buttons to jump between SRT entries
+- Current entry indicator: "Entry 2 of 5"
+- Quick jump dropdown to select specific subtitle
+
+#### Timing Tools
+- **Insert Timing**: Auto-generates timestamps based on duration
+- **Adjust Timing**: Shift all subsequent subtitles by offset
+- **Validate Timing**: Check for overlaps and gaps
+
+#### Per-Entry Parameter Application
+- Apply parameters to **current entry only**
+- Apply parameters to **selected entries**
+- Apply parameters to **all entries**
+- Clear all tags from current entry
+
+#### SRT Validation
+- Check syntax: proper index/timestamp/text format
+- Warn on timing overlaps
+- Suggest fixes for malformed entries
+- Preview final SRT before saving
+
+### SRT Tag Application Examples
+
+#### Applying Character to Current Entry
+```
+Select entry 2
+Click "Add Character" → [Alice]
+Result:
+2
+00:00:04,500 --> 00:00:09,500
+[Alice] And I'm Bob with parameters!
+```
+
+#### Adding Parameters to Selected Entries
+```
+Select entries 1-3
+Click [Seed] → 42
+Result:
+1
+00:00:01,000 --> 00:00:04,000
+[Alice|seed:42] Hello! I'm Alice speaking...
+
+2
+00:00:04,500 --> 00:00:09,500
+[Alice|seed:42] And I'm Bob with parameters!
+
+3
+00:00:10,000 --> 00:00:14,000
+[Alice|seed:42] Zurück zum Erzähler in Deutsch.
+```
+
+---
+
+## UI Layout
+
+```
+┌─ Advanced TTS Editor ─────────────────────────────┐
+│ ┌───────────────┬─────────────────────────────────┐│
+│ │   SIDEBAR     │      TEXTAREA                   ││
+│ │               │      (multiline editor)         ││
+│ │ ╔═══════════╗ │  1                              ││
+│ │ ║ Character ║ │  00:00:01,000 --> 00:00:04,000 ││
+│ │ ║ [Dropdown]║ │  [Alice] Hello there!          ││
+│ │ ║[+ Add]    ║ │                                 ││
+│ │ ╚═══════════╝ │  2                              ││
+│ │               │  00:00:04,500 --> 00:00:09,500 ││
+│ │ ╔═══════════╗ │  [Bob|seed:42] Nice to meet     ││
+│ │ ║Parameters ║ │                                 ││
+│ │ ║[Seed  ▼] ║ │  3                              ││
+│ │ ║[Temp  -/-]║ │  00:00:10,000 --> 00:00:14,000 ││
+│ │ ║[CFG   -/-]║ │  [pause:1s]                     ││
+│ │ ║[Speed -/-]║ │                                 ││
+│ │ ╚═══════════╝ │                                 ││
+│ │               │                                 ││
+│ │ ╔═══════════╗ │                                 ││
+│ │ ║  Pause    ║ │                                 ││
+│ │ ║[  1.0  ▼] │ │                                 ││
+│ │ ║[Insert]  ║ │                                 ││
+│ │ ╚═══════════╝ │                                 ││
+│ │               │                                 ││
+│ │ ╔═══════════╗ │                                 ││
+│ │ ║ Presets   ║ │                                 ││
+│ │ ║P1[Save]   ║ │                                 ││
+│ │ ║   [Load]  ║ │                                 ││
+│ │ ║   [Del ]  ║ │                                 ││
+│ │ ║P2[Save]   ║ │                                 ││
+│ │ ║   [Load]  ║ │                                 ││
+│ │ ║   [Del ]  ║ │                                 ││
+│ │ ║P3[Save]   ║ │                                 ││
+│ │ ║   [Load]  ║ │                                 ││
+│ │ ║   [Del ]  ║ │                                 ││
+│ │ ╚═══════════╝ │                                 ││
+│ │               │                                 ││
+│ │ ╔═══════════╗ │                                 ││
+│ │ ║ History   ║ │                                 ││
+│ │ ║[↶][↷] 5/10║ │                                 ││
+│ │ ╚═══════════╝ │                                 ││
+│ │               │                                 ││
+│ │ [Format]      │                                 ││
+│ │ [Validate]    │                                 ││
+│ └───────────────┴─────────────────────────────────┘│
+│ Status: SRT Mode | Entry 2/5                     │
+└────────────────────────────────────────────────────┘
+```
+
+---
+
+## Session Persistence & State Management
+
+**Complete State Persistence**:
+- **Text Content**: Full edit history with all undo/redo states
+- **Undo/Redo Stack**: Complete history serialized and restored on session reload
+- **Preset Data**: All custom presets and their labels
+- **Sidebar Settings**: Last used character, language, parameters, pause duration
+- **UI State**: Sidebar panel expanded/collapsed state, last active tab
+- **Cursor Position**: Last known caret position restored on reload
+- **SRT Mode Settings**: Timing configuration, entry navigation state
+
+**Storage Mechanism**:
+- Serialize all state to ComfyUI node widget state
+- Automatic backup to browser localStorage for redundancy
+- JSON-based format for easy inspection and manual recovery
+- No data loss on browser refresh, ComfyUI restart, or system reboot
+
+---
+
+## Technical Implementation
+
+### Backend (Python Node)
+
+**File**: `nodes/text/tts_tag_editor_node.py`
+
+- Extend standard ComfyUI node pattern
+- Discover available characters/voices from `utils/voice/discovery.py`
+- Parse existing tags using `utils/text/character_parser.py` and `utils/text/segment_parameters.py`
+- Validate parameters against engine specs
+- **Full State Serialization**: Handle undo/redo stack, presets, and all sidebar settings in node state
+- Detect and validate SRT format
+- **History Persistence**: Serialize complete edit history to widget state for session recovery
+
+### Frontend (TypeScript/Vue)
+
+**Files**:
+- `web/tts_tag_editor.js` - Core editor logic
+- `web/tts_tag_editor_interface.js` - ComfyUI integration
+- `web/tts_tag_editor_ui.js` - Sidebar components
+- `web/tts_tag_editor.css` - Styling and color schemes
+- `web/tts_tag_editor_srt.js` - SRT-specific formatting and validation
+
+**Features**:
+- Extend standard multiline widget with sidebar panel
+- Implement event handlers for all buttons/inputs
+- Add CSS for visual tag highlighting and styling
+- **Undo/Redo Implementation**: Maintain immutable history stack with efficient state snapshots
+- **Keyboard Shortcuts**: Ctrl+Z (Undo), Ctrl+Shift+Z (Redo), and other common editor shortcuts
+- SRT format detection and specialized rendering
+- Syntax highlighting for SRT entries
+- **Persistent State Manager**: Auto-save to localStorage and node widget state
+
+### Key Components
+
+1. **Character Discovery**: Hook into existing voice system
+2. **Parameter Validation**: Reuse existing parameter filtering logic
+3. **Tag Parsing**: Leverage existing character parser and parameter parser (supports mid-sentence tags)
+4. **Preset Storage**: Use node widget serialization with full persistence
+5. **UI State Management**: Track cursor position, selection, active preset, all sidebar settings
+6. **Undo/Redo Manager**: Immutable history stack with efficient snapshots, keyboard support
+7. **History Persistence**: Serialize complete undo/redo stack to localStorage and widget state
+8. **SRT Detection & Validation**: Detect SRT format, validate structure, highlight syntax
+9. **SRT Navigation**: Jump between entries, manage per-entry operations
+10. **State Recovery**: Auto-restore all editor state on session reload (text, history, presets, settings)
+
+### Integration Points
+
+- `utils/voice/discovery.py` - Character discovery
+- `utils/text/character_parser.py` - Tag parsing
+- `utils/text/segment_parameters.py` - Parameter validation
+- `utils/timing/parser.py` - SRT format parsing and validation
+- Existing language models for language support
+
+---
+
+## Features to Consider Later (v2.0)
+
+- Syntax highlighting with language server protocol (LSP)
+- Auto-complete for character names and parameters
+- Export/import presets as JSON
+- Macro system for complex tag sequences
+- Real-time validation with detailed error messages
+- Voice preview playback directly from tags
+- Template system for common scenes/dialogues
+- Regex tag replacement for bulk editing
+- Keyboard shortcuts for common operations
+- SRT subtitle duration calculator
+- Automatic SRT timing generation from text audio duration
+- Batch character/parameter application across multiple SRT files
+
+---
+
+## Files to Create
+
+1. `nodes/text/tts_tag_editor_node.py` - Backend node
+2. `web/tts_tag_editor.js` - Main editor implementation
+3. `web/tts_tag_editor_ui.js` - Sidebar UI components
+4. `web/tts_tag_editor_srt.js` - SRT-specific features
+5. `web/tts_tag_editor.css` - Styling
+
+---
+
+## Usage Examples
+
+### Basic Text Editing with Mid-Sentence Tags
+
+```
+Once upon a time, [Narrator] there was a beautiful princess.
+[Alice] She lived in a [grand|seed:42] castle.
+[Bob] Who [de:Alice] spoke many languages.
+Back to [Bob|temperature:0.7] normal text.
+```
+
+### Multi-Parameter Example
+
+```
+[Alice|seed:42|temperature:0.3|cfg:0.7] She spoke softly.
+[Bob|seed:100|temperature:0.8|cfg:1.0] He shouted loudly!
+[de:Merchant|seed:55] Guten Tag, meine Damen!
+```
+
+### SRT with Multilingual Characters
+
+```
+1
+00:00:01,000 --> 00:00:04,000
+[Narrator] Welcome to our international story.
+
+2
+00:00:04,500 --> 00:00:09,500
+[de:Alice|temperature:0.5] Hallo! Ich bin Alice.
+
+3
+00:00:09,600 --> 00:00:14,000
+[fr:Bob|temperature:0.6] Bonjour! Je suis Bob.
+
+4
+00:00:14,100 --> 00:00:18,000
+[Narrator] And they lived happily ever after.
+```
+
+### Preset Application Example
+
+Saved Preset: "Action Scene" = `[|seed:random|temperature:0.8|cfg:1.2]`
+
+Apply to selection:
+```
+Before:
+[Hero] I will save you!
+[Villain] Never!
+[Hero] This ends now!
+
+After:
+[Hero|seed:42|temperature:0.8|cfg:1.2] I will save you!
+[Villain|seed:100|temperature:0.8|cfg:1.2] Never!
+[Hero|seed:55|temperature:0.8|cfg:1.2] This ends now!
+```
+
+---
+
+## Related Documentation
+
+- [Character Switching Guide](CHARACTER_SWITCHING_GUIDE.md) - Multi-character and language switching
+- [Parameter Switching Guide](PARAMETER_SWITCHING_GUIDE.md) - Per-segment parameter control
+- [ChatterBox Special Tokens Guide](CHATTERBOX_V2_SPECIAL_TOKENS.md) - Emotion and sound tokens
+- README.md - Complete feature overview

docs/BUMP_SCRIPT_INSTRUCTIONS.md

@@ -1,41 +1,123 @@
 # Version Bump Instructions for ComfyUI ChatterBox Voice
 
+**⚠️ ALWAYS specify which component/engine is affected in changelog (e.g., "Fix Multiline TTS Tag Editor text persistence" not just "Fix text persistence")**
+
 ## Quick Reference for Future Version Bumps
 
+### ⚠️ DO NOT CHECK CURRENT VERSION - The script auto-increments!
+
+**Simply run with `patch`, `minor`, or `major` - NO need to look up current version**
+
 ### Recommended Command (Separate Commit & Changelog)
+
+**⚠️ IMPORTANT: Use positional arguments, NOT --commit/--changelog flags**
+
 ```bash
-python3 scripts/bump_version_enhanced.py <version> --commit "<commit_desc>" --changelog "<changelog_desc>"
+# EASIEST: Just use 'patch' - script auto-increments the version
+python3 scripts/bump_version_enhanced.py patch "<commit_desc>" "<changelog_desc>"
+
+# OR: Specify exact version if needed
+python3 scripts/bump_version_enhanced.py <version> "<commit_desc>" "<changelog_desc>"
 ```
 
 ### Examples
+
+#### Multiline Format (Recommended Standard)
+
 ```bash
-# Patch release (bug fixes)
-python3 scripts/bump_version_enhanced.py 3.2.9 \
-  --commit "Fix character alias resolution and F5-TTS import issues" \
-  --changelog "Bug fixes and stability improvements"
+# Patch release (bug fixes) - CORRECT FORMAT
+python3 scripts/bump_version_enhanced.py 3.2.9 "Fix character alias resolution
 
-# Minor release (new features)  
-python3 scripts/bump_version_enhanced.py 3.3.0 \
-  --commit "Add character support system for F5-TTS generation" \
-  --changelog "Add character support for F5-TTS generation"
+Technical details:
+- Fix parser bypassing character tags in single mode
+- Improve character name validation logic  
+- Add fallback handling for unrecognized tags" "Fix character name handling issues
 
-# Major release (breaking changes)
-python3 scripts/bump_version_enhanced.py 4.0.0 \
-  --commit "Complete architecture refactoring with new modular structure" \
-  --changelog "Project restructure for better maintainability"
+- Fix character tags not being removed from TTS output
+- Improve character name recognition accuracy
+- Better error handling for invalid character names"
+
+# Minor release (new features) - CORRECT FORMAT  
+python3 scripts/bump_version_enhanced.py 3.3.0 "Add Higgs Audio 2 TTS engine
+
+Implementation details:
+- Integrate boson_multimodal voice cloning system
+- Add unified adapter for consistent interface
+- Implement voice preset management" "Add Higgs Audio 2 TTS engine with voice cloning
+
+- New realistic voice synthesis engine
+- Voice cloning from short audio samples
+- Multiple built-in voice presets available"
+
+# Major release (breaking changes) - CORRECT FORMAT
+python3 scripts/bump_version_enhanced.py 4.0.0 "Complete unified architecture implementation
+
+Breaking changes:
+- Migrate all nodes to unified interface pattern
+- Consolidate engine adapters and processors
+- Remove deprecated standalone variants" "Major architecture upgrade to unified system
+
+- All TTS engines now use consistent interface
+- Better performance and memory management  
+- Simplified workflows with consolidated nodes
+- Breaking: old standalone nodes removed"
 ```
 
+#### Auto-Increment Examples (Recommended)
+```bash
+# Auto-increment patch version (4.5.25 → 4.5.26) - CORRECT FORMAT
+python3 scripts/bump_version_enhanced.py patch "Fix character parsing issues" "Fix character name handling in TTS generation"
+
+# Auto-increment minor version (4.5.25 → 4.6.0) - CORRECT FORMAT  
+python3 scripts/bump_version_enhanced.py minor "Add new TTS engine support" "Add Higgs Audio 2 TTS engine with voice cloning"
+```
+
+#### Single-Line Format (Only for Super Minor Changes)
+```bash
+python3 scripts/bump_version_enhanced.py patch "Fix typo in node tooltip" "Fix typo in audio analyzer tooltip"
+```
+
+#### Dry-Run Preview (Test Before Committing)
+```bash
+python3 scripts/bump_version_enhanced.py patch "Fix preview issues" "Fix preview not reflecting filter parameters" --dry-run
+```
+
+#### Auto-Categorization System
+**IMPORTANT**: The script automatically sorts each line of multiline changelog entries into categories based on keywords:
+
+**Keywords for each category:**
+- **Fixed** section: "fix", "bug", "error", "issue", "resolve", "correct", "patch", "crash", "problem", "broken", "compatibility"
+- **Added** section: "add", "new", "implement", "feature", "create", "introduce", "support" (also default for unmatched lines)
+- **Changed** section: "improve", "enhance", "optimize", "update", "modify", "better", "performance", "refactor"  
+- **Removed** section: "remove", "delete", "deprecate", "drop", "eliminate"
+
+**Writing Tips:**
+- **Be intentional with word choice** - start bullet points with appropriate keywords
+- **Line order doesn't matter** - script will reorganize lines by category automatically  
+- **Each line categorized individually** - mix different change types in one changelog description
+- **Commit vs Changelog focus:**
+  - **Commit**: Technical implementation details for developers
+  - **Changelog**: User-facing benefits and impacts
+
+**Bash Syntax Notes:**
+- Multiline strings need proper quoting (opening quote on first line, closing quote on last line)
+- Use `\` (backslash) for line continuation in bash commands
+- Don't add manual category prefixes like "Fixed:" - script handles categorization automatically!
+
 ### Interactive Mode (Recommended for Complex Changes)
+
 ```bash
 python3 scripts/bump_version_enhanced.py 3.2.9 --interactive
 ```
 
 ### Legacy Mode (Same Description for Both)
+
 ```bash
 python3 scripts/bump_version_enhanced.py 3.2.9 "Fix bugs and improve stability"
 ```
 
 ### What the Script Does
+
 - Updates version in `nodes.py`, `README.md`, and `pyproject.toml` 
 - Updates changelog with user-focused description
 - Creates git commit with developer-focused description
@@ -45,36 +127,42 @@ python3 scripts/bump_version_enhanced.py 3.2.9 "Fix bugs and improve stability"
 ### When to Bump Versions
 
 #### Patch (x.x.X)
+
 - Bug fixes
 - Documentation updates
 - Performance improvements
 - Security patches
 
 #### Minor (x.X.0)
+
 - New features
 - New node types
 - Enhanced functionality
 - Backward-compatible changes
 
 #### Major (X.0.0)
+
 - Breaking changes
 - API changes
 - Architecture refactoring
 - Incompatible updates
 
 ### Pre-Bump Checklist
+
 1. ✅ All changes tested and working
 2. ✅ User confirms functionality works
 3. ✅ Git working directory is clean
 4. ✅ All important changes committed
 
 ### Post-Bump Actions
+
 - Script handles git commit automatically
 - Consider pushing to remote if appropriate
 - Update any external documentation
 - Notify users of significant changes
 
 ### Important Notes
+
 - **Never manually edit version files** - always use the script
 - **Only bump when user confirms changes work**
 - **Read detailed guide**: `docs/Dev reports/CLAUDE_VERSION_MANAGEMENT_GUIDE.md`
@@ -83,16 +171,81 @@ python3 scripts/bump_version_enhanced.py 3.2.9 "Fix bugs and improve stability"
 ### Commit vs Changelog Guidelines
 
 #### Commit Description (--commit)
+
 - **Developer diary**: What this specific version bump does
 - **Include internal details**: Bug fixes, refactoring, technical changes
 - **Development perspective**: "Fixed F5-TTS edit issues after refactoring"
 - **Can mention temporary problems**: "Restore functionality broken by restructure"
 
-#### Changelog Description (--changelog)  
+#### Changelog Description (--changelog)
+
 - **User perspective only**: Write for users upgrading from previous version
+- **Use layman terms**: Explain WHAT the feature does, not just technical names
 - **Don't document internal fixes**: Skip temporary issues introduced and fixed during development
 - **Focus on net result**: What changed for the user, not the development process
 - **Example**: If refactoring broke something then fixed it, only mention the refactoring benefit
 
+**🎯 CRITICAL: Always Specify Engine/Component When Applicable**
+- **Include engine name OR component name in titles**: Users need to know which TTS engine or node/feature is affected
+- **❌ Bad**: "Fix SRT processing issues"
+- **✅ Good**: "Fix ChatterBox SRT processing issues"
+- **❌ Bad**: "Fix text persistence in workflows"
+- **✅ Good**: "Fix Multiline TTS Tag Editor text persistence in workflows"
+- **❌ Bad**: "Fix voice cloning errors"
+- **✅ Good**: "Fix Higgs Audio voice cloning errors"
+- **❌ Bad**: "Improve audio quality"
+- **✅ Good**: "Improve F5-TTS audio quality"
+
+**Engine-Specific Title Examples:**
+- "Fix Higgs Audio compatibility with transformers 4.46.3+"
+- "Add VibeVoice multi-speaker support"  
+- "Fix ChatterBox character switching in SRT mode"
+- "Improve F5-TTS speech editing accuracy"
+- "Fix RVC voice conversion model loading"
+- "Add Audio Wave Analyzer timing extraction"
+
+**User-Friendly Language Rules:**
+- ❌ "Add support for MelBandRoFormer models" 
+- ✅ "Add support for advanced audio separation models (vocal/instrumental isolation)"
+- ❌ "Fix VibeVoice parameter propagation to cached engine instances"
+- ✅ "Fix VibeVoice voice settings not applying correctly in some cases"
+- ❌ "Resolve ChatterboxVC loading with .safetensors format"
+- ✅ "Fix voice conversion failing to load newer model formats"
+
+**Always explain obscure terms:**
+- Model names → What they do (e.g., "audio separation", "voice conversion", "text-to-speech")
+- Technical errors → User impact (e.g., "crashes when...", "fails to generate audio")
+- Internal components → User-facing features (e.g., "character voice switching", "long audio generation")
+
 #### Key Principle
-**Commit = Development diary, Changelog = User release notes**
+
+**Commit = Development diary, Changelog = User release notes**
+
+## Troubleshooting
+
+### Common Issues
+
+**"Git working directory is not clean"**
+```bash
+# Check what files are modified
+git status
+
+# Add/commit changes first, then run version bump
+git add .
+git commit -m "Prepare for version bump"
+```
+
+**"Error: Invalid version format"**
+- Use semantic versioning: `4.5.25` (not `v4.5.25` or `4.5`)
+- Or use auto-increment: `patch`, `minor`, `major`
+
+**Bash syntax errors with multiline**
+- Make sure opening quote is on same line as `--commit` or `--changelog`  
+- Make sure closing quote is on its own line
+- Use `\` for line continuation
+
+**Want to see what will happen before committing?**
+```bash
+# Add --dry-run to preview changelog categorization
+python3 scripts/bump_version_enhanced.py patch "description" "changelog" --dry-run
+```

docs/CHARACTER_SWITCHING_GUIDE.md

@@ -10,6 +10,7 @@ ChatterBox Voice now supports seamless character and narrator switching with **l
 
 - **🎭 Universal Character Parsing** - Works with both F5TTS and ChatterBox engines
 - **🌍 Language-Aware Switching** - Switch both character AND language with `[lang:character]` syntax
+- **⚙️ Per-Segment Parameters** - Control seed, temperature, cfg, and other parameters per segment with `[Alice|seed:42|temperature:0.5]` syntax (see [Parameter Switching Guide](PARAMETER_SWITCHING_GUIDE.md))
 - **🔄 Robust Fallback System** - No errors when characters or languages not found
 - **📁 Voice Folder Integration** - Organized character voice management
 - **📺 SRT Support** - Character and language switching in subtitle timing
@@ -67,7 +68,7 @@ Back to the narrator in default language.
 **Note:** If a language model is not available, the system will fallback to the default model with a warning.
 
 ### 3. Voice File Structure
-Organize character voices using filenames in `voices_examples/`:
+Organize character voices using filenames in `models/voices` (preferred) or in the custom_node TTS Audio Suite folder under `voices_examples/`:
 
 ```
 voices_examples/
@@ -406,3 +407,4 @@ These guidelines help ensure optimal F5-TTS generation quality and prevent commo
 ---
 
 *For technical support or feature requests, please check the main README or create an issue on GitHub.*
+

docs/CHATTERBOX_V2_SPECIAL_TOKENS.md

@@ -0,0 +1,140 @@
+# ChatterBox Official 23-Lang v2 Special Tokens
+
+## 🚧 EXPERIMENTAL FEATURE - LIMITED FUNCTIONALITY
+
+**IMPORTANT: These special tokens are present in the v2 vocabulary but may not work as expected!**
+
+Based on community testing and the lack of official documentation from ResembleAI, these emotion/sound tokens appear to be an **incomplete implementation**:
+
+- ✅ **Tokens exist** in the v2 tokenizer vocabulary
+- ⚠️ **Limited effect** - tokens may produce minimal or no audible changes
+- ❌ **No official documentation** - ResembleAI has not published usage guidelines
+- 🔬 **Experimental** - model may not be fully trained to respond to these tokens
+
+**What works (partially):**
+- Some users report `<laughter> hahaha` produces slight effects
+- Results are inconsistent and unreliable
+
+**Community Discussion:**
+- See [ResembleAI/chatterbox Issue #186](https://github.com/resemble-ai/chatterbox/issues/186) - "Use of Emotional Tags Like [laughter] During Generation"
+- No official response from ResembleAI team on proper usage
+
+**Our implementation is ready for when/if ResembleAI improves this feature. Feel free to experiment, but don't expect production-ready results.**
+
+---
+
+## Overview
+
+ChatterBox v2 vocabulary includes special tokens for emotions, sounds, and vocal effects. While the tokenizer supports these tags, the model's response to them is limited and undocumented.
+
+## ⚠️ IMPORTANT: Use Angle Brackets `<tag>`
+
+To avoid conflicts with the character switching system `[CharacterName]`, **use angle brackets `<>` for v2 special tokens**:
+
+- ✅ **Correct**: `<giggle>`, `<sigh>`, `<whisper>`
+- ❌ **Wrong**: `[giggle]`, `[sigh]`, `[whisper]` (conflicts with character names)
+
+The system will automatically convert `<emotion>` → `[emotion]` internally for ChatterBox v2.
+
+## New Special Tokens
+
+### Emotional Expressions
+- `<giggle>` - Light laughter
+- `<laughter>` - Full laughter
+- `<guffaw>` - Loud, boisterous laugh
+- `<sigh>` - Sighing sound
+- `<cry>` - Crying sound
+- `<gasp>` - Gasping sound
+- `<groan>` - Groaning sound
+
+### Breathing & Speech Modifiers
+- `<inhale>` - Inhaling/breath in
+- `<exhale>` - Exhaling/breath out
+- `<whisper>` - Whispered speech
+- `<mumble>` - Mumbled speech
+- `<UH>` - Hesitation sound (uh)
+- `<UM>` - Hesitation sound (um)
+
+### Vocal Performances
+- `<singing>` - Singing voice
+- `<music>` - Musical sounds
+- `<humming>` - Humming sound
+- `<whistle>` - Whistling sound
+
+### Body Sounds
+- `<cough>` - Coughing sound
+- `<sneeze>` - Sneezing sound
+- `<sniff>` - Sniffing sound
+- `<snore>` - Snoring sound
+- `<clear_throat>` - Throat clearing
+- `<chew>` - Chewing sound
+- `<sip>` - Sipping/drinking sound
+- `<kiss>` - Kissing sound
+
+### Animal Sounds
+- `<bark>` - Dog barking
+- `<howl>` - Howling sound
+- `<meow>` - Cat meowing
+
+### Other
+- `<shhh>` - Shushing sound
+- `<gibberish>` - Nonsensical speech
+
+## Usage Examples
+
+```
+Hello there! <giggle> I'm so happy to see you.
+Wait... <UM> I think I forgot something. <sigh>
+<whisper> This is a secret message.
+Look at that! <gasp> It's amazing!
+<singing> La la la la la!
+```
+
+## Important Notes
+
+### ✅ No Conflicts - Safe to Use!
+
+The angle bracket syntax `<emotion>` is specifically designed to avoid conflicts:
+
+- **Character switching** uses `[CharacterName]` - no conflict
+- **Pause tags** use `[pause:2]`, `[wait:1.5]` - no conflict
+- **v2 special tokens** use `<giggle>`, `<sigh>` - no conflict
+
+### Combining with Other Systems
+
+You can freely mix all three systems:
+
+```
+[Alice] Hello! <giggle> Nice to meet you. [pause:0.5] How are you? <whisper> I have a secret.
+[Bob] <gasp> Really? Tell me more! [wait:1]
+```
+
+**Processing order:**
+1. Character tags `[CharacterName]` are extracted first
+2. Pause tags `[pause:XX]` are processed second
+3. v2 special tags `<emotion>` are converted last (to `[emotion]` for the engine)
+
+This ensures everything works together seamlessly!
+
+## Model Version Selection
+
+To use these special tokens:
+
+1. Set **model_version** to **v2** in the ChatterBox Official 23-Lang Engine node
+2. The v2 model will automatically download the enhanced tokenizer files
+3. Special tokens will be processed during generation
+
+## Technical Details
+
+### New v2 Model Files
+- `t3_mtl23ls_v2.safetensors` - Enhanced T3 model with improved tokenization
+- `grapheme_mtl_merged_expanded_v1.json` - Enhanced grapheme/phoneme mappings with 118 special tokens
+- Improved Russian stress handling
+- Enhanced multilingual tokenizer fixes
+
+### Cache Invalidation
+
+The TTS Audio Suite automatically manages cache keys to prevent conflicts:
+- v1 and v2 generations are cached separately
+- Switching between versions will regenerate audio with the correct model
+- Cache keys include `model_version` to ensure proper invalidation

docs/DEPENDENCY_MANAGEMENT_GUIDE.md

@@ -0,0 +1,302 @@
+# Dependency Management Guide
+
+**TTS Audio Suite - Complete dependency compatibility documentation**
+
+This document tracks all package dependencies, conflicts, solutions, and constraints implemented in the install script for universal compatibility across Python versions and platforms.
+
+## Quick Reference
+
+| Package | Version Constraint | Reason | Solution |
+|---------|-------------------|---------|----------|
+| `torch` | `>=2.6.0` for security | CVE-2025-32434 (Higgs Audio) | Smart CUDA detection + upgrade |
+| `numpy` | `>=2.2.0,<2.3.0` | Numba compatibility | Version constraints |
+| `numba` | `>=0.61.2` | NumPy 2.2+ support | Auto-upgrade from 0.61.0 |
+| `librosa` | --no-deps | Forces NumPy downgrade | Isolated installation |
+| `descript-audiotools` | --no-deps | Protobuf 3.19 conflict | Isolated installation |
+| `faiss-cpu` | `>=1.7.4` | Cross-platform compatibility | Linux: try GPU, Windows: CPU |
+
+---
+
+## Critical Dependencies & Conflicts
+
+### **Python 3.13 Audio-Separator + Resampy Issue**
+
+**Problem:** Audio-Separator v0.36.1 claims Python 3.13 support, but dependency `resampy 0.4.3` fails
+- Audio-Separator loads successfully ✅
+- resampy compilation fails with numba in Python 3.13 ❌
+- `NUMBA_DISABLE_JIT=1` doesn't prevent resampy from attempting compilation
+
+**Impact:** Certain UVR models fail with Audio-Separator, force RVC fallback
+```
+Failed in nopython mode pipeline (step: nopython frontend)
+Untyped global name '_resample_loop_s': Cannot determine Numba type of <class 'function'>
+```
+
+**Root Cause:** resampy 0.4.3 has incomplete Python 3.13 + numba support despite Audio-Separator's claims
+
+**Current Solution:** 
+- Audio-Separator fails → RVC fallback works for compatible models (UVR v2/v3 architecture)  
+- Some models (e.g., UVR-DeNoise.pth) incompatible with both engines
+- Clean error messages implemented to avoid confusion
+
+**Model Compatibility:**
+- ✅ **Working Models:** UVR-DeEcho-DeReverb.pth, 6_HP-Karaoke-UVR.pth (via RVC fallback)
+- ❌ **Broken Models:** UVR-DeNoise.pth, models requiring Audio-Separator-specific architectures
+
+**Future Fix:** Wait for resampy or Audio-Separator to fix Python 3.13 numba compilation
+
+---
+
+### 1. **NumPy + Numba Compatibility Crisis**
+
+**Problem:** Different numba versions support different NumPy ranges
+- `numba 0.61.0`: Supports NumPy ≤ 2.1.x ❌
+- `numba 0.61.2+`: Supports NumPy 2.2.x ✅
+
+**Impact:** Core engine loading failures
+```
+❌ ChatterboxTTS not available: Numba needs NumPy 2.1 or less. Got NumPy 2.2.
+❌ F5-TTS not available: Numba needs NumPy 2.1 or less. Got NumPy 2.2.
+```
+
+**Solution:** Smart version detection and upgrade
+```python
+# Detect numba version and upgrade if needed
+if numba_version == "0.61.0" or numba_version == "0.61.1":
+    pip install --upgrade "numba>=0.61.2"
+```
+
+**Constraint:** `numpy>=2.2.0,<2.3.0` + `numba>=0.61.2`
+
+---
+
+### 2. **PyTorch Security Vulnerability (CVE-2025-32434)**
+
+**Problem:** Higgs Audio engine requires PyTorch 2.6+ for security
+- PyTorch < 2.6: transformers blocks torch.load() due to CVE
+- PyTorch 2.6+: Security issue resolved
+
+**Impact:** Higgs Audio fails to load models
+```
+ValueError: Due to a serious vulnerability issue in torch.load...
+```
+
+**Solution:** Smart CUDA detection and upgrade
+- Detect system CUDA version (12.8 → cu124)
+- Install PyTorch 2.6+ with correct CUDA support
+- Force reinstall if switching CPU↔CUDA variants
+
+**Constraint:** `torch>=2.6.0` with appropriate CUDA index
+
+---
+
+### 3. **Librosa Dependency Hell**
+
+**Problem:** librosa forces NumPy downgrades
+- librosa dependencies conflict with our NumPy constraints
+- Causes cascade of package downgrades
+
+**Impact:** Breaks carefully balanced dependency tree
+
+**Solution:** `--no-deps` installation
+```python
+pip install librosa --no-deps
+# Then manually install librosa's actual dependencies
+```
+
+**Dependencies manually installed:**
+- `lazy_loader>=0.1`, `msgpack>=1.0`, `pooch>=1.1`
+- `soxr>=0.3.2`, `decorator>=4.3.0`, `joblib>=1.0`
+- `scikit-learn>=1.1.0`, `audioread>=2.1.9`
+
+---
+
+### 4. **descript-audiotools Protobuf Conflict**
+
+**Problem:** Forces protobuf downgrade 6.x → 3.19.x
+- Breaks other packages expecting modern protobuf
+- Cascading compatibility issues
+
+**Solution:** `--no-deps` installation with manual dependencies
+```python
+pip install descript-audiotools --no-deps
+```
+
+**Dependencies manually installed:**
+- `flatten-dict`, `ffmpy`, `importlib-resources`
+- `randomname`, `markdown2`, `pyloudnorm`
+- `pystoi`, `torch-stoi`, `ipython`, `tensorboard`
+
+---
+
+### 5. **FAISS Platform Compatibility**
+
+**Problem:** faiss-gpu availability varies by platform
+- Linux: `faiss-gpu-cu12`, `faiss-gpu-cu11` available ✅
+- Windows: No GPU version available via pip ❌
+
+**Solution:** Platform-specific installation
+```python
+if not is_windows and cuda_detected:
+    try:
+        pip install faiss-gpu-cu12  # Linux + CUDA
+    except:
+        pip install faiss-cpu      # Fallback
+else:
+    pip install faiss-cpu          # Windows/CPU-only
+```
+
+---
+
+## Python Version Compatibility
+
+### Python 3.13 Specific Issues
+
+| Package | Issue | Solution | Status |
+|---------|-------|----------|--------|
+| `MediaPipe` | Binary incompatible | OpenSeeFace fallback | ✅ Working |
+| `numba` | JIT compilation issues | Disable JIT in Python 3.13 | ✅ Working |
+| `audioop` packages | Not available | Remove from dependencies | ✅ Fixed |
+| `resampy` | Numba compilation fails in Audio-Separator | RVC fallback works for compatible models | ⚠️ Partial |
+
+### Python 3.12 Compatibility
+
+| Package | Issue | Solution | Status |
+|---------|-------|----------|--------|
+| All engines | NumPy/numba mismatch | Smart version detection | ✅ Working |
+| PyTorch CUDA | CPU vs CUDA variants | Force reinstall detection | ✅ Working |
+
+---
+
+## Install Script Logic Flow
+
+### 1. **Environment Detection**
+```python
+- Python version (3.12, 3.13+)
+- Platform (Windows, Linux, macOS)  
+- CUDA availability (nvidia-smi detection)
+- CUDA version (12.8 → cu124, 11.8 → cu118)
+```
+
+### 2. **PyTorch Installation**
+```python
+- Check existing version and CUDA support
+- Upgrade if needed (security + compatibility)
+- Platform-specific CUDA index selection
+```
+
+### 3. **Core Dependencies**
+```python
+- Smart package checking (skip if satisfied)
+- Safe packages installed normally
+- Sub-dependencies for --no-deps packages
+```
+
+### 4. **NumPy + Numba Resolution**  
+```python
+- Detect numba version compatibility
+- Upgrade numba if needed (0.61.0 → 0.61.2+)
+- Install constrained NumPy (>=2.2.0,<2.3.0)
+```
+
+### 5. **Problematic Package Isolation**
+```python
+- Install with --no-deps to prevent conflicts
+- Manually install required sub-dependencies
+- Maintain compatibility with main dependency tree
+```
+
+---
+
+## Package Categories
+
+### ✅ **SAFE Packages** (Normal installation)
+- `transformers>=4.46.3`, `accelerate`, `datasets`
+- `soundfile>=0.12.0`, `sounddevice>=0.4.0`
+- `jieba`, `pypinyin`, `unidecode`
+- `conformer>=0.3.2`, `x-transformers`, `vocos`
+- `s3tokenizer>=0.1.7`, `resemble-perth`
+
+### ⚠️ **PROBLEMATIC Packages** (--no-deps required)
+- `librosa` - Forces NumPy downgrade
+- `descript-audiotools` - Protobuf conflicts  
+- `descript-audio-codec` - Dependency conflicts
+- `cached-path` - Forces package downgrades
+- `torchcrepe` - Conflicts via librosa
+- `onnxruntime` - Forces NumPy 2.3.x (Python 3.13 only)
+
+### 🎯 **VERSION-CRITICAL Packages**
+- `torch>=2.6.0` - Security requirement
+- `numpy>=2.2.0,<2.3.0` - Numba compatibility
+- `numba>=0.61.2` - NumPy 2.2+ support
+- `transformers>=4.46.3` - Higgs Audio compatibility
+
+---
+
+## Testing Matrix
+
+### Verified Working Combinations
+
+| Python | Platform | PyTorch | NumPy | Numba | Status |
+|--------|----------|---------|-------|--------|--------|
+| 3.12.6 | Windows | 2.7.1+cu126 | 2.2.6 | 0.61.2 | ✅ All engines |
+| 3.13.x | Windows | 2.8.0+cpu | 2.2.6 | 0.61.2 | ✅ All engines |
+| 3.12.6 | Linux | 2.6.0+cu121 | 2.2.6 | 0.61.2 | ✅ All engines |
+
+### Known Broken Combinations
+
+| Python | Platform | PyTorch | NumPy | Numba | Issue |
+|--------|----------|---------|-------|--------|-------|
+| 3.12.6 | Windows | Any | 2.2.6 | 0.61.0 | ❌ Engine loading |
+| Any | Any | <2.6.0 | Any | Any | ❌ Higgs Audio CVE |
+
+---
+
+## Troubleshooting Guide
+
+### Engine Loading Failures
+```
+❌ ChatterboxTTS not available: Numba needs NumPy 2.1 or less
+```
+**Cause:** numba 0.61.0 with NumPy 2.2+  
+**Fix:** Run install script to upgrade numba to 0.61.2+
+
+### Higgs Audio Model Loading
+```
+❌ CVE-2025-32434 vulnerability error
+```
+**Cause:** PyTorch < 2.6.0  
+**Fix:** Run install script to upgrade PyTorch 2.6+
+
+### Windows CUDA Issues  
+```
+AssertionError: Torch not compiled with CUDA enabled
+```
+**Cause:** CPU PyTorch on CUDA system  
+**Fix:** Install script detects and reinstalls CUDA PyTorch
+
+---
+
+## Maintenance Notes
+
+### When Adding New Dependencies
+1. **Check conflicts** with existing constraints
+2. **Test with problematic packages** (librosa, descript-*)  
+3. **Update this documentation**
+4. **Add to appropriate category** (Safe/Problematic/Critical)
+
+### When Supporting New Python Versions
+1. **Test numba compatibility** with NumPy constraints
+2. **Check for binary package availability**
+3. **Update compatibility matrix**
+4. **Add version-specific workarounds if needed**
+
+### When Updating Major Dependencies
+1. **Review cascade effects** on sub-dependencies
+2. **Test with --no-deps packages**
+3. **Update version constraints**
+4. **Verify cross-platform compatibility**
+
+---
+
+*Last updated: Version 4.8.19 - Python 3.13 Audio-Separator + resampy compatibility issues documented*

docs/Dev reports/CACHE_ANALYSIS.md

@@ -0,0 +1,702 @@
+# TTS Audio Suite Generation/Audio Caching System - Comprehensive Analysis
+
+## Executive Summary
+
+The TTS Audio Suite has **MULTIPLE FRAGMENTED CACHING SYSTEMS** operating independently with inconsistent cache key construction. The generation cache is hard to fix and often misses because:
+
+1. **Multiple independent cache implementations** (at least 3 separate systems)
+2. **Inconsistent cache key generation** across engines
+3. **Missing parameter normalization** causing floating-point precision issues
+4. **Undocumented parameter handling** with hidden transformations
+5. **Multiple cache levels** (local, unified, instance-level) not coordinated
+6. **Text transformation side-effects** affecting cache keys
+
+---
+
+## 1. CURRENT CACHING SYSTEMS
+
+### System 1: Unified Audio Cache (Primary but underused)
+**Location:** `/home/linux/ComfyUI_TTS_Audio_Suite/utils/audio/cache.py`
+
+- **Global Storage:** `GLOBAL_AUDIO_CACHE = {}` (line 13)
+- **Class:** `AudioCache` (line 235)
+- **Access:** `get_audio_cache()` function (line 343)
+- **Key Generator:** Abstract `CacheKeyGenerator` with engine-specific implementations
+  - `F5TTSCacheKeyGenerator` (line 25)
+  - `ChatterBoxCacheKeyGenerator` (line 50)
+  - `ChatterBoxOfficial23LangCacheKeyGenerator` (line 76)
+  - `HiggsAudioCacheKeyGenerator` (line 103)
+  - `VibeVoiceCacheKeyGenerator` (line 144)
+  - `IndexTTSCacheKeyGenerator` (line 184)
+
+**Issue:** This unified system exists but is ONLY used in ChatterBox TTS node through `create_cache_function()`. Most nodes use local caches instead.
+
+### System 2: Node-Local Audio Cache (ChatterBox)
+**Location:** `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/chatterbox/chatterbox_srt_node.py`
+
+- **Global Storage:** `GLOBAL_AUDIO_CACHE = {}` (line 46)
+- **Separate from unified cache** - different dict, different module
+- **Methods:**
+  - `_generate_segment_cache_key()` (line 361)
+  - `_get_cached_segment_audio()` (line 381)
+  - `_cache_segment_audio()` (line 385)
+
+**Issue:** This is a DUPLICATE system that doesn't use the unified cache. Creates cache misses when switching between unified node and legacy node.
+
+### System 3: Voice Conversion Iteration Cache
+**Location:** `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/unified/voice_changer_node.py`
+
+- **Global Storage:** `GLOBAL_RVC_ITERATION_CACHE = {}` (line 48)
+- **Purpose:** Cache iterative refinement passes for RVC voice conversion
+- **Methods:**
+  - `_generate_rvc_cache_key()` (line 581)
+  - `_get_cached_rvc_iterations()` (line 620)
+  - `_cache_rvc_result()` (line 628)
+
+**Issue:** RVC-specific, limited to 5 iterations max (line 634), not integrated with audio cache.
+
+### System 4: Engine Instance Cache (Multiple locations)
+**Location:** `tts_text_node.py` (line 134), `voice_changer_node.py` (line 90)
+
+- **Storage:** Instance variable `self._cached_engine_instances = {}`
+- **Purpose:** Cache engine node instances to prevent model reloading
+- **Cache Key:** Hash of `engine_type` + stable parameters (line 182)
+- **Timestamp-based invalidation:** Uses global `_global_cache_invalidation_flag`
+
+**Issue:** This is ENGINE caching, not GENERATION caching. Confuses the user about what's being cached.
+
+### System 5: Voice Discovery Cache (Persistent)
+**Location:** `/home/linux/ComfyUI_TTS_Audio_Suite/utils/voice/cache_manager.py`
+
+- **Storage:** Disk-based JSON in `.cache/voice_discovery.json`
+- **Purpose:** Cache voice character discovery to speed up initialization
+- **Background refresh:** Uses threading to update cache after ComfyUI loads
+
+**Issue:** This is voice DISCOVERY cache, not generation audio cache.
+
+---
+
+## 2. CACHE KEY CONSTRUCTION - DETAILED ANALYSIS
+
+### ChatterBox Unified Cache Key
+**File:** `/home/linux/ComfyUI_TTS_Audio_Suite/utils/audio/cache.py:50-73`
+
+```python
+cache_data = {
+    'text': params.get('text', ''),                    # FULL TEXT WITH CHARACTER PREFIX
+    'exaggeration': params.get('exaggeration', 1.0),
+    'temperature': params.get('temperature', 0.8),
+    'cfg_weight': params.get('cfg_weight', 1.0),
+    'seed': params.get('seed', 0),
+    'audio_component': params.get('audio_component', ''),  # HASHED AUDIO CONTENT
+    'model_source': params.get('model_source', ''),        # LANGUAGE + SOURCE
+    'device': params.get('device', ''),                     # NOT RESOLVED (can vary)
+    'language': params.get('language', 'English'),
+    'character': params.get('character', 'narrator'),
+    'engine': 'chatterbox',
+    'repetition_penalty': params.get('repetition_penalty', 1.0),
+    'min_p': params.get('min_p', 0.0),
+    'top_p': params.get('top_p', 1.0)
+}
+cache_string = str(sorted(cache_data.items()))
+cache_key = hashlib.md5(cache_string.encode()).hexdigest()
+```
+
+**Problems:**
+1. **No floating-point rounding** - 0.8 vs 0.80000001 = different keys
+2. **`audio_component` is hashed waveform** - but text still includes character prefix (duplication)
+3. **Device resolution inconsistency** - stores "auto" literally, causing misses when it resolves to "cuda"
+4. **Character prefixed to text** - line 282 of cache.py: `cache_params['text'] = f"{cache_params.get('character', 'narrator')}:{text_content}"`
+
+### ChatterBox Legacy Cache Key
+**File:** `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/chatterbox/chatterbox_srt_node.py:361-379`
+
+```python
+cache_data = {
+    'text': subtitle_text,
+    'exaggeration': exaggeration,
+    'temperature': temperature,
+    'cfg_weight': cfg_weight,
+    'seed': seed,
+    'audio_prompt_component': audio_prompt_component,
+    'model_source': model_source,
+    'device': device,
+    'language': language,
+    'engine': 'chatterbox_srt'
+}
+cache_string = str(sorted(cache_data.items()))
+cache_key = hashlib.md5(cache_string.encode()).hexdigest()
+```
+
+**Problems:**
+1. Uses `chatterbox_srt` as engine (different from `chatterbox`)
+2. No character prefix in text (different from unified)
+3. Stores different parameters (no repetition_penalty, min_p, top_p)
+4. **Result: Cache misses when using SRT node vs TTS node**
+
+### F5-TTS Unified Cache Key
+**File:** `/home/linux/ComfyUI_TTS_Audio_Suite/utils/audio/cache.py:25-47`
+
+```python
+cache_data = {
+    'text': params.get('text', ''),
+    'model_name': params.get('model_name', ''),        # INCLUDES MODEL NAME
+    'device': params.get('device', ''),
+    'audio_component': params.get('audio_component', ''),
+    'ref_text': params.get('ref_text', ''),            # REFERENCE TEXT (!)
+    'temperature': params.get('temperature', 0.8),
+    'speed': params.get('speed', 1.0),
+    'target_rms': params.get('target_rms', 0.1),
+    'cross_fade_duration': params.get('cross_fade_duration', 0.15),
+    'nfe_step': params.get('nfe_step', 32),
+    'cfg_strength': params.get('cfg_strength', 2.0),
+    'seed': params.get('seed', 0),
+    'character': params.get('character', 'narrator'),
+    'engine': 'f5tts'
+}
+```
+
+**Critical Issues:**
+1. **Includes `ref_text` in cache key** - Same text with different reference audio = different key (CORRECT)
+2. **No floating-point rounding** on ANY of the 9 float parameters
+3. **Device not resolved** - "auto" won't match "cuda"
+4. **`cross_fade_duration`** is POST-PROCESSING, shouldn't be in key (doesn't affect generation)
+5. **`audio_component`** should be enough to distinguish references, shouldn't also need ref_text
+
+### VibeVoice Cache Key
+**File:** `/home/linux/ComfyUI_TTS_Audio_Suite/utils/audio/cache.py:144-181`
+
+```python
+# FLOATING POINT ROUNDING PRESENT (but inconsistent)
+cfg_scale = round(float(cfg_scale), 3)
+temperature = round(float(temperature), 3)
+top_p = round(float(top_p), 3)
+
+cache_data = {
+    'text': params.get('text', ''),
+    'cfg_scale': cfg_scale,
+    'temperature': temperature,
+    'top_p': top_p,
+    'use_sampling': params.get('use_sampling', False),
+    'seed': params.get('seed', 42),
+    'model_source': params.get('model_source', 'vibevoice-1.5B'),
+    'device': params.get('device', 'auto'),
+    'max_new_tokens': params.get('max_new_tokens'),
+    'multi_speaker_mode': params.get('multi_speaker_mode', 'Custom Character Switching'),
+    'audio_component': params.get('audio_component', ''),
+    'character': params.get('character', 'narrator'),
+    'inference_steps': params.get('inference_steps', 20),
+    'attention_mode': params.get('attention_mode', 'eager'),
+    'quantize': params.get('quantize', False),
+    'engine': 'vibevoice'
+}
+```
+
+**Good:** Has rounding for floats
+**Bad:** Only rounds 3 params, misses others. Device still "auto". Includes post-processing params like attention_mode (affects inference, not generation output).
+
+### IndexTTS Cache Key
+**File:** `/home/linux/ComfyUI_TTS_Audio_Suite/utils/audio/cache.py:184-232`
+
+```python
+# SELECTIVE ROUNDING (inconsistent)
+temperature = round(float(temperature), 3)
+top_p = round(float(top_p), 3)
+emotion_alpha = round(float(emotion_alpha), 3)
+
+cache_data = {
+    'text': params.get('text', ''),
+    'speaker_audio': params.get('speaker_audio', ''),      # AUDIO PATH STRING
+    'emotion_audio': params.get('emotion_audio', ''),      # AUDIO PATH STRING
+    'emotion_alpha': emotion_alpha,
+    'emotion_vector': params.get('emotion_vector'),        # LIST (unhashable!)
+    'use_emotion_text': params.get('use_emotion_text', False),
+    'emotion_text': params.get('emotion_text', ''),
+    'use_random': params.get('use_random', False),
+    'seed': params.get('seed', 0),
+    'do_sample': params.get('do_sample', True),
+    'temperature': temperature,
+    'top_p': top_p,
+    'top_k': params.get('top_k', 30),
+    'length_penalty': params.get('length_penalty', 0.0),   # NOT ROUNDED!
+    'num_beams': params.get('num_beams', 3),
+    'repetition_penalty': params.get('repetition_penalty', 10.0),  # NOT ROUNDED!
+    'max_mel_tokens': params.get('max_mel_tokens', 1500),
+    'max_text_tokens_per_segment': params.get('max_text_tokens_per_segment', 120),
+    'interval_silence': params.get('interval_silence', 200),
+    'model_name': params.get('model_name', 'IndexTTS-2'),
+    'device': params.get('device', 'auto'),
+    'character': params.get('character', 'narrator'),
+    'use_torch_compile': params.get('use_torch_compile', False),
+    'use_accel': params.get('use_accel', False),
+    'stream_return': params.get('stream_return', False),
+    'more_segment_before': params.get('more_segment_before', 0),
+    'engine': 'index_tts'
+}
+```
+
+**Critical Issues:**
+1. **Stores audio paths as strings** - temp paths will vary, causing cache misses
+2. **`emotion_vector` is a list** - unhashable, str() conversion is fragile
+3. **Selective float rounding** - only 3 params rounded, others like `length_penalty`, `repetition_penalty` are not
+4. **Mixed use of AUDIO and PATH** - speaker_audio and emotion_audio are path strings, not hashes
+
+---
+
+## 3. CACHE MISS ROOT CAUSES
+
+### Root Cause #1: Duplicate Cache Systems
+Two separate `GLOBAL_AUDIO_CACHE` dicts exist:
+- `/home/linux/ComfyUI_TTS_Audio_Suite/utils/audio/cache.py:13`
+- `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/chatterbox/chatterbox_srt_node.py:46`
+- `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/f5tts/f5tts_srt_node.py:43`
+
+**Result:** Using unified TTS Text node and then SRT node = cache miss (different dicts)
+
+### Root Cause #2: Floating-Point Precision
+Example from ChatterBox:
+```python
+# Temperature=0.8 in unified cache
+temperature = params.get('temperature', 0.8)  # Key generated with 0.8
+
+# Temperature from ComfyUI slider
+temperature = 0.80000001  # Different key!
+```
+
+**Affected engines:** ChatterBox, F5-TTS (all 9 float params), all others without rounding
+
+### Root Cause #3: Device Resolution Inconsistency
+```python
+# Unified cache stores device literally
+'device': params.get('device', ''),  # Stores "auto"
+
+# But actual resolution varies
+tts_text_node.py line 159:
+from utils.device import resolve_torch_device
+resolved_device = resolve_torch_device(config.get('device', 'auto'))  # Returns "cuda"
+```
+
+**Result:** Engine cache key includes resolved device, generation cache key includes "auto". Mismatch!
+
+### Root Cause #4: Text Transformation Side-Effects
+**In `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/chatterbox/chatterbox_tts_node.py`:**
+
+Line 282 of unified cache adds character prefix:
+```python
+cache_params['text'] = f"{cache_params.get('character', 'narrator')}:{text_content}"
+```
+
+But line 449 applies crash protection AFTER cache key generation:
+```python
+protected_text = self._pad_short_text_for_chatterbox(processed_text, crash_protection_template)
+# Cache key was generated with unpadded text
+# But generation uses padded text
+```
+
+**Result:** 
+- Cache key generated: `narrator:hello`
+- Cache lookup tries: `narrator:hello`
+- Cache miss because audio was generated with: `narrator:hmm ,, hello hmm ,,`
+
+### Root Cause #5: Audio Component Hash Inconsistency
+```python
+# chatterbox_tts_node.py line 603-605
+stable_audio_component = self._generate_stable_audio_component(
+    inputs.get("reference_audio"), inputs.get("audio_prompt_path", "")
+)
+
+# Generates hash from WAVEFORM CONTENT
+# But if same file accessed via:
+# 1. Dropdown (direct path)
+# 2. Character Voices node (copied to temp)
+# = Different file path = Different hash = CACHE MISS
+```
+
+### Root Cause #6: Parameter Duplication in Cache Key
+**In F5-TTS cache key:**
+```python
+'audio_component': params.get('audio_component', ''),  # Hash of waveform
+'ref_text': params.get('ref_text', ''),                # Reference text
+```
+
+If same text with different reference audio:
+- audio_component differs ✓ (good - prevents collision)
+- But if reference audio is different FILE with same CONTENT:
+  - audio_component might be same ✓
+  - ref_text is same ✓
+  - But actual voice characteristics differ ✗
+
+### Root Cause #7: Untracked Parameter Transformations
+```python
+# chatterbox_tts_node.py line 434-436
+processed_text, pause_segments = PauseTagProcessor.preprocess_text_with_pause_tags(
+    inputs["text"], True
+)
+# Cache key uses processed_text, but what if PauseTagProcessor changes the text?
+# [pause:500ms] removed? Transformed? Cache key would differ!
+```
+
+### Root Cause #8: Engine Instance Cache vs Generation Cache Confusion
+```python
+# tts_text_node.py line 182
+cache_key = f"{engine_type}_{hashlib.md5(str(sorted(stable_params.items())).encode()).hexdigest()[:8]}"
+# This is ENGINE instance cache
+
+# But users think they're caching generation results
+# They're actually just reusing the same engine model
+# Generation parameters affect the OUTPUT, not the CACHE KEY
+```
+
+---
+
+## 4. PARAMETER ANALYSIS: What Should/Shouldn't Invalidate Cache
+
+### ChatterBox Parameters
+
+| Parameter | Should Invalidate? | Current Behavior | Issue |
+|-----------|-------------------|------------------|-------|
+| `text` | YES | YES | Text transformation side-effects |
+| `character` | YES | YES | Included in cache key |
+| `exaggeration` | YES | YES | No rounding |
+| `temperature` | YES | YES | No rounding - floating point issues |
+| `cfg_weight` | YES | YES | No rounding |
+| `seed` | YES | YES | ✓ Correct |
+| `audio_component` | YES | YES | ✓ Hashed correctly |
+| `language` | YES | YES | ✓ Correct |
+| `device` | NO* | YES | *Should NOT - output doesn't differ |
+| `reference_audio` | YES (indirect) | YES | Via audio_component ✓ |
+| `crash_protection_template` | YES | NO* | *MISSING - changes text! |
+| `enable_chunking` | MAYBE | NO | Pre-processing, doesn't affect segment cache |
+| `chunk_combination_method` | NO | NO | Post-processing ✓ |
+| `silence_between_chunks_ms` | NO | NO | Post-processing ✓ |
+| `batch_size` | NO | NO | Streaming mode, doesn't affect generation ✓ |
+
+### F5-TTS Parameters
+
+| Parameter | Should Invalidate? | Current Behavior | Issue |
+|-----------|-------------------|------------------|-------|
+| `text` | YES | YES | ✓ |
+| `ref_text` | YES | YES | ✓ Correct - different reference = different voice |
+| `model_name` | YES | YES | ✓ Different model |
+| `seed` | YES | YES | ✓ |
+| `temperature` | YES | YES | No rounding |
+| `speed` | NO* | YES | *WRONG - doesn't affect generation, F5 handles this |
+| `target_rms` | NO* | YES | *WRONG - post-processing normalization |
+| `cfg_strength` | YES | YES | Affects generation quality |
+| `nfe_step` | YES | YES | Affects generation quality |
+| `cross_fade_duration` | NO | YES | *WRONG - post-processing |
+| `audio_component` | YES | YES | ✓ Reference audio identifier |
+| `character` | YES | YES | ✓ |
+| `device` | NO | YES | *WRONG - doesn't affect generation |
+
+---
+
+## 5. DESIGN FLAWS IDENTIFIED
+
+### Flaw #1: Architectural Fragmentation
+**Problem:** 5 independent cache systems with no coordination
+- Unified cache underused
+- Legacy SRT nodes use separate cache
+- Voice conversion has separate cache
+- Engine instance cache conflates with generation cache
+- Voice discovery cache is orthogonal
+
+**Impact:** Cache misses when switching between node types, impossible to manage/clear all caches
+
+### Flaw #2: Floating-Point Precision Not Normalized
+**Problem:** 
+```python
+# Different rounding across engines
+VibeVoice: round(x, 3)
+IndexTTS: round(x, 3) but only 3 params
+ChatterBox: No rounding
+F5-TTS: No rounding
+```
+
+**Impact:** Workflow parameter change from UI slider (0.8) vs code (0.80000001) = cache miss
+
+### Flaw #3: Device Parameter Included in Cache Key
+**Problem:**
+```python
+# Unified cache stores device
+'device': params.get('device', ''),
+
+# But device="auto" + resolved device="cuda" = different keys
+# Yet output is identical!
+```
+
+**Impact:** Unnecessary cache misses
+
+### Flaw #4: Text Pre-Processing Side-Effects Not Tracked
+**Problems:**
+1. Crash protection padding happens AFTER cache key generation
+2. Pause tag preprocessing happens AFTER cache key generation
+3. Character prefix added during cache key generation
+4. Text chunking happens separately
+
+**Impact:** Cache key is for unprocessed text, but generation uses processed text
+
+### Flaw #5: Audio Component Hashing Fragile
+**Problem:**
+```python
+# audio_component = hash of waveform bytes
+# If reference audio loaded different ways:
+# 1. From dropdown -> actual file path
+# 2. From Character Voices -> temp file
+# 3. From direct audio input -> memory waveform
+# = Same audio, potentially different hashes
+```
+
+**Impact:** Cache misses for same voice when referenced differently
+
+### Flaw #6: Mixed Scopes in Cache Keys
+**Problem:**
+```python
+# Per-segment cache parameters (text, audio, seed, temperature)
+# vs
+# Workflow-level parameters (device, model, language)
+# = Same cache key namespace treats both equally
+
+# Result: Changing device invalidates ALL segment caches
+# But device doesn't affect generation!
+```
+
+### Flaw #7: No Cache Invalidation Strategy
+**Problem:**
+- Global caches cleared how? Only manual `audio_cache.clear_cache()`
+- No TTL/LRU eviction
+- Memory can grow unbounded
+- No detection of "stale" cache entries after model updates
+
+**Impact:** Users can't trust cache after model updates; memory leaks
+
+### Flaw #8: Engine Instance Cache != Generation Cache
+**Problem:**
+```python
+# Users expect:
+# "Cache generation results for fast re-run"
+
+# What actually happens:
+# "Cache engine instance to avoid reloading model"
+
+# Different concepts, but code conflates them!
+```
+
+**Impact:** Users confused why re-running with different parameters still reuses cache
+
+---
+
+## 6. SPECIFIC EXAMPLES OF CACHE MISSES
+
+### Example 1: Temperature Slider Changes
+```
+User workflow:
+1. Set temperature=0.8, generate (CACHE HIT expected next time)
+2. Adjust slider to temperature=0.80000001
+3. Run again, expect CACHE HIT
+4. Result: CACHE MISS (different float representation)
+```
+
+### Example 2: Using Unified TTS vs Legacy SRT Node
+```
+User workflow:
+1. Use "🎤 TTS Text" (unified) node with ChatterBox engine
+   -> Audio stored in utils/audio/cache.py:GLOBAL_AUDIO_CACHE
+2. Switch to "ChatterBox TTS SRT" legacy node
+   -> Looks in nodes/chatterbox/chatterbox_srt_node.py:GLOBAL_AUDIO_CACHE
+3. Result: CACHE MISS (different dict)
+```
+
+### Example 3: Crash Protection Changes Cache
+```
+User workflow:
+1. Generate with default crash_protection_template="hmm ,, {seg} hmm ,,"
+   Cache key: hash("narrator:hello")
+   Audio generated from: "narrator:hmm ,, hello hmm ,,"
+   
+2. Change crash_protection_template="" (disable padding)
+3. Try to regenerate same text "hello"
+4. Result: CACHE HIT returns audio for "hmm ,, hello hmm ,,"
+   But user expects audio for "hello"
+   = WRONG AUDIO RETURNED
+```
+
+### Example 4: Device Parameter Mismatch
+```
+User workflow:
+1. Set device="auto" on first run
+   Engine instance cache key: "f5tts_abc123"
+   Generation cache key includes 'device': 'auto'
+   
+2. Device resolves to "cuda" internally
+   
+3. Switch device="cuda" explicitly
+   Engine instance cache key: "f5tts_abc123" (same!)
+   Generation cache key now includes 'device': 'cuda' (different!)
+   
+4. Result: Engine reused (good) but generation cache misses (bad)
+```
+
+### Example 5: Reference Audio Path Variation
+```
+User workflow:
+1. Use Character Voices node
+   -> Reference audio hashed from temp file
+   -> audio_component = "ref_audio_abc123_24000"
+   -> Cache key includes this hash
+   
+2. Switch to dropdown selector (same voice file)
+   -> Reference audio hashed from permanent file
+   -> audio_component = "file_audio_def456_24000"
+   -> DIFFERENT HASH despite same voice content!
+   
+3. Result: CACHE MISS
+```
+
+---
+
+## 7. CURRENT IMPLEMENTATION PATTERNS
+
+### Pattern 1: ChatterBox TTS Node (Good Model)
+File: `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/chatterbox/chatterbox_tts_node.py:454-479`
+
+```python
+# Generates stable audio component ONCE
+stable_audio_component = self._generate_stable_audio_component(...)
+
+# Creates cache function with ALL parameters
+cache_fn = create_cache_function(
+    "chatterbox",
+    character=character,
+    exaggeration=exaggeration,
+    # ... all params
+    audio_component=audio_component,
+)
+
+# Uses cache function for lookup/store
+cached_audio = cache_fn(protected_text)
+if cached_audio:
+    return cached_audio
+
+audio = self.generate_tts_audio(protected_text, ...)
+cache_fn(protected_text, audio_result=audio)
+```
+
+**Strengths:**
+- Uses unified cache system
+- Generates audio component once
+- Uses cache_fn pattern
+
+**Weaknesses:**
+- Applies crash protection AFTER defining cache params
+- Cache key includes protected text, but params generated with unprotected text
+- No floating-point rounding
+
+### Pattern 2: F5-TTS Node (Different Approach)
+File: `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/f5tts/f5tts_node.py`
+
+**Observation:** Doesn't implement caching at generation level! Instead:
+- Relies on unified cache through base node
+- Or relies on F5-TTS model-level caching internally
+
+### Pattern 3: Voice Conversion (Iteration Caching)
+File: `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/unified/voice_changer_node.py:154-175`
+
+```python
+cache_key = self._generate_rvc_cache_key(processed_source_audio, rvc_model, config)
+cached_iterations = self._get_cached_rvc_iterations(cache_key, refinement_passes)
+
+if refinement_passes in cached_iterations:
+    return cached_iterations[refinement_passes]  # Full cache hit
+
+# Start from highest cached iteration
+for i in range(refinement_passes, 0, -1):
+    if i in cached_iterations:
+        current_audio = cached_iterations[i][0]
+        start_iteration = i
+        break
+
+# Continue from cached iteration
+for iteration in range(start_iteration, refinement_passes):
+    converted_audio = rvc_engine.convert_voice(...)
+    self._cache_rvc_result(cache_key, iteration_num, ...)
+```
+
+**Strengths:**
+- Smart iteration-level caching
+- Can resume from partial progress
+- Limited to 5 iterations (memory bound)
+
+**Weaknesses:**
+- Separate system from generation cache
+- Not integrated with audio cache
+
+---
+
+## 8. ROOT CAUSE RANKING
+
+| Rank | Root Cause | Impact | Difficulty to Fix |
+|------|-----------|--------|-------------------|
+| 1 | Multiple cache dicts (fragmentation) | Very High | Medium |
+| 2 | No floating-point rounding | High | Low |
+| 3 | Device parameter in cache key | High | Low |
+| 4 | Text pre-processing side-effects | High | Medium |
+| 5 | Crash protection not in cache key | Medium | Low |
+| 6 | Audio component hashing fragile | Medium | Medium |
+| 7 | Parameter duplication in F5-TTS key | Low | Low |
+| 8 | No cache invalidation strategy | Medium | High |
+
+---
+
+## 9. DESIGN ISSUES SUMMARY
+
+### Consolidated Cache Architecture Needed
+Current: 5 independent systems
+Desired: 1 unified system with compartments
+
+### Parameter Normalization Missing
+- No rounding of floats to consistent precision
+- Device parameter should be resolved before cache key
+- Text transformations should be applied before cache key generation
+
+### Scope Separation Missing
+- Per-segment parameters (text, seed, audio)
+- Workflow parameters (device, model, language)
+- Processing parameters (chunking, silence)
+
+Should use:
+- Segment-level cache key = text + audio + generation params
+- Workflow-level identifier = device + model + language (separate tracking)
+
+### Cache Key Construction Inconsistent
+- F5-TTS includes too many parameters (speed, target_rms, cross_fade_duration)
+- ChatterBox missing crash_protection in key
+- IndexTTS stores audio paths instead of hashes
+- No engine has consistent float rounding
+
+---
+
+## 10. RECOMMENDATIONS
+
+### Short Term (Easy Fixes)
+1. **Add float rounding** to all engines (3 decimal places)
+2. **Remove device from generation cache key** (doesn't affect output)
+3. **Remove cross_fade/target_rms/speed from F5-TTS cache key**
+4. **Add crash_protection_template to ChatterBox cache key**
+5. **Fix audio_component hashing** to be consistent regardless of load method
+
+### Medium Term (Moderate Effort)
+1. **Consolidate cache dicts** into single GLOBAL_AUDIO_CACHE
+2. **Separate cache scopes:**
+   - Segment cache key = hash(text + audio_component + generation_params)
+   - Workflow tracking = (engine_type, model_name, device, language)
+3. **Move text transformation BEFORE cache key generation**
+4. **Implement cache size limits** with LRU eviction
+
+### Long Term (Architectural)
+1. **Unified cache system** with clear API
+2. **Configuration-driven cache parameters** per engine (explicit list of what invalidates cache)
+3. **Cache statistics and debugging** UI in ComfyUI
+4. **TTL-based invalidation** after model updates
+5. **Persistent cache option** for heavy workflows (disk-based)
+

docs/Dev reports/CACHE_ISSUES_SUMMARY.md

@@ -0,0 +1,280 @@
+# TTS Audio Suite Cache Issues - Executive Summary
+
+## The Problem
+
+Generation cache is **hard to fix and often misses** due to **architectural fragmentation** and **inconsistent parameter handling**.
+
+## Quick Facts
+
+- **5 separate cache systems** operating independently
+- **8 major root causes** identified with code locations
+- **22 specific cache key construction issues** across engines
+- **At least 5 real-world cache miss scenarios** documented
+
+## Root Causes (Ranked by Impact)
+
+### 1. Multiple Cache Dictionaries (CRITICAL)
+**Files:**
+- `/home/linux/ComfyUI_TTS_Audio_Suite/utils/audio/cache.py:13` (Unified cache)
+- `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/chatterbox/chatterbox_srt_node.py:46` (Legacy ChatterBox)
+- `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/f5tts/f5tts_srt_node.py:43` (Legacy F5-TTS)
+
+**Problem:** Using unified "🎤 TTS Text" node then switching to "ChatterBox TTS SRT" legacy node = cache miss because they use different `GLOBAL_AUDIO_CACHE` dicts.
+
+### 2. No Float Rounding (HIGH PRIORITY FIX)
+**Affected files:**
+- `/home/linux/ComfyUI_TTS_Audio_Suite/utils/audio/cache.py:50-73` (ChatterBox)
+- `/home/linux/ComfyUI_TTS_Audio_Suite/utils/audio/cache.py:25-47` (F5-TTS)
+- All legacy SRT nodes
+
+**Problem:** Temperature=0.8 vs 0.80000001 = different cache keys despite being same parameter
+
+**Impact:** Moving UI slider = cache miss
+
+### 3. Device Parameter Inconsistency (HIGH)
+**Problem:** Cache key stores "auto" but engine instance cache resolves to "cuda"
+- Engine cache resolved: `resolve_torch_device("auto")` → "cuda"
+- Generation cache literal: 'device': "auto"
+- Result: Cache misses when switching explicit device selections
+
+### 4. Text Pre-Processing Side-Effects (HIGH)
+**File:** `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/chatterbox/chatterbox_tts_node.py:449`
+
+**Problem:**
+```
+1. Crash protection padding applied AFTER cache key generation
+2. Pause tag preprocessing applied separately
+3. Cache key is for "hello", audio generated from "hmm ,, hello hmm ,,"
+4. If crash_protection_template changed, wrong audio returned
+```
+
+### 5. Crash Protection Not In Cache Key (MEDIUM)
+**File:** `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/chatterbox/chatterbox_tts_node.py`
+
+**Problem:** Changing `crash_protection_template` doesn't invalidate cache
+- Same text → same cache key
+- But protection template changes actual text used for generation
+- Result: Returns audio for different protected text
+
+### 6. Audio Component Hashing Fragile (MEDIUM)
+**File:** `/home/linux/ComfyUI_TTS_Audio_Suite/utils/audio/audio_hash.py:24`
+
+**Problem:** Same voice file, different loading methods = different hashes
+- From dropdown: hashes actual file → "file_audio_abc123"
+- From Character Voices (temp): hashes temp file → "ref_audio_def456"
+- From direct audio: hashes waveform → "ref_audio_ghi789"
+- All are same audio, different hashes = cache misses
+
+### 7. Parameter Duplication in F5-TTS (LOW)
+**File:** `/home/linux/ComfyUI_TTS_Audio_Suite/utils/audio/cache.py:25-47`
+
+**Problem:** F5-TTS cache key includes post-processing parameters
+- `speed` doesn't affect generation (F5 handles this)
+- `target_rms` is normalization (post-processing)
+- `cross_fade_duration` is for combining chunks (post-processing)
+- Changing these shouldn't invalidate cache, but does
+
+### 8. No Cache Invalidation Strategy (MEDIUM)
+**Problem:**
+- No TTL/LRU eviction
+- Memory grows unbounded
+- No detection of "stale" entries after model updates
+- Manual `audio_cache.clear_cache()` only clears unified cache, not legacy caches
+
+## Design Issues Summary
+
+### Issue #1: Architectural Fragmentation
+```
+Unified Cache (utils/audio/cache.py)
+├─ ChatterBoxCacheKeyGenerator
+├─ F5TTSCacheKeyGenerator
+├─ VibeVoiceCacheKeyGenerator
+└─ IndexTTSCacheKeyGenerator
+   
+Legacy ChatterBox Cache (nodes/chatterbox/chatterbox_srt_node.py)
+   └─ Independent _generate_segment_cache_key()
+
+Legacy F5-TTS Cache (nodes/f5tts/f5tts_srt_node.py)
+   └─ Same as ChatterBox SRT
+
+Voice Conversion Cache (nodes/unified/voice_changer_node.py)
+   └─ GLOBAL_RVC_ITERATION_CACHE (separate dict)
+
+Engine Instance Cache (tts_text_node.py)
+   └─ self._cached_engine_instances (confused with generation cache)
+
+Voice Discovery Cache (utils/voice/cache_manager.py)
+   └─ Persistent disk cache (unrelated to generation)
+```
+
+**Result:** Can't centrally manage cache; users confused about what's cached
+
+### Issue #2: Parameter Normalization Missing
+No consistent float rounding across engines:
+- VibeVoice: `round(x, 3)` for 3 params only
+- IndexTTS: `round(x, 3)` for 3 params, others unrounded
+- ChatterBox: No rounding
+- F5-TTS: No rounding
+
+### Issue #3: Scope Separation Missing
+Cache key conflates different parameter types:
+- **Per-segment** (text, seed, audio) - should invalidate cache
+- **Workflow** (device, model, language) - should NOT invalidate individual segments
+- **Processing** (chunking, silence) - should NOT invalidate cache
+
+Currently all in same namespace, causing false invalidations.
+
+## Real-World Examples
+
+### Example 1: Temperature Slider (Most Common)
+```
+1. Set temperature=0.8, generate audio
+2. Adjust slider slightly (0.800001)
+3. Expect cache hit, get cache miss
+```
+
+### Example 2: Node Switching
+```
+1. Use "🎤 TTS Text" node → cache stored in utils/audio/cache.py
+2. Switch to "ChatterBox TTS SRT" legacy node → looks in nodes/chatterbox/
+3. Same text but different cache dict = miss
+```
+
+### Example 3: Crash Protection
+```
+1. Generate "hello" with crash_protection="hmm ,, {seg} hmm ,,"
+   Audio generated from: "hmm ,, hello hmm ,,"
+2. Disable crash protection and regenerate
+3. Cache returns old audio with padding, user expects bare audio
+```
+
+### Example 4: Device Parameter
+```
+1. Use device="auto" on GPU machine
+2. Set device="cuda" explicitly
+3. Engine cache reuses (good) but generation cache misses (bad)
+```
+
+## Parameters That SHOULD/SHOULDN'T Invalidate Cache
+
+### ChatterBox Should Invalidate
+- text, character, exaggeration, temperature, cfg_weight, seed, audio_component, language
+- **MISSING:** crash_protection_template (changes actual text!)
+
+### ChatterBox Should NOT Invalidate
+- device (doesn't affect generation output)
+- chunk_combination_method, silence_between_chunks_ms (post-processing)
+- batch_size (streaming mode setting)
+
+### F5-TTS Should Invalidate
+- text, ref_text, model_name, seed, temperature, cfg_strength, nfe_step, audio_component, character
+
+### F5-TTS Should NOT Invalidate
+- speed (F5 handles this, doesn't affect segment generation)
+- target_rms (post-processing normalization)
+- cross_fade_duration (post-processing for combining segments)
+- device (doesn't affect generation output)
+
+## Fix Priority
+
+### Must Fix (Breaks functionality)
+1. Add float rounding (trivial, high impact)
+2. Consolidate cache dicts (medium effort, high impact)
+3. Add crash_protection_template to cache key (trivial, high impact)
+
+### Should Fix (Improves reliability)
+4. Remove device from generation cache key (trivial)
+5. Remove post-processing params from F5-TTS key (trivial)
+6. Fix audio_component hashing consistency (medium effort)
+
+### Nice to Have (Long-term improvements)
+7. Implement cache size limits with LRU eviction
+8. Add cache statistics/debugging UI
+9. Implement TTL-based invalidation after model updates
+10. Option for persistent (disk-based) cache
+
+## Current Cache Key Examples
+
+### ChatterBox (Unified, Line 50-73)
+```python
+cache_data = {
+    'text': "narrator:protected_hello",     # Includes prefix + crash protection
+    'exaggeration': 0.5,                    # NO ROUNDING
+    'temperature': 0.8,                     # NO ROUNDING
+    'cfg_weight': 0.5,                      # NO ROUNDING
+    'seed': 42,
+    'audio_component': 'file_audio_abc_24k',
+    'model_source': 'chatterbox_english',
+    'device': 'auto',                       # NOT RESOLVED
+    'language': 'English',
+    'character': 'narrator',
+    'repetition_penalty': 1.0,              # NO ROUNDING
+    'min_p': 0.0,                           # NO ROUNDING
+    'top_p': 1.0,                           # NO ROUNDING
+    'engine': 'chatterbox'
+}
+```
+
+### F5-TTS (Unified, Line 25-47)
+```python
+cache_data = {
+    'text': "narrator:hello",
+    'model_name': 'F5TTS_Base',
+    'device': 'auto',                       # NOT RESOLVED
+    'audio_component': 'file_audio_def_24k',
+    'ref_text': 'reference text here',      # Correct: different ref = different voice
+    'temperature': 0.8,                     # NO ROUNDING
+    'speed': 1.0,                           # WRONG: post-processing
+    'target_rms': 0.1,                      # WRONG: post-processing
+    'cross_fade_duration': 0.15,            # WRONG: post-processing
+    'nfe_step': 32,
+    'cfg_strength': 2.0,                    # NO ROUNDING
+    'seed': 0,
+    'character': 'narrator',
+    'engine': 'f5tts'
+}
+```
+
+### ChatterBox SRT Legacy (Line 361-379)
+Uses separate cache dict, different engine name ('chatterbox_srt' vs 'chatterbox'),
+different parameters → cache misses when switching nodes!
+
+## File References
+
+All specific issues traced to source:
+
+**Unified Cache Core:**
+- `/home/linux/ComfyUI_TTS_Audio_Suite/utils/audio/cache.py` (339 lines)
+  - ChatterBoxCacheKeyGenerator (50-73)
+  - F5TTSCacheKeyGenerator (25-47)
+  - VibeVoiceCacheKeyGenerator (144-181)
+  - IndexTTSCacheKeyGenerator (184-232)
+  - AudioCache class (235-336)
+
+**Audio Hashing:**
+- `/home/linux/ComfyUI_TTS_Audio_Suite/utils/audio/audio_hash.py` (142 lines)
+  - generate_stable_audio_component (24-126)
+
+**Node Implementation:**
+- `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/chatterbox/chatterbox_tts_node.py`
+  - _pad_short_text_for_chatterbox (136-177) - text transformation
+  - _generate_tts_with_pause_tags (408-546) - text preprocessing
+  - Crash protection applied line 449 AFTER cache key generation
+
+- `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/chatterbox/chatterbox_srt_node.py`
+  - _generate_segment_cache_key (361-379) - separate cache logic
+
+- `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/f5tts/f5tts_srt_node.py`
+  - GLOBAL_AUDIO_CACHE (43) - duplicate dict
+
+- `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/unified/tts_text_node.py`
+  - _create_proper_engine_node_instance (136-441) - engine caching
+  - GLOBAL_AUDIO_CACHE (44) - local storage
+
+## Conclusion
+
+The generation cache system has **good intentions but fragmented execution**. The unified cache exists but is underutilized; legacy implementations use separate dicts; engine caching is conflated with generation caching.
+
+**Key insight:** The system needs **consolidation and normalization**, not complete redesign. All the pieces are there, they just need to work together consistently.
+
+See CACHE_ANALYSIS.md for detailed technical breakdown with line numbers and code examples.

docs/Dev reports/CLAUDE_VERSION_MANAGEMENT_GUIDE.md

@@ -1,140 +0,0 @@
-# Claude Version Management Guide
-
-**This file contains detailed instructions for Claude Code when performing version bumps.**
-
-## Automated Version Bumping Instructions
-
-### Primary Command
-**NEW DEFAULT**: `python3 scripts/bump_version_enhanced.py <version> "<commit_desc>" "<changelog_desc>"`
-
-**Auto-increment support**: Use `patch`, `minor`, or `major` instead of version numbers for automatic increment.
-
-### Usage Examples
-
-#### NEW Default Usage (RECOMMENDED)
-```bash
-# Auto-increment with separate descriptions
-python3 scripts/bump_version_enhanced.py patch "Fix character tag removal bug in single character mode" "Fix unrecognized character tags not being removed from TTS output"
-
-python3 scripts/bump_version_enhanced.py minor "Add RVC TTS engine support" "Add new RVC (Real-time Voice Conversion) TTS engine with voice cloning capabilities"
-
-# Manual version with separate descriptions  
-python3 scripts/bump_version_enhanced.py 3.4.3 "Fix crash in audio processing" "Fix crashes when processing certain audio file formats"
-```
-
-#### Legacy Usage (DEPRECATED)
-```bash
-python3 scripts/bump_version_enhanced.py patch --legacy "Fix character tag removal bug"
-```
-
-#### Multiline Description
-```bash
-python3 scripts/bump_version_enhanced.py 3.0.2 "Fix missing sounddevice dependency\nAdd comprehensive error handling\nUpdate installation documentation"
-```
-
-#### Interactive Mode (for detailed changelog entries)
-```bash
-python3 scripts/bump_version_enhanced.py 3.0.2 --interactive
-```
-
-#### From File
-```bash
-python3 scripts/bump_version_enhanced.py 3.0.2 --file changelog.txt
-```
-
-#### Dry Run First (RECOMMENDED)
-```bash
-python3 scripts/bump_version_enhanced.py 3.0.2 "Description" --dry-run
-```
-
-### When to Bump Versions
-- **Patch (x.x.X)**: Bug fixes, dependency updates, small tweaks
-- **Minor (x.X.x)**: New features, significant enhancements  
-- **Major (X.x.x)**: Breaking changes, major architecture changes
-
-### Files Auto-Updated by Script
-- README.md (title version)
-- nodes.py (VERSION constant)
-- pyproject.toml (version field)
-- srt/__init__.py (__version__)
-- core/__init__.py (__version__)
-- CHANGELOG.md (new entry with date)
-
-### Script Features
-- **Automatic categorization**: Sorts changelog entries into Added/Fixed/Changed/Removed
-- **Multiline commit messages**: Properly formatted for git
-- **Rollback protection**: Restores files if anything fails
-- **Version validation**: Ensures semantic versioning format
-- **Backup system**: Creates backup before making changes
-
-### Changelog Entry Categories (IMPROVED v3.4.0+)
-The script automatically categorizes based on enhanced keywords and context:
-- **Added**: add, new, implement, feature, create, introduce, support, language switching, syntax, bracket, character, integration
-- **Fixed**: fix, bug, error, issue, resolve, correct, patch
-- **Changed**: update, enhance, improve, change, modify, optimize, performance, smart, efficient, reduced, eliminated, loading
-- **Removed**: remove, delete, deprecate, drop
-
-**Smart Context Detection**:
-- Recognizes emoji section headers (🌍, 📋, 🚀, 🔧, etc.)
-- Uses section context for ambiguous items
-- Defaults to "Added" for feature releases instead of "Fixed"
-- Skips main title line to avoid categorization errors
-
-### Example Multiline Changelog Output
-```markdown
-## [3.0.2] - 2025-07-14
-
-### Fixed
-- Fix missing sounddevice dependency
-- Add comprehensive error handling
-
-### Changed
-- Update installation documentation
-- Simplify installation process by including all dependencies by default
-```
-
-### Fallback Process
-If the automated script fails, refer to: `docs/Dev reports/VERSION_UPDATE_GUIDE.md`
-
-### Important Notes
-- **Never manually update version files** - the script handles all 6 files automatically
-- **Script auto-commits** with proper multiline message format
-- **Always dry run first** to preview changes
-- **Use interactive mode** for complex changelog entries
-- **The script validates** version format and provides backup/rollback
-- **Commit messages**: Keep them clean and descriptive. NEVER add Claude co-author credits or "Generated with Claude Code" footers
-
-### CRITICAL: Fix Poor Changelog Generation
-
-**Problem**: Script generates detailed commit messages but poorly categorized changelog entries.
-
-**Root Cause**: The categorization algorithm has flawed keyword matching that can misclassify bug fixes as "Added" items.
-
-**Solutions**:
-
-1. **Start descriptions with clear action words** for better categorization:
-```bash
-# Good - starts with "Fix" for proper categorization
-python3 scripts/bump_version_enhanced.py patch "Fix character tag removal bug in single character mode\nRoot cause: TTS nodes bypassed character parser\nResult: Unrecognized character tags are now properly removed"
-
-# Bad - ambiguous wording leads to miscategorization  
-python3 scripts/bump_version_enhanced.py patch "Character tag removal bug in single character mode\nRoot cause: TTS nodes bypassed character parser\nResult: Character tags are now properly removed"
-```
-
-2. **Use separate commit and changelog descriptions** for precise control:
-```bash
-python3 scripts/bump_version_enhanced.py patch --commit "Fix character tag removal bug" --changelog "Fix character tags not being removed in single character mode"
-```
-
-**For complex changes, use interactive mode**:
-```bash
-python3 scripts/bump_version_enhanced.py 3.2.8 --interactive
-```
-
-This feeds the script proper technical details that get automatically categorized into good changelog entries.
-
-### IMPORTANT: v3.4.0 Script Improvements
-
-**Fixed Issues**: Enhanced multiline parsing with better keyword matching and context detection.
-
-**Always verify changelog output** - if categorization is wrong, manually fix CHANGELOG.md before pushing.

docs/Dev reports/COMFYUI_REGRESSION_FINDINGS.md

@@ -0,0 +1,67 @@
+# ComfyUI VRAM Regression Analysis - Python 3.12
+
+## Summary
+ComfyUI versions after v0.3.48 introduce severe VRAM spikes specifically in Python 3.12 environments when using TTS Audio Suite. The issue affects model memory management and causes significant performance degradation.
+
+## Affected Versions
+- **Last Working**: ComfyUI v0.3.56 + ComfyUI-frontend v1.25.11
+- **First Broken**: ComfyUI v0.3.57 + ComfyUI-frontend v1.25.11
+- **Environment**: Python 3.12 only (Python 3.13 unaffected)
+- **Root Cause**: Commit e2d1e5da "Enable Convolution AutoTuning" enables `torch.backends.cudnn.benchmark = True`
+
+## Symptoms
+- Massive VRAM spikes (multi-GB) occurring BEFORE and AFTER TTS inference, not during actual generation
+- Spikes happen during model loading and audio processing/cleanup phases
+- Affects all TTS engines: ChatterBox, ChatterBox 23-lang, IndexTTS-2
+- Performance severely degraded on 24GB VRAM systems
+- Issue does NOT occur during the TTS inference itself
+
+## Testing Results
+
+### ComfyUI v0.3.48 (Working)
+```
+Environment: Python 3.12
+TTS Audio Suite: v4.10.8 (latest)
+Result: Flat VRAM usage, no spikes
+All engines tested: ✅ ChatterBox, ✅ ChatterBox 23-lang, ✅ IndexTTS-2
+```
+
+### ComfyUI v0.3.59 (Broken)
+```
+Environment: Python 3.12
+TTS Audio Suite: v4.10.8 (latest)
+Result: Severe VRAM spikes after inference
+All engines affected: ❌ ChatterBox, ❌ ChatterBox 23-lang, ❌ IndexTTS-2
+```
+
+## Root Cause Analysis
+**Specific Commit**: e2d1e5da "Enable Convolution AutoTuning" by contentis <lspindler@nvidia.com>
+
+The commit enables `torch.backends.cudnn.benchmark = True` when `--fast autotune` is used. This CUDNN benchmarking:
+1. Tests multiple convolution algorithms to find the fastest
+2. **Allocates significant extra VRAM during algorithm benchmarking**
+3. Appears to interact poorly with Python 3.12's garbage collection
+4. Causes VRAM spikes during model operations, especially with TTS model wrappers
+
+## Reproduction Steps
+1. Install Python 3.12 environment
+2. Install ComfyUI v0.3.59
+3. Install TTS Audio Suite v4.10.8
+4. Load any TTS engine workflow
+5. Generate audio
+6. Observe VRAM spikes before and after generation (not during the actual TTS inference)
+
+## Workaround
+Downgrade to ComfyUI v0.3.48 + ComfyUI-frontend v1.2.65
+
+## Next Steps
+1. Systematically test ComfyUI versions between v0.3.48 and v0.3.59
+2. Identify exact commit that introduced the regression
+3. Create detailed GitHub issue for ComfyUI project
+4. Provide minimal reproduction case
+
+## Technical Details
+- Issue is environment-specific (Python 3.12 only)
+- Affects ComfyUI's model wrapper system
+- Related to PyTorch CUDA memory management integration
+- No code changes needed in TTS Audio Suite - issue is upstream in ComfyUI

docs/Dev reports/HIGGS_AUDIO_2_INTEGRATION_SPECIFICATION.md

@@ -0,0 +1,426 @@
+# Higgs Audio 2 Integration Specification
+
+**TTS Audio Suite v4.4.0 - Technical Documentation**
+
+*Complete technical specification for Higgs Audio 2 engine integration with unified architecture*
+
+---
+
+## Overview
+
+The Higgs Audio 2 integration adds a new high-quality TTS engine to the TTS Audio Suite with advanced voice cloning capabilities. This integration follows the established unified architecture patterns while introducing minimal dependencies and maintaining full backward compatibility.
+
+### Key Features
+
+- **High-Quality TTS Generation**: 3B parameter model with advanced audio synthesis
+- **Voice Cloning**: Clone any voice from 30+ seconds of reference audio
+- **Multi-Language Support**: English, Chinese dialects, and extensible to other languages
+- **Unified Architecture**: Seamless integration with existing TTS Text and TTS SRT nodes
+- **Chunking Support**: Handle unlimited text length with intelligent chunking
+- **Preset Voices**: 8 built-in voice presets for instant use
+- **Caching System**: Integrated with existing audio cache for performance
+- **Minimal Dependencies**: Only 2 essential new dependencies added
+
+---
+
+## Architecture Design
+
+### Modular Components (500-600 lines each)
+
+The Higgs Audio 2 integration follows the established modular architecture principle:
+
+```
+engines/higgs_audio/
+├── higgs_audio.py              # Main engine wrapper (688 lines)
+├── higgs_audio_downloader.py   # Model downloader (357 lines)
+├── boson_multimodal/           # Complete reference implementation
+└── __init__.py
+
+engines/adapters/
+└── higgs_audio_adapter.py      # Unified adapter (363 lines)
+
+nodes/engines/
+└── higgs_audio_engine_node.py  # Configuration node (153 lines)
+
+voices_examples/higgs_audio/
+├── config.json                 # Voice preset configuration
+├── belinda.wav                 # Voice presets (8 total)
+├── en_woman.wav
+└── ...
+```
+
+### Integration Points
+
+1. **Engine Configuration**: `HiggsAudioEngineNode` provides user interface
+2. **Engine Wrapper**: `HiggsAudioEngine` handles model loading and generation
+3. **Unified Adapter**: `HiggsAudioEngineAdapter` bridges to unified nodes
+4. **Unified Nodes**: Existing `TTS Text` and `TTS SRT` nodes support higgs_audio
+5. **Model Management**: Integrated with `unified_downloader` system
+6. **Caching**: Uses existing `AudioCache` with custom key generator
+
+---
+
+## Technical Implementation Details
+
+### Dependencies
+
+**Minimal Essential Dependencies Added:**
+- `descript-audio-codec`: Audio codec for Higgs Audio tokenizer
+- `vector_quantize_pytorch`: Vector quantization for audio processing
+
+**Avoided Dependencies** (handled internally or replaced):
+- `dacite` → Direct dict handling
+- `boto3`, `s3fs` → Models from HuggingFace, not S3
+- `json_repair` → Python's standard `json` module
+- `pydantic` → Standard Python dataclasses
+- `pydub` → Existing `librosa`/`torchaudio`
+- `jieba`, `langid` → Optional Chinese text processing
+- `accelerate` → Optional optimization
+
+### Text Chunking Strategy
+
+**Unified Chunking System:**
+```python
+# Uses superior ImprovedChatterBoxChunker instead of custom implementation
+max_chars = tokens_to_chars(max_tokens_per_chunk)  # Convert token limit
+chunks = ImprovedChatterBoxChunker.split_into_chunks(text, max_chars)
+```
+
+**Benefits:**
+- Character-based chunking (more predictable than token approximation)
+- Better sentence boundary detection
+- Consistent across all engines
+- Overlap support for context preservation
+
+### Voice Cloning Implementation
+
+**Voice Priority System:**
+1. **`auto`**: Use preset if selected, otherwise reference audio
+2. **`preset_dropdown`**: Prefer preset, fallback to reference
+3. **`reference_input`**: Prefer reference, fallback to preset
+4. **`force_preset`**: Only use preset, ignore reference
+
+**Voice Processing:**
+```python
+# Convert ComfyUI audio to base64 for boson_multimodal
+audio_base64 = self._audio_to_base64(reference_audio)
+audio_content = AudioContent(raw_audio=audio_base64, audio_url="")
+messages.append(Message(role="assistant", content=[audio_content]))
+```
+
+### Model Management
+
+**Organized Download Structure:**
+```
+ComfyUI/models/TTS/HiggsAudio/
+└── higgs-audio-v2-3B/
+    ├── generation/
+    │   ├── config.json
+    │   ├── model.safetensors
+    │   └── generation_config.json
+    └── tokenizer/
+        ├── config.json
+        └── pytorch_model.bin
+```
+
+**Auto-Download System:**
+- Integrated with `unified_downloader`
+- Direct downloads (no HuggingFace cache duplication)
+- Organized folder structure
+- Model reuse across sessions
+
+### Caching System
+
+**HiggsAudioCacheKeyGenerator:**
+```python
+cache_data = {
+    'text': text,
+    'voice_preset': voice_preset,
+    'model_path': model_path,
+    'tokenizer_path': tokenizer_path,
+    'temperature': temperature,
+    'top_p': top_p,
+    'top_k': top_k,
+    'reference_audio_hash': audio_hash,  # If present
+    'engine': 'higgs_audio'
+}
+```
+
+**Cache Integration:**
+- Registered with `AudioCache.register_cache_key_generator()`
+- Automatic cache key generation
+- Reference audio hashing for consistency
+- Shared with existing cache system
+
+---
+
+## Node Integration Details
+
+### Engine Configuration Node
+
+**`HiggsAudioEngineNode` Parameters:**
+- `model`: Model selection (higgs-audio-v2-3B, local models)
+- `device`: Device selection (auto, cuda, cpu)
+- `voice_preset`: Voice preset selection (8 presets + voice_clone)
+- `audio_priority`: Voice source priority logic
+- `system_prompt`: Generation guidance prompt
+- `temperature`: Sampling temperature (0.0-2.0)
+- `top_p`: Nucleus sampling (0.1-1.0)
+- `top_k`: Top-k sampling (-1 to 100)
+- `max_new_tokens`: Generation limit (128-4096)
+- `max_tokens_per_chunk`: Chunking limit (100-500)
+- `silence_between_chunks_ms`: Inter-chunk silence (0-2000ms)
+- `enable_chunking`: Chunking toggle
+- `enable_cache`: Cache toggle
+- `seed`: Reproducibility seed
+
+### Unified Node Integration
+
+**TTS Text Node Integration:**
+```python
+elif engine_type == "higgs_audio":
+    # Create wrapper with adapter
+    result = engine_instance.generate_tts_audio(
+        text=text,
+        char_audio=reference_audio_dict,
+        char_text=reference_text,
+        character="narrator",
+        **config_parameters
+    )
+```
+
+**TTS SRT Node Integration:**
+```python
+elif engine_type == "higgs_audio":
+    # Parse SRT and generate per segment
+    srt_segments = parse_srt(srt_content)
+    for segment in srt_segments:
+        segment_audio = engine_instance.generate_srt_audio(...)
+        audio_segments.append(segment_audio)
+    # Combine with timing logic
+```
+
+---
+
+## Voice Preset System
+
+### Built-in Voice Presets
+
+1. **belinda**: Female, clear English pronunciation
+2. **en_woman**: Female, natural English voice
+3. **en_man**: Male, clear English voice
+4. **chadwick**: Male, distinctive character voice
+5. **vex**: Female, character voice with edge
+6. **mabel**: Female, warm conversational voice
+7. **broom_salesman**: Male, character voice
+8. **zh_man_sichuan**: Male, Sichuan Chinese dialect
+
+### Voice Configuration
+
+**`voices_examples/higgs_audio/config.json`:**
+```json
+{
+  "belinda": {
+    "transcript": "This is Belinda's voice with clear pronunciation and natural intonation."
+  },
+  "en_woman": {
+    "transcript": "A natural English woman's voice for general purpose use."
+  }
+}
+```
+
+**Voice Discovery:**
+- Automatic preset loading from `config.json`
+- Audio file validation (`.wav` format)
+- Integration with Character Voices system
+- Custom reference audio support
+
+---
+
+## Performance Characteristics
+
+### Model Specifications
+
+- **Model Size**: ~3B parameters
+- **Memory Requirements**: 8GB+ VRAM recommended
+- **Sample Rate**: 24,000 Hz
+- **Generation Speed**: Similar to F5-TTS with chunking
+- **Voice Quality**: High-fidelity with natural prosody
+
+### Chunking Performance
+
+**Recommended Settings:**
+- `max_tokens_per_chunk`: 225 tokens (~787 characters)
+- `silence_between_chunks_ms`: 100ms
+- `enable_chunking`: True for texts >225 tokens
+
+**Benefits:**
+- Unlimited text length support
+- Consistent voice quality across chunks
+- Natural pauses between segments
+- Memory efficiency for long texts
+
+### Caching Benefits
+
+- **Cache Hits**: Instant audio retrieval
+- **Cache Misses**: Full generation with caching
+- **Storage**: Efficient tensor storage
+- **Cleanup**: Automatic cache management
+
+---
+
+## Error Handling and Validation
+
+### Parameter Validation
+
+**HiggsAudioEngineAdapter.validate_parameters():**
+```python
+# Temperature validation
+temperature = max(0.0, min(2.0, float(temperature)))
+
+# Top-p validation  
+top_p = max(0.1, min(1.0, float(top_p)))
+
+# Voice preset validation
+if voice_preset not in available_presets:
+    validated["voice_preset"] = "voice_clone"
+```
+
+### Error Recovery
+
+1. **Model Loading Failures**: Clear error messages with resolution steps
+2. **Audio Processing Errors**: Graceful fallback to basic TTS
+3. **Reference Audio Issues**: Automatic fallback to preset voices
+4. **Memory Issues**: Chunking and cache cleanup
+5. **Network Issues**: Retry logic for model downloads
+
+### Logging and Diagnostics
+
+```python
+print(f"🚀 Loading Higgs Audio 2 engine: {model_path}")
+print(f"✅ Higgs Audio 2 engine loaded successfully")
+print(f"🎤 Generating Higgs Audio for {len(chunks)} chunk(s)")
+print(f"  ✅ Generated audio shape: {audio_tensor.shape}")
+```
+
+---
+
+## Compatibility and Migration
+
+### Backward Compatibility
+
+- **Existing Workflows**: No changes required
+- **Character Voices**: Full integration with existing system
+- **Cache System**: Shared cache across all engines
+- **Audio Processing**: Standard ComfyUI audio format
+
+### Forward Compatibility
+
+- **New Engines**: Adapter pattern supports easy addition
+- **Feature Extensions**: Modular design allows enhancement
+- **Model Updates**: Download system supports model versioning
+- **Voice Expansion**: Preset system supports additional voices
+
+### Migration Notes
+
+**From Direct Higgs Audio Nodes:**
+- Use Higgs Audio Engine + TTS Text instead of separate nodes
+- Voice presets automatically detected
+- All parameters preserved with validation
+
+**From Other TTS Engines:**
+- Same unified interface
+- Voice switching supported
+- Cache keys remain unique per engine
+
+---
+
+## Testing and Validation
+
+### Manual Testing Approach
+
+1. **Model Loading**: Verify automatic download and caching
+2. **Voice Presets**: Test all 8 built-in presets
+3. **Voice Cloning**: Test with various reference audio lengths
+4. **Text Chunking**: Test with short and very long texts
+5. **Audio Quality**: Compare with reference implementation
+6. **Error Handling**: Test edge cases and failure modes
+
+### Integration Testing
+
+1. **Unified Nodes**: Verify seamless engine switching
+2. **Character Voices**: Test with Character Voices node
+3. **SRT Processing**: Test with subtitle timing
+4. **Cache System**: Verify cache hits and misses
+5. **Memory Management**: Test cleanup and resource usage
+
+### Performance Benchmarks
+
+**Text Processing:**
+- Short text (50 chars): ~2-3 seconds
+- Medium text (500 chars): ~5-8 seconds  
+- Long text (2000+ chars): Chunked processing
+
+**Memory Usage:**
+- Base model: ~6GB VRAM
+- Peak generation: ~8GB VRAM
+- Cache overhead: ~100MB per hour of audio
+
+---
+
+## Future Enhancements
+
+### Planned Features
+
+1. **Streaming Support**: Real-time generation for long texts
+2. **Voice Training**: Custom voice preset creation
+3. **Multi-Speaker**: Support for conversation-style generation
+4. **Advanced Editing**: Word-level editing like F5-TTS
+5. **Emotion Control**: Emotional tone parameters
+
+### Extension Points
+
+1. **New Voice Presets**: Easy addition through config.json
+2. **Language Models**: Support for additional languages
+3. **Audio Effects**: Post-processing pipeline integration
+4. **Quality Settings**: Speed vs quality trade-offs
+5. **Batch Processing**: Multiple texts with different voices
+
+---
+
+## Credits and Acknowledgments
+
+### Original Implementation
+- **HiggsAudio Team**: Original Higgs Audio v2 model and boson_multimodal implementation
+- **ShmuelRonen**: ComfyUI-HiggsAudio_2 reference implementation
+- **Anthropic**: Claude Code development assistance
+
+### Integration Team
+- **TTS Audio Suite**: Unified architecture integration
+- **Community Contributors**: Voice presets and testing
+
+### Third-Party Libraries
+- **descript-audio-codec**: Audio tokenization
+- **vector_quantize_pytorch**: Vector quantization
+- **PyTorch**: Deep learning framework
+- **torchaudio**: Audio processing
+
+---
+
+## Conclusion
+
+The Higgs Audio 2 integration successfully adds high-quality voice cloning capabilities to the TTS Audio Suite while maintaining the unified architecture principles. The implementation:
+
+✅ **Follows Established Patterns**: Uses existing adapter and unified node systems  
+✅ **Minimal Dependencies**: Only 2 essential new dependencies  
+✅ **Superior Chunking**: Uses unified chunking system for consistency  
+✅ **Full Integration**: Works with all existing unified nodes  
+✅ **Voice Cloning**: Supports both presets and custom reference audio  
+✅ **Performance**: Efficient caching and model management  
+✅ **Backward Compatible**: No breaking changes to existing workflows  
+
+This integration demonstrates the power of the unified architecture, allowing complex new engines to be added with minimal disruption while providing users with a consistent, powerful interface for high-quality TTS generation.
+
+---
+
+*Generated for TTS Audio Suite v4.4.0 - Higgs Audio 2 Integration*

docs/Dev reports/HIGGS_AUDIO_STATELESS_WRAPPER_IMPLEMENTATION.md

@@ -0,0 +1,293 @@
+# Higgs Audio Stateless Wrapper Implementation - Complete Journey
+
+## **Problem Statement**
+Higgs Audio 2 could not integrate with ComfyUI's native model management system due to CUDA Graph memory corruption during model unloading. The goal was to apply the same stateless wrapper pattern used for ChatterBox streaming to enable safe model unloading while preserving performance.
+
+## **Original Performance Baseline**
+- **50 it/s (tokens/second)** generation speed
+- **CUDA Graph optimization** enabled
+- **11GB VRAM usage** 
+- **Voice cloning functional**
+- **No safe model unloading** - crashes on "Unload Models"
+
+---
+
+## **Implementation Attempts & Results**
+
+### **Attempt 1: Basic Stateless Wrapper Creation**
+**Approach**: Created StatelessHiggsAudioWrapper following ChatterBox pattern
+- Created `engines/higgs_audio/stateless_wrapper.py`
+- Implemented tensor isolation with `.detach().clone()`
+- Wrapped HiggsAudioEngine with stateless methods
+
+**Result**: ❌ Circular dependency causing infinite loop
+**Fix**: Modified unified interface to create HiggsAudioServeEngine directly
+
+### **Attempt 2: Direct Serve Engine Integration**  
+**Approach**: Bypass HiggsAudioEngine.initialize_engine() to avoid circular calls
+- Modified `utils/models/unified_model_interface.py`
+- Direct HiggsAudioServeEngine creation
+- Manual component setup
+
+**Result**: ❌ TypeError: 'generator' object is not callable
+**Fix**: Changed `@property` decorator to regular method for `parameters()`
+
+### **Attempt 3: Device Mismatch Fixes - Round 1**
+**Approach**: Fix "Input type torch.FloatTensor and weight type torch.cuda.FloatTensor" errors
+- Added device enforcement in serve engine input preparation
+- Modified audio tokenizer to ensure tensors on correct device
+
+**Result**: ❌ Still device mismatches in semantic model
+**Progress**: Error moved from one component to another (good sign)
+
+### **Attempt 4: Deep Device Consistency**
+**Approach**: Comprehensive device fixing throughout audio pipeline
+- Fixed `get_regress_target()` method in audio tokenizer
+- Added `next(model.parameters()).device` for accurate device detection
+- Fixed encoder device handling in `_xcodec_encode()`
+
+**Result**: ❌ Device mismatch moved to KV cache operations
+**Lesson**: Device issues were systemic, requiring comprehensive fixes
+
+### **Attempt 5: KV Cache Device Synchronization**
+**Approach**: Fix cache_position tensor CPU/CUDA mismatch in transformers library
+- Added comprehensive KV cache device enforcement
+- Implemented cache tensor movement to model device
+- Enhanced input tensor device verification
+
+**Result**: ❌ Persistent device mismatch in cache_utils.py:1220
+**Issue**: Deep transformers library issue with cache_position tensor creation
+
+### **Attempt 6: Disable KV Cache (Temporary)**
+**Approach**: Remove StaticCache to bypass device issues
+- Set `use_cache=False`
+- Removed `past_key_values_buckets` parameter
+
+**Result**: ✅ No crashes BUT ❌ Performance degraded to ~11 it/s
+**Lesson**: StaticCache is critical for performance
+
+### **Attempt 7: Smart KV Cache Recreation**
+**Approach**: Recreate caches when device mismatch detected
+- Device mismatch detection in `_prepare_kv_caches()`
+- Dynamic cache recreation with correct device
+- Restore StaticCache functionality
+
+**Result**: ❌ Still ~12 it/s performance with occasional device issues
+
+### **Attempt 8: Performance Optimization - Remove Tensor Cloning**
+**Approach**: Optimize stateless wrapper by removing expensive operations
+- Changed `.detach().clone()` to `.detach()` only
+- Removed redundant device operations
+- Added performance caching
+
+**Result**: ❌ Still ~12 it/s performance
+**Discovery**: Cloning wasn't the main bottleneck
+
+### **Attempt 9: Bypass Stateless Wrapper Test**
+**Approach**: Temporarily disable stateless wrapper to isolate performance issue
+- Modified adapter to use original `generate()` method
+- Test if wrapper was causing slowdown
+
+**Result**: ❌ Still ~12 it/s performance even without wrapper
+**Conclusion**: Performance issue was deeper in the loading approach
+
+### **Attempt 10: Root Cause Discovery - CPU→GPU Loading**
+**Approach**: Identified that CPU-first loading was breaking CUDA Graph optimization
+- Original approach: Load directly on GPU
+- Our approach: Load on CPU → Move to GPU
+- CUDA Graphs are compiled for specific device configurations
+
+**Root Cause Found**: CPU→GPU loading breaks CUDA Graph optimization
+
+### **Attempt 11: Direct GPU Loading**
+**Approach**: Restore original loading pattern
+- Modified `unified_model_interface.py` to load directly on target device
+- Removed CPU→GPU transfer code
+- Fixed data collator device mismatch (`torch.full()` device parameter)
+
+**Result**: ✅ **54.92 tok/s performance restored!** BUT ❌ Crashes on model unload
+**Issue**: CUDA Graph warnings → Process termination
+
+### **Attempt 12: Comprehensive CUDA Graph Cleanup**
+**Approach**: Multi-layer CUDA Graph cleanup with official PyTorch APIs
+- Disabled CUDA Graph compilation during cleanup
+- Used `torch.cuda.reset_accumulated_memory_stats()`
+- Multiple CUDA synchronization passes
+- Complete KV cache tensor replacement
+
+**Result**: ❌ Still crashes with CUDA warnings
+**Issue**: CUDA Graph allocations persist at PyTorch level
+
+### **Attempt 13: Complete Engine Destruction**
+**Approach**: Nuclear option - destroy and recreate entire serve engine
+- Store model paths for recreation
+- Destroy all engine components (model, tokenizer, caches)
+- Set engine to None to force complete recreation
+- Mark for recreation with stored paths
+
+**Result**: ❌ Still crashes with identical CUDA warnings
+**Issue**: CUDA Graph captures happen at memory allocator level, persist even after object destruction
+
+### **Attempt 14: Surgical CUDA Graph Preservation (FAILED)**
+**Approach**: Keep CUDA Graphs but clean them surgically
+- Exit inference mode for tensor modifications
+- Replace cache tensors with CPU versions
+- Use official PyTorch CUDA Graph management APIs
+
+**Result**: ❌ Cache structure errors and persistent CUDA warnings
+**Issue**: Cache structure is complex (lists), CUDA Graph captures are too deep
+
+### **Attempt 15: PyTorch 2.2+ Modern CUDA Graph APIs (FAILED)**
+**Approach**: Research and implement modern PyTorch CUDA Graph cleanup APIs
+- Researched PyTorch 2.2, 2.3, 2.4 improvements
+- Implemented torch.cuda.graph() context manager approach
+- Added memory pool context managers and proper cleanup sequences
+
+**Result**: ❌ Context manager API errors and persistent CUDA warnings
+**Issue**: Modern APIs require different parameters, CUDA Graph captures persist at allocator level
+
+### **Attempt 16: CUDA Graph Toggle Implementation (PARTIAL SUCCESS)**
+**Approach**: Accept fundamental incompatibility and implement user choice toggle
+- Added `enable_cuda_graphs` boolean parameter to Higgs Audio Engine node
+- Implemented conditional loading: StaticCache vs DynamicCache
+- Added parameter passing chain: Node → Config → Adapter → Engine → Factory
+- Conditional memory unloading protection based on toggle setting
+
+**Result**: ✅ **Toggle UI working** BUT ❌ **CUDA Graphs still created in Memory Safe mode**
+**Progress**: 
+- ✅ Parameter passing chain complete
+- ✅ "Memory Safe (CUDA Graphs OFF)" vs "High Performance (CUDA Graphs ON)" modes working
+- ✅ Conditional cleanup logic implemented
+- ✅ Memory unloading protection working
+- ❌ **Critical Issue**: StaticCache still created even when CUDA Graphs disabled
+- ❌ **Still crashes**: CUDA allocation warnings persist in "safe" mode
+
+**Technical Discovery**: Environment variables set too late - serve engine constructor creates StaticCache before disable flags take effect
+
+---
+
+## **CONCLUSION: Fundamental Architectural Challenge**
+
+After **16 different implementation attempts**, the core issue remains unsolved:
+
+### **Root Cause Analysis**:
+1. **CUDA Graphs are compiled at the PyTorch C++ level** - cannot be disabled via Python APIs alone
+2. **StaticCache constructor immediately allocates CUDA Graph memory** - happens before environment variables take effect  
+3. **Memory allocator captures references during first generation** - these become "captured allocations"
+4. **ComfyUI's model unloading tries to free captured allocations** - triggers fatal CUDA warnings
+5. **Even DynamicCache approach fails** - underlying model still creates CUDA Graph optimizations
+
+### **Current Status: Toggle Implementation (Partial Success)**
+
+**✅ Successfully Implemented:**
+- **User toggle UI** with clear performance/safety tradeoffs
+- **Complete parameter passing chain** from node to factory
+- **Conditional cleanup logic** that prevents crashes when CUDA Graphs enabled
+- **Memory unloading protection** with clear user messaging
+- **Mode detection and appropriate warnings**
+
+**❌ Still Failing:**
+- **StaticCache still created** even in "Memory Safe" mode
+- **CUDA Graph allocations still captured** during generation
+- **Crashes still occur** on model unload in both modes
+- **Environment variable timing** - set after engine constructor runs
+
+### **Technical Findings:**
+
+1. **Performance Impact**: Confirmed ~40% speed reduction when CUDA Graphs disabled (55+ tok/s → ~35 tok/s)
+2. **Memory Management**: 11GB VRAM freed successfully, but CUDA state corruption persists
+3. **Architectural Conflict**: ComfyUI's dynamic model management fundamentally incompatible with CUDA Graph memory capture
+4. **PyTorch Limitation**: No clean API to disable CUDA Graphs after model initialization
+
+### **Final Assessment**: 
+
+The toggle provides **user choice** but both modes have limitations:
+- **High Performance Mode**: Full speed, but model cannot be unloaded (restart required)
+- **Memory Safe Mode**: Still crashes due to architectural conflicts (implementation incomplete)
+
+**Recommendation**: Document current state as "partial solution" - users can choose performance vs theoretical memory safety, understanding both have constraints.
+
+---
+
+## **Key Technical Insights Discovered**
+
+### **1. CUDA Graph Optimization Requirements**
+- Must load directly on target GPU device
+- CPU→GPU transfer breaks CUDA Graph compilation
+- StaticCache is essential for performance (50+ it/s vs 12 it/s)
+
+### **2. Device Consistency Challenges**
+- Device mismatches occur at multiple pipeline stages:
+  - Audio tokenizer input processing
+  - Semantic model tensor operations  
+  - Encoder operations
+  - KV cache tensor operations
+  - Data collator tensor concatenation
+- Requires systematic fixes throughout entire pipeline
+
+### **3. Memory Management Conflicts**
+- CUDA Graphs capture memory allocations during generation
+- Model unloading tries to free "captured" allocations
+- Results in warnings → crashes
+- Requires explicit CUDA Graph cleanup before unloading
+
+### **4. Performance Bottlenecks Identified**
+- **Primary**: CPU→GPU loading approach (54 it/s → 12 it/s impact)
+- **Secondary**: Tensor cloning operations (moderate impact)
+- **Tertiary**: Excessive device synchronization (minor impact)
+
+---
+
+## **Files Modified**
+
+### **Core Implementation**
+- `engines/higgs_audio/stateless_wrapper.py` - Main stateless wrapper
+- `utils/models/unified_model_interface.py` - Factory integration
+- `engines/adapters/higgs_audio_adapter.py` - Adapter integration
+- `utils/models/comfyui_model_wrapper.py` - ComfyUI integration
+
+### **Device Consistency Fixes**
+- `engines/higgs_audio/boson_multimodal/serve/serve_engine.py` - Input/cache device handling
+- `engines/higgs_audio/boson_multimodal/audio_processing/higgs_audio_tokenizer.py` - Tokenizer device handling
+- `engines/higgs_audio/boson_multimodal/data_collator/higgs_audio_collator.py` - Collator device handling
+
+---
+
+## **Current Status (As of Final Context)**
+
+### **✅ Confirmed Working**
+- **Voice cloning functionality** - Full feature preservation
+- **Memory management** - 11GB VRAM freed successfully
+- **Device consistency** - No device mismatch crashes
+- **High performance potential** - 54.92 tok/s achieved
+
+### **❌ Outstanding Issues**
+- **CUDA Graph cleanup crashes** - Hybrid solution untested
+- **Performance vs Safety tradeoff** - Need both simultaneously
+
+### **🔬 Untested Solution**
+- **Hybrid approach** with comprehensive CUDA Graph cleanup
+- **Expected outcome**: 54+ it/s performance + no crashes + safe unloading
+- **Status**: Implementation complete, testing required
+
+---
+
+## **Recommended Next Steps**
+
+1. **Test hybrid solution** - Verify 54 it/s + no crashes
+2. **If crashes persist**: Investigate alternative CUDA Graph cleanup methods
+3. **Performance monitoring**: Ensure no regression from cleanup operations
+4. **Stress testing**: Multiple load/unload cycles
+5. **Documentation update**: Record final working solution
+
+---
+
+## **Lessons Learned**
+
+1. **Device loading approach is critical** for CUDA Graph optimization
+2. **Systematic device consistency** required throughout pipeline
+3. **CUDA Graph cleanup** must be explicit and comprehensive
+4. **Performance vs Safety** requires sophisticated balancing, not simple tradeoffs
+5. **Deep integration challenges** require understanding of underlying frameworks (transformers, CUDA)
+
+This implementation represents a complex balance between PyTorch CUDA Graph optimization, device management, and ComfyUI model lifecycle integration.

docs/Dev reports/MODEL_LOADING_ANALYSIS.md

@@ -0,0 +1,633 @@
+# TTS Audio Suite Model Loading Architecture Analysis
+
+## EXECUTIVE SUMMARY
+
+The codebase shows a **two-tiered model loading architecture**:
+1. **Unified Model Interface Layer** - Relatively new, factory-pattern based, ComfyUI-integrated
+2. **Legacy/Fallback Systems** - Multiple engine-specific loaders still in use, creating redundancy
+
+**Critical Finding**: The unified interface exists but is **NOT consistently adopted**. Some engines use it fully, others have dual implementations, creating inconsistent patterns and potential device/cache conflicts.
+
+---
+
+## 1. UNIFIED INTERFACE USAGE & ADOPTION RATE
+
+### Current Status
+**File**: `/home/linux/ComfyUI_TTS_Audio_Suite/utils/models/unified_model_interface.py` (817 lines)
+
+### Adoption by Engine
+
+| Engine | Status | Primary Loader | Fallback | Notes |
+|--------|--------|----------------|----------|-------|
+| ChatterBox | DUAL | Unified Interface | Manager.load_tts_model() | Uses unified, but manager has legacy fallback logic (lines 252-439 in manager.py) |
+| F5-TTS | DUAL | Unified Interface | SmartModelLoader + direct F5TTS calls | Hybrid approach with multiple fallback paths |
+| ChatterBox Official 23-Lang | UNIFIED | Unified Interface | None documented | Cleanest implementation, language-aware cache key (line 216-217) |
+| Higgs Audio | UNIFIED | Unified Interface | None documented | Stateless wrapper integration (line 566) |
+| RVC | UNIFIED | Unified Interface (4 factories) | None documented | Most comprehensive (VC, hubert, separator_mdx, separator_demucs) |
+| VibeVoice | UNIFIED | Unified Interface | None documented | Engine pattern with integrated initialization |
+| IndexTTS-2 | UNIFIED | Unified Interface | None documented | Path resolution and auto-download handling |
+
+**Adoption Rate**: ~85% of new engines use unified interface, but legacy fallbacks still exist
+
+### Factory Registration Architecture
+**Location**: Lines 327-813 in unified_model_interface.py
+
+```python
+def initialize_all_factories():  # Line 805
+    register_chatterbox_factory()
+    register_chatterbox_23lang_factory()
+    register_f5tts_factory()
+    register_higgs_audio_factory()
+    register_rvc_factory()
+    register_vibevoice_factory()
+    register_index_tts_factory()
+```
+
+Each factory is registered with key format: `"{engine_name}_{model_type}"` (line 55)
+
+---
+
+## 2. FACTORY PATTERN CONSISTENCY ANALYSIS
+
+### Issue 1: Inconsistent Error Handling
+
+**ChatterBox Factory** (lines 401-444)
+```python
+def chatterbox_tts_factory(**kwargs):
+    if ChatterboxTTS is None:
+        raise RuntimeError("ChatterboxTTS not available - check installation")
+    # Has fallback mixing logic
+```
+
+**F5TTS Factory** (lines 449-497)
+```python
+def f5tts_factory(**kwargs):
+    # Uses smart_model_loader.load_model_if_needed()
+    # No inline error handling - delegates to SmartModelLoader
+```
+
+**VibeVoice Factory** (lines 623-657)
+```python
+def vibevoice_factory(**kwargs):
+    try:
+        import accelerate
+    except ImportError as e:
+        print(f"⚠️ Unified Interface: accelerate not available: {e}")
+    # Continues without raising - soft failure pattern
+```
+
+**Problem**: Three different error handling patterns:
+1. ChatterBox: Raises RuntimeError for missing imports
+2. F5TTS: Delegates to SmartModelLoader (different exception path)
+3. VibeVoice: Prints warning and continues (no exception)
+
+### Issue 2: Inconsistent Signatures
+
+| Factory | Signature Pattern | Special Handling |
+|---------|------------------|------------------|
+| ChatterBox | `device, language, model_path` | Multi-step fallback with German special case (lines 366-383) |
+| F5TTS | `device, model_name` | Delegates to smart_loader callback |
+| Higgs Audio | `device, model_path, tokenizer_path, enable_cuda_graphs` | CUDA-specific parameters |
+| RVC | `model_path, index_path, device` | Multiple model types (VC, hubert, separators) |
+| IndexTTS-2 | `device, use_fp16, use_cuda_kernel, use_deepspeed` | Optimization flags |
+
+**Problem**: Each factory has different parameter semantics and handling logic.
+
+### Issue 3: Device Resolution Inconsistencies
+
+**Unified Interface Approach** (line 141):
+```python
+from utils.device import resolve_torch_device
+target_device = resolve_torch_device(config.device)
+```
+
+**Factory-Level Device Resolution**:
+- **ChatterBox** (line 410): `device = kwargs.get("device", "auto")`
+  - Passed directly to ChatterboxTTS.from_local/from_pretrained
+  - Device resolution happens IN the engine, not in unified interface
+  
+- **F5TTS** (line 453): `device = kwargs.get("device", "auto")`
+  - Passed to smart_model_loader
+  - Then to ChatterBoxF5TTS.from_local/from_pretrained
+  - **Two levels of resolution possible**
+
+- **Higgs Audio** (line 508): `device = kwargs.get("device", "cuda")`
+  - Default is "cuda" not "auto"
+  - **Inconsistent default**
+
+- **VibeVoice** (line 637): `device = kwargs.get("device", "auto")`
+
+**Problem**: Device resolution happens at MULTIPLE levels, with inconsistent defaults and no guarantee of final device being correct.
+
+---
+
+## 3. MODEL FILE LOADING PATTERNS - ALL INSTANCES
+
+### Load Functions in Codebase
+
+| File | Function | Pattern | Issues |
+|------|----------|---------|--------|
+| manager.py (line 226-439) | `ModelManager.load_tts_model()` | Try local → Try HuggingFace → Fallback English | Has language fallback logic |
+| manager.py (line 441-577) | `ModelManager.load_vc_model()` | Try local → Try HuggingFace | Supports language switching |
+| f5tts_manager.py (line 108-223) | `F5TTSModelManager.load_f5tts_model()` | Try unified → Fallback unified+direct | Dual system |
+| f5tts_base_node.py (line 86-192) | `BaseF5TTSNode.load_f5tts_model()` | Try unified → Fallback SmartLoader | Three-tier fallback |
+| unified_model_interface.py (line 111-178) | `UnifiedModelInterface.load_model()` | Check cache → Load via factory | Direct factory invocation |
+
+### Format Detection Logic
+
+**ChatterBox** (manager.py, lines 108-120):
+- Detects required files: `ve.`, `t3_cfg.`, `s3gen.`, `tokenizer.json`
+- Can be `.pt` or `.safetensors` extensions
+- Directory-based model location
+
+**F5TTS** (f5tts_manager.py, lines 79-83):
+- Checks for ANY `.safetensors` or `.pt` files in directory
+- Less specific detection (any model file matches)
+
+**ChatterBox Official 23-Lang** (unified_model_interface.py, lines 776-785):
+- Has explicit `from_local()` and `from_pretrained()` methods
+- Model version parameter (v1 vs v2)
+
+**Higgs Audio** (higgs_audio_engine_node.py, lines 79-105):
+- Requires subdirectories: `generation/` and `tokenizer/`
+- Checks for `config.json` and model files in both
+
+**RVC** (manager.py, no dedicated loader):
+- Model loading happens in factory only
+- Format: model_path and index_path parameters
+
+**Problem**: Each engine has DIFFERENT detection logic with NO unified format detection system. Code duplication across manager.py, f5tts_manager.py, and node discovery methods.
+
+---
+
+## 4. DEVICE MANAGEMENT ANALYSIS
+
+### Unified Interface Device Handling
+
+**Location**: unified_model_interface.py, lines 139-146
+```python
+# Check if force reload is needed
+if force_reload:
+    tts_model_manager.remove_model(cache_key)
+
+# Get existing model if cached
+wrapper = tts_model_manager.get_model(cache_key)
+if wrapper is not None:
+    # Ensure model is on correct device
+    from utils.device import resolve_torch_device
+    target_device = resolve_torch_device(config.device)
+    
+    if wrapper.current_device != target_device:
+        wrapper.model_load(target_device)
+    return wrapper.model
+```
+
+**Problem 1**: Device comparison uses string comparison (`wrapper.current_device != target_device`). If one is `"auto"` and other is resolved `"cuda"`, they won't match even though they're the same device.
+
+**ComfyUI Wrapper Device Handling**
+
+**Location**: base_wrapper.py, lines 77-87
+```python
+# Convert device to torch.device object for ComfyUI compatibility
+from utils.device import resolve_torch_device
+device_name = resolve_torch_device(model_info.device)
+
+if isinstance(device_name, str):
+    if device_name == "cuda":
+        self.device = torch.device("cuda", torch.cuda.current_device() if torch.cuda.is_available() else 0)
+    else:
+        self.device = torch.device(device_name)
+else:
+    self.device = device_name
+```
+
+**Problem 2**: Device is resolved, converted to torch.device, but wrapper stores it as torch.device object while unified interface uses strings. **Type mismatch**.
+
+### Auto Device Resolution
+
+**Location**: torch_device_resolver.py, lines 10-51
+
+```python
+def resolve_torch_device(device: str) -> str:
+    if device == "auto":
+        if torch.backends.mps.is_available():
+            return "mps"
+        elif torch.cuda.is_available():
+            return "cuda"
+        elif hasattr(torch, "xpu") and torch.xpu.is_available():
+            return "xpu"
+        else:
+            return "cpu"
+```
+
+**Problem 3**: Called in multiple places with no caching. If GPU becomes unavailable between calls (memory pressure unload), subsequent calls might return different device. **No locking or consistency guarantee**.
+
+### Device Mismatch Issues Found
+
+1. **BaseNode.resolve_device()** (base_node.py, lines 124-136):
+   ```python
+   def resolve_device(self, device: str) -> str:
+       if device == "auto":
+           return "cuda" if torch.cuda.is_available() else "cpu"
+       return device
+   ```
+   **This is DIFFERENT from torch_device_resolver.py** - doesn't check MPS or XPU!
+
+2. **Factory defaults** - Higgs Audio uses `device="cuda"` as default (line 508 unified_model_interface.py), while others use `"auto"`.
+
+3. **ComfyUI wrapper** (line 141 base_wrapper.py):
+   ```python
+   from utils.device import resolve_torch_device
+   target_device = resolve_torch_device(config.device)
+   ```
+   But then converts to torch.device object, not string comparison.
+
+---
+
+## 5. CACHING & MEMORY MANAGEMENT
+
+### Unified Interface Cache Key Generation
+
+**Location**: unified_model_interface.py, lines 205-234
+
+```python
+def _generate_cache_key(self, config: UnifiedModelConfig) -> str:
+    components = [
+        config.engine_name,
+        config.model_type,
+        config.model_name,
+        config.device,
+    ]
+    
+    # CRITICAL FIX: ChatterBox Official 23-Lang is multilingual - language shouldn't affect cache
+    if config.engine_name != "chatterbox_official_23lang":
+        components.append(config.language or "default")
+    
+    # Add path/repo info if available
+    if config.model_path:
+        components.append(f"path:{Path(config.model_path).name}")
+    elif config.repo_id:
+        components.append(f"repo:{config.repo_id}")
+    
+    # Add additional_params to ensure attention_mode, quantization, etc. trigger reloads
+    if config.additional_params:
+        sorted_params = sorted(config.additional_params.items())
+        params_str = "_".join([f"{k}:{v}" for k, v in sorted_params])
+        components.append(f"params:{params_str}")
+    
+    cache_key = "_".join(components)
+    return cache_key
+```
+
+**Issues**:
+1. **Line 211**: Device is ALWAYS included, but device can be "auto" which resolves differently on each call → **Cache key instability**
+2. **Line 216-217**: Special-case hardcoding for chatterbox_official_23lang (comment says "CRITICAL FIX") suggests cache key generation is fragile
+3. **No address hashing**: Full strings concatenated, very long cache keys possible
+4. **Path name only** (line 221): If two models named identically in different paths, cache key collision possible
+
+### ComfyUI Model Manager Caching
+
+**Location**: model_manager.py, lines 44-127
+
+```python
+def load_model(self, 
+               model_factory_func, 
+               model_key: str,
+               model_type: str,
+               engine: str, 
+               device: str,
+               force_reload: bool = False,
+               **factory_kwargs) -> ComfyUIModelWrapper:
+    
+    # Check if already cached
+    if model_key in self._model_cache and not force_reload:
+        wrapper = self._model_cache[model_key]
+        is_valid = getattr(wrapper, '_is_valid_for_reuse', True)
+        
+        if not is_valid:
+            # For Higgs Audio with CUDA graph corruption...
+            # For VibeVoice, try to recover...
+            # For IndexTTS-2, attempt recovery...
+```
+
+**Issues**:
+1. **Engine-specific recovery logic** (lines 54-117): 
+   - Higgs Audio: Reset CUDA graph state (line 137)
+   - VibeVoice: Clear internal caches (line 77)
+   - IndexTTS-2: Clear device-cached state (line 102)
+   - **Each engine needs custom recovery code** - not scalable
+
+2. **Validity flag** (line 48):
+   ```python
+   is_valid = getattr(wrapper, '_is_valid_for_reuse', True)
+   ```
+   Only Higgs Audio sets this to False (base_wrapper.py, line 302). **VibeVoice and IndexTTS-2 corrupted models won't be detected**.
+
+3. **No smart cache management** - older models not cleared to make room for new ones automatically.
+
+### SmartModelLoader Cache
+
+**Location**: smart_loader.py, lines 23-225
+
+```python
+class SmartModelLoader:
+    def __init__(self):
+        self._global_model_cache: Dict[str, ModelInfo] = {}
+        self._current_models: Dict[str, Dict[str, Any]] = {}
+    
+    def _generate_cache_key(self, engine_type: str, model_name: str, device: str) -> str:
+        key_data = f"{engine_type}_{model_name}_{device}"
+        return hashlib.md5(key_data.encode()).hexdigest()[:16]
+```
+
+**Issues**:
+1. **MD5 hashing** (line 44): Different from unified interface string concatenation
+2. **THREE separate cache systems**:
+   - UnifiedModelInterface: String concatenation
+   - ComfyUITTSModelManager: Direct dict with string keys
+   - SmartModelLoader: MD5 hashes
+   - **Models can exist in multiple caches simultaneously**
+
+3. **Cross-engine reuse** (lines 59-84):
+   ```python
+   def _check_cross_engine_cache(self, engine_type: str, model_name: str, device: str) -> Optional[Any]:
+       cache_key = self._generate_cache_key(engine_type, model_name, device)
+       if cache_key in self._global_model_cache:
+           model_info = self._global_model_cache[cache_key]
+           model_instance = model_info.model_instance
+           # CRITICAL: Ensure model is on correct device after cache reuse
+           actual_device = "auto" if device == "auto" and torch.cuda.is_available() else device
+   ```
+   Device resolution happens again here! **Multiple resolution points for "auto"**.
+
+---
+
+## 6. ERROR HANDLING PATTERNS
+
+### Missing Fallback Mechanisms
+
+**ChatterBox** (unified_model_interface.py, lines 340-398):
+- ✅ Has multi-level fallback:
+  1. Try provided path
+  2. Try local language model
+  3. Try HuggingFace language
+  4. Special fallback for German variants
+  5. Final fallback to English
+- ✅ Graceful 401 authorization error handling (line 362)
+
+**F5-TTS** (unified_model_interface.py, lines 449-497):
+- ⚠️ Delegates to SmartModelLoader - error handling unclear
+- ⚠️ No explicit fallback documented
+- ⚠️ Smart loader has callback pattern but errors propagate up
+
+**Higgs Audio** (unified_model_interface.py, lines 502-571):
+- ❌ No fallback mechanism
+- ⚠️ Sets up CUDA graphs, can fail silently if disabled
+- ⚠️ Complex initialization with multiple environment variables
+
+**RVC** (unified_model_interface.py, lines 574-618):
+- ❌ No fallback for missing models
+- ⚠️ get_vc() call at line 590 - error propagates directly
+- ❌ No graceful degradation
+
+**VibeVoice** (unified_model_interface.py, lines 621-657):
+- ⚠️ Soft failure for accelerate import (line 631)
+- ❌ No fallback if model download fails
+
+**IndexTTS-2** (unified_model_interface.py, lines 660-746):
+- ✅ Good error detection for external IndexTTS installations (lines 688-708)
+- ✅ Checks config.yaml existence (line 726)
+- ⚠️ Raises RuntimeError - no fallback
+
+### Import Error Handling
+
+**Pattern 1 - Try/Except on Import** (most factories)
+```python
+try:
+    from engines.chatterbox.tts import ChatterboxTTS
+except ImportError:
+    ChatterboxTTS = None
+```
+
+**Pattern 2 - Direct Import with No Error Handling** (RVC)
+```python
+from engines.rvc.impl.vc_infer_pipeline import get_vc  # Line 578
+from engines.rvc.impl.config import config as rvc_config  # Line 579
+```
+
+**Pattern 3 - ImportError Message** (IndexTTS)
+```python
+except ImportError as e:
+    raise ImportError(f"IndexTTS-2 dependencies not available. Error: {e}")
+```
+
+**Problem**: No consistent pattern for handling missing dependencies.
+
+---
+
+## 7. SPECIFIC PROBLEMS IDENTIFIED
+
+### Problem 1: "auto" Device Instability
+**Severity**: HIGH
+**Location**: Multiple (torch_device_resolver.py, base_node.py, smart_loader.py, unified_model_interface.py)
+
+Device resolution for "auto" happens at 4+ different places with NO caching. If CUDA becomes unavailable mid-execution, subsequent calls return CPU device while cache key still contains "auto".
+
+**Example**:
+1. Model loads on CUDA with cache key: `"chatterbox_tts_auto"`
+2. ComfyUI clears VRAM
+3. Next inference call: `resolve_torch_device("auto")` now returns "cpu"
+4. Cache lookup fails (expects "auto", not "cpu")
+5. Model reloads on CPU unnecessarily
+
+### Problem 2: Dual Unified Interface Usage
+**Severity**: MEDIUM
+**Location**: manager.py (lines 252-273), f5tts_manager.py (lines 132-145)
+
+Both ChatterBox and F5-TTS managers try unified interface FIRST, then fall back to legacy systems. Creates:
+- Inconsistent code paths
+- Debug complexity
+- Potential for models cached in one system, used in another
+
+```python
+# manager.py, line 253
+try:
+    model = load_tts_model(engine_name="chatterbox", ...)
+except Exception as e:
+    print(f"⚠️ Failed to load ChatterBox model via unified interface: {e}")
+    # Fall back to original logic if unified interface fails
+    pass
+```
+
+### Problem 3: Cache Key Collision Risk
+**Severity**: MEDIUM
+**Location**: unified_model_interface.py (lines 205-234)
+
+Cache key uses basename only for paths:
+```python
+if config.model_path:
+    components.append(f"path:{Path(config.model_path).name}")  # Line 221
+```
+
+If two different paths have identically-named model directories, cache collision occurs:
+- `/models/TTS/chatterbox/English/` 
+- `/backup/models/chatterbox/English/`
+- Both generate cache component: `path:English`
+
+### Problem 4: Device Type Mismatch
+**Severity**: MEDIUM
+**Location**: base_wrapper.py (lines 77-87), unified_model_interface.py (line 141-146)
+
+Unified interface stores device as string, ComfyUI wrapper converts to torch.device:
+```python
+# unified_model_interface.py - uses string
+target_device = resolve_torch_device(config.device)  # Returns "cuda"
+
+# base_wrapper.py - stores torch.device
+self.device = torch.device("cuda", ...)  # torch.device object
+```
+
+Later comparison:
+```python
+if wrapper.current_device != target_device:  # "cuda" != torch.device("cuda")
+```
+**Type mismatch in comparison**.
+
+### Problem 5: Engine-Specific Recovery Code Not Scalable
+**Severity**: MEDIUM
+**Location**: model_manager.py (lines 54-117)
+
+Each engine has custom recovery logic for corrupted models:
+```python
+if engine == "higgs_audio":
+    # Reset CUDA graphs
+elif engine == "vibevoice":
+    # Clear caches
+elif engine == "index_tts":
+    # Clear device state
+```
+
+When new engine added, must modify model_manager.py. No extensible pattern.
+
+### Problem 6: Three Separate Caching Systems
+**Severity**: HIGH
+**Location**: unified_model_interface.py, model_manager.py, smart_loader.py
+
+Models can be cached in:
+1. **UnifiedModelInterface**: Via ComfyUI manager
+2. **SmartModelLoader**: Global cache with MD5 keys
+3. **Manager classes**: Manual dicts (ChatterBox manager, F5-TTS manager)
+
+**No synchronization between them**. A model cleared from one cache remains in others.
+
+### Problem 7: Inconsistent Error Handling
+**Severity**: MEDIUM
+**Location**: All factories in unified_model_interface.py
+
+- ChatterBox: Raises RuntimeError (line 408)
+- F5-TTS: Delegates to SmartModelLoader
+- VibeVoice: Prints warning and continues (line 631)
+- RVC: Direct ImportError (line 578)
+- IndexTTS-2: Custom ImportError message (line 742)
+
+**No consistent error contract** - callers don't know what exceptions to expect.
+
+### Problem 8: Cache Key Instability with Language
+**Severity**: LOW
+**Location**: unified_model_interface.py (line 216)
+
+```python
+if config.engine_name != "chatterbox_official_23lang":
+    components.append(config.language or "default")
+```
+
+Special-casing suggests original cache key generation was broken. Comment says "CRITICAL FIX" but introduces inconsistency for future engines.
+
+---
+
+## 8. FACTORY SIGNATURE ANALYSIS
+
+| Parameter | ChatterBox | F5TTS | Higgs | RVC | VibeVoice | IndexTTS |
+|-----------|-----------|-------|-------|-----|-----------|----------|
+| device | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| model_name | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
+| language | ✅ | ⚠️ (ignored) | ❌ | ❌ | ❌ | ❌ |
+| model_path | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
+| repo_id | ✅ | ⚠️ (not used) | ❌ | ❌ | ❌ | ❌ |
+| additional_params | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ |
+
+**No consistent parameter contract**.
+
+---
+
+## 9. RECOMMENDATIONS FOR UNIFICATION
+
+### Quick Wins (Low Risk)
+
+1. **Fix auto device instability**:
+   - Cache resolved device in resolve_torch_device()
+   - Don't store "auto" in cache keys, only resolved device
+   
+2. **Consistent device comparison**:
+   - Resolve both sides before comparison
+   - Store device as string consistently across wrapper and interface
+
+3. **Consolidate error handling**:
+   - Create base exception class for model loading errors
+   - Standardize all factories to raise same exceptions
+
+### Medium Effort (Medium Risk)
+
+4. **Unified fallback handler**:
+   - Extract ChatterBox fallback logic to generic utility
+   - Apply to all engines consistently
+
+5. **Standardize cache keys**:
+   - Use hash of full path, not basename only
+   - Consider device as "resolved" device, not "auto"
+
+6. **Single cache system**:
+   - Deprecate SmartModelLoader and manager-specific caches
+   - Route everything through ComfyUI manager
+
+### Major Refactor (High Risk)
+
+7. **Factory interface standardization**:
+   - Define common parameter set all factories accept
+   - Use dataclass for factory parameters
+   - Implement validation layer
+
+8. **Engine-agnostic recovery**:
+   - Create engine capability registry
+   - Generic recovery pattern with hooks
+
+---
+
+## FILES REQUIRING CHANGES
+
+### Core Files
+- `/home/linux/ComfyUI_TTS_Audio_Suite/utils/models/unified_model_interface.py` - Cache key generation, factory signatures
+- `/home/linux/ComfyUI_TTS_Audio_Suite/utils/models/comfyui_model_wrapper/model_manager.py` - Engine-specific recovery logic
+- `/home/linux/ComfyUI_TTS_Audio_Suite/utils/device/torch_device_resolver.py` - Auto device caching
+- `/home/linux/ComfyUI_TTS_Audio_Suite/utils/models/manager.py` - Unified interface dual-loading
+- `/home/linux/ComfyUI_TTS_Audio_Suite/utils/models/f5tts_manager.py` - Unified interface dual-loading
+
+### Supporting Files
+- `/home/linux/ComfyUI_TTS_Audio_Suite/utils/models/smart_loader.py` - Cache integration
+- `/home/linux/ComfyUI_TTS_Audio_Suite/utils/models/comfyui_model_wrapper/base_wrapper.py` - Device type handling
+- `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/base/base_node.py` - Device resolution duplication
+- `/home/linux/ComfyUI_TTS_Audio_Suite/nodes/base/f5tts_base_node.py` - Triple-tier fallback
+
+---
+
+## SUMMARY TABLE
+
+| Aspect | Current State | Issues | Risk Level |
+|--------|---------------|--------|-----------|
+| Unified Interface Coverage | 85% of engines | Legacy fallbacks still exist | MEDIUM |
+| Factory Consistency | Inconsistent | Different error handling, parameters | HIGH |
+| Device Management | Decentralized | 4+ resolution points, type mismatch | HIGH |
+| Caching Strategy | Triple system | No synchronization | CRITICAL |
+| Fallback Mechanisms | Partial | ChatterBox has, others don't | MEDIUM |
+| Error Handling | Inconsistent | 5 different patterns | MEDIUM |
+

docs/Dev reports/MODEL_LOADING_REFACTOR_REPORT.md

@@ -0,0 +1,641 @@
+# TTS Audio Suite - Model Loading Architecture: Improvement Report
+
+**Date**: November 13, 2025
+**Status**: Comprehensive Analysis Complete
+**Audience**: Technical Maintainers / Decision Makers
+
+---
+
+## EXECUTIVE SUMMARY
+
+The TTS Audio Suite has a **functional but fragmented model loading architecture**. A unified interface exists and is being adopted (~85% coverage), but critical architectural issues create maintenance burden, cache instability, and potential device management failures.
+
+**The unified system works, but the inconsistencies around it create real problems.**
+
+### Key Finding
+| Aspect | Current State |
+|--------|---------------|
+| **System Functionality** | ✅ Works without errors |
+| **Architecture Quality** | ⚠️ Fragmented (legacy + new coexist) |
+| **Risk Level** | ⚠️ Medium - edge cases exist |
+| **Technical Debt** | 🔴 HIGH - 3 caching systems, 5 error patterns |
+| **Maintainability** | 🟡 Declining - new engines need custom code |
+
+---
+
+## PART 1: CURRENT STATE ASSESSMENT
+
+### 1.1 Unified Interface Adoption
+
+**What exists**: `utils/models/unified_model_interface.py` (817 lines) - A complete factory-based model loading system.
+
+**Adoption rate by engine**:
+```
+✅ UNIFIED (clean implementations):
+  - ChatterBox Official 23-Lang (100% using load_tts_model)
+  - Higgs Audio (stateless wrapper integration working)
+  - RVC (4 factory types: vc, hubert, separator_mdx, separator_demucs)
+  - VibeVoice (engine initialization pattern)
+  - IndexTTS-2 (path resolution + auto-download)
+
+⚠️ DUAL IMPLEMENTATIONS (fallback to legacy):
+  - ChatterBox (tries unified → falls back to manager.load_tts_model())
+  - F5-TTS (tries unified → falls back to SmartModelLoader)
+```
+
+**Reality**: 85% coverage looks good, but the 15% with fallbacks creates two code paths, making debugging harder.
+
+### 1.2 What's Working Well
+
+✅ **ComfyUI Integration**: Models are properly registered with ComfyUI's VRAM management. The wrapper system (comfyui_model_wrapper/) correctly handles model lifecycle.
+
+✅ **Factory Pattern**: Extensible design - adding new engines requires registering a factory. New engines (Higgs, RVC, VibeVoice) implemented cleanly.
+
+✅ **ChatterBox Fallback Logic**: Sophisticated fallback chain:
+```
+Try provided path → Try local language model → Try HuggingFace →
+Try German fallback → Final fallback to English
+```
+This is well-thought-out and prevents total failures.
+
+✅ **Language-Aware Caching**: ChatterBox Official 23-Lang has special case to NOT include language in cache key (multilingual model special case).
+
+✅ **Force Reload**: System properly handles force_reload flag for model reloading.
+
+---
+
+## PART 2: CRITICAL ISSUES IDENTIFIED
+
+### Issue #1: "auto" Device Instability (SEVERITY: HIGH)
+
+**Problem**: Device resolution for "auto" happens in multiple places with NO caching:
+- `torch_device_resolver.py` (resolve_torch_device)
+- `base_node.py` (resolve_device)
+- `smart_loader.py` (internal resolution)
+- `unified_model_interface.py` (factory level)
+
+**The Bug**:
+```python
+# Call 1: CUDA available
+device_key = generate_cache_key(..., device="auto")  # "auto" stored in key
+model = load_model(device="auto")  # Resolves to "cuda"
+
+# ComfyUI clears VRAM, CUDA unavailable
+
+# Call 2: CUDA unavailable
+device_key = generate_cache_key(..., device="auto")  # Still "auto"
+model = load_model(device="auto")  # Now resolves to "cpu"
+# CACHE MISS - model reloaded unnecessarily!
+```
+
+**Impact**:
+- Unnecessary model reloads during long ComfyUI sessions
+- VRAM thrashing if GPU memory pressure forces CPU fallback
+- Corrupts caching logic for "auto" device selection
+
+**Root Cause**: Cache keys use the STRING "auto" instead of resolved device name.
+
+---
+
+### Issue #2: Triple Caching System with No Sync (SEVERITY: CRITICAL)
+
+**Three separate caches exist**:
+
+1. **UnifiedModelInterface** (via ComfyUI model manager)
+   - Cache keys: `"chatterbox_tts_cuda_English_path:model"`
+   - String concatenation based
+
+2. **SmartModelLoader**
+   - Cache keys: `md5hash("chatterbox_tts_English_cuda")[:16]`
+   - MD5 hash based
+   - Used as fallback by F5-TTS and ChatterBox managers
+
+3. **Manager Classes** (manager.py, f5tts_manager.py)
+   - Manual dictionaries
+   - Different key formats per manager
+
+**The Problem**:
+```
+Model loaded → Cached in Unified → Also in SmartLoader → Also in Manager
+User clears VRAM → Only UnifiedModelInterface clears → Model still in other caches
+Next load → Stale model returned from SmartLoader cache → Device mismatch
+```
+
+**Impact**:
+- Models can be loaded multiple times while already cached
+- Cache clearing commands don't fully clear models
+- Memory leaks possible if stale models accumulate
+- Confusing behavior for users clearing VRAM between inferences
+
+---
+
+### Issue #3: Device Type Mismatch (SEVERITY: MEDIUM)
+
+**Two different device representations**:
+
+```python
+# Unified interface uses: strings
+target_device = resolve_torch_device(config.device)  # Returns "cuda"
+
+# ComfyUI wrapper uses: torch.device objects
+self.device = torch.device("cuda")  # torch.device object
+```
+
+**The comparison fails**:
+```python
+if wrapper.current_device != target_device:  # "cuda" != torch.device("cuda")
+    # Type mismatch - comparison always True even if same device!
+```
+
+**Impact**:
+- Device switch detection broken
+- Models might not move to correct device
+- Silent failures (model stays on old device)
+
+---
+
+### Issue #4: Cache Key Collision Risk (SEVERITY: MEDIUM)
+
+**Current logic** (unified_model_interface.py, line 221):
+```python
+if config.model_path:
+    components.append(f"path:{Path(config.model_path).name}")
+```
+
+Uses basename only, not full path:
+```
+/models/TTS/chatterbox/English/  → cache component: "path:English"
+/backup/models/chatterbox/English/  → cache component: "path:English"
+# COLLISION!
+```
+
+**Impact**:
+- Two different model directories with same name → same cache key
+- Randomly returns one or the other
+- Non-deterministic behavior
+- Difficult to debug
+
+---
+
+### Issue #5: Inconsistent Factory Signatures (SEVERITY: HIGH)
+
+**Parameter inconsistency**:
+
+| Factory | device | model_name | language | model_path | additional_params |
+|---------|--------|-----------|----------|-----------|------------------|
+| ChatterBox | ✅ | ✅ | ✅ | ✅ | ✅ |
+| F5-TTS | ✅ | ✅ | ⚠️ (ignored) | ✅ | ❌ |
+| Higgs Audio | ✅ | ✅ | ❌ | ✅ | ✅ |
+| RVC | ✅ | ❌ | ❌ | ✅ | ❌ |
+| VibeVoice | ✅ | ✅ | ❌ | ❌ | ✅ |
+| IndexTTS | ✅ | ❌ | ❌ | ✅ | ✅ |
+
+**Problem**: Each factory has different parameter set. Adding new parameter requires updating individual factories.
+
+**Also**: Default values inconsistent (Higgs Audio defaults device="cuda" while others use "auto").
+
+---
+
+### Issue #6: Five Different Error Handling Patterns (SEVERITY: MEDIUM)
+
+```python
+# Pattern 1: ChatterBox - Raises RuntimeError
+raise RuntimeError("ChatterboxTTS not available - check installation")
+
+# Pattern 2: F5-TTS - Delegates to SmartModelLoader (different exceptions)
+model, _ = smart_model_loader.load_model_if_needed(...)
+
+# Pattern 3: VibeVoice - Soft failure (prints, continues)
+except ImportError as e:
+    print(f"⚠️ Unified Interface: accelerate not available: {e}")
+
+# Pattern 4: RVC - Direct ImportError
+from engines.rvc.impl.vc_infer_pipeline import get_vc
+
+# Pattern 5: IndexTTS - Custom ImportError message
+raise ImportError(f"IndexTTS-2 dependencies not available. Error: {e}")
+```
+
+**Impact**:
+- Callers can't predict exception types
+- Try/except blocks can't handle all cases
+- Different behaviors across engines for same failure scenario
+
+---
+
+### Issue #7: Device Resolution Duplication (SEVERITY: MEDIUM)
+
+**Multiple device resolution functions with different logic**:
+
+1. **torch_device_resolver.py** (authoritative):
+```python
+def resolve_torch_device(device: str) -> str:
+    if device == "auto":
+        if torch.backends.mps.is_available():  # ← Checks MPS
+            return "mps"
+        elif torch.cuda.is_available():
+            return "cuda"
+```
+
+2. **base_node.py** (partial):
+```python
+def resolve_device(self, device: str) -> str:
+    if device == "auto":
+        return "cuda" if torch.cuda.is_available() else "cpu"
+        # ← MISSING MPS and XPU checks!
+```
+
+**Impact**:
+- Different code paths return different devices
+- MPS (Apple Silicon) users get CPU fallback in some cases
+- Inconsistent behavior across node types
+
+---
+
+## PART 3: ROOT CAUSE ANALYSIS
+
+Why did this happen?
+
+1. **Evolutionary Development**: Unified interface added later, legacy systems still working → coexistence strategy chosen
+2. **Engine Diversity**: Each engine (ChatterBox, F5-TTS, Higgs, RVC) has different requirements → factories handle differently
+3. **Lack of Specification**: No defined contract for what a factory must implement
+4. **Incremental Adoption**: New engines use unified interface, but integration with old engines takes time
+
+**None of this is "bad code"** - it's a natural architecture that emerges when adding new systems to existing ones.
+
+---
+
+## PART 4: DOES THIS NEED TO BE FIXED?
+
+### Short Answer
+**For users**: System works. All engines load, all models work.
+
+**For maintainers**: Accumulating technical debt that will cause problems as more engines are added.
+
+### The Real Risk
+
+The biggest risk is **in future maintenance**:
+
+```
+Every new engine added will need:
+- Custom factory implementation
+- Custom error handling
+- Possibly custom device logic
+- Possibly custom recovery code (for Higgs-like corruption issues)
+
+This complexity INCREASES with more engines.
+```
+
+**Current**: 7 engines, manageable
+
+**In 1 year**: 10-15 engines likely, architecture breaks down
+
+### Scenarios Where Current Issues Cause Real Problems
+
+1. **User with GPU+CPU hybrid setup** (or laptop with power management):
+   - Device shifts from CUDA to CPU unexpectedly
+   - Models reload incorrectly
+   - Performance degrades mysteriously
+
+2. **ComfyUI with tight VRAM budget**:
+   - Clear VRAM → device resolves to CPU
+   - Cache misses → reloads in CPU VRAM
+   - Compounds memory pressure
+
+3. **Adding new engine similar to Higgs** (requires special recovery):
+   - Need to modify model_manager.py with new engine-specific code
+   - Risk breaking existing engines' recovery logic
+
+4. **Long-running generation sessions**:
+   - Triple cache accumulation
+   - Memory leaks from stale cached models
+   - Slow degradation over 8+ hours
+
+---
+
+## PART 5: IMPROVEMENT RECOMMENDATIONS
+
+### Tier 1: Quick Wins (1-2 hours, Low Risk)
+
+These can be implemented immediately with minimal testing:
+
+#### 1.1 Fix "auto" Device Instability
+**Change**: Cache resolved device in cache key, not "auto" string
+
+```python
+# Before: Cache key includes "auto"
+cache_key = f"chatterbox_tts_auto"
+
+# After: Resolve first
+resolved = resolve_torch_device(device)
+cache_key = f"chatterbox_tts_{resolved}"
+```
+
+**Impact**: Eliminates unnecessary model reloads on device changes
+**Risk**: Very low - just earlier resolution
+
+#### 1.2 Fix Device Comparison
+**Change**: Resolve device on both sides before comparison
+
+```python
+# Before
+if wrapper.current_device != target_device:  # String vs torch.device
+
+# After
+from utils.device import resolve_torch_device
+target_resolved = resolve_torch_device(config.device)
+actual_resolved = resolve_torch_device(wrapper.current_device)
+if target_resolved != actual_resolved:
+```
+
+**Impact**: Device movement actually works
+**Risk**: Very low - just standardization
+
+#### 1.3 Consolidate Device Resolution
+**Change**: Have single `resolve_torch_device()` function, use everywhere
+
+**Current**: base_node.py has own resolve_device() missing MPS/XPU checks
+
+**Fix**: Replace all `resolve_device()` calls with `resolve_torch_device()`
+
+**Impact**: Consistent behavior across all components
+**Risk**: Low - just replacing with more complete function
+
+---
+
+### Tier 2: Medium Effort (4-6 hours, Medium Risk)
+
+These require more testing but are valuable:
+
+#### 2.1 Unify Caching System
+**Change**: Remove SmartModelLoader and manager-specific caches, route through unified interface
+
+**Current architecture**:
+```
+ComfyUI Manager ←─┐
+SmartModelLoader  ├─ THREE SYSTEMS (no sync)
+Manager dicts ────┘
+```
+
+**After**:
+```
+All caches → ComfyUI Manager (single source of truth)
+```
+
+**Benefit**:
+- Cache clearing actually clears everything
+- No model duplication across caches
+- Simpler debugging
+
+**Risk**: Medium - must ensure all code paths work through single system
+
+#### 2.2 Fix Cache Key Collision
+**Change**: Hash full path, not just basename
+
+```python
+# Before
+components.append(f"path:{Path(config.model_path).name}")
+
+# After
+import hashlib
+path_hash = hashlib.md5(str(config.model_path).encode()).hexdigest()[:8]
+components.append(f"path:{path_hash}")
+```
+
+**Impact**: Eliminates collision risk
+**Risk**: Low - just better hashing
+
+#### 2.3 Standardize Error Handling
+**Change**: Create base exception class, use everywhere
+
+```python
+class TTSModelLoadingError(Exception):
+    """Base exception for TTS model loading failures"""
+    pass
+
+class TTSModelNotFoundError(TTSModelLoadingError):
+    pass
+
+class TTSDeviceError(TTSModelLoadingError):
+    pass
+```
+
+Update all factories to raise consistent exceptions.
+
+**Impact**: Callers can reliably catch and handle errors
+**Risk**: Low - just standardization
+
+---
+
+### Tier 3: Major Refactor (2-3 days, High Risk)
+
+These provide architectural improvements but require extensive testing:
+
+#### 3.1 Standardize Factory Signatures
+**Change**: Define common parameter interface
+
+```python
+@dataclass
+class FactoryParameters:
+    """Standard parameters all factories accept"""
+    device: str  # Required
+    model_name: str  # Required
+    language: Optional[str] = None  # Optional
+    model_path: Optional[str] = None  # Optional
+    repo_id: Optional[str] = None  # Optional
+    additional_params: Optional[Dict] = None  # For engine-specific
+```
+
+Each factory receives this dataclass, extracts needed fields.
+
+**Benefit**:
+- Adding new factories is simpler
+- No parameter confusion
+- Extensible without modifying unified interface
+
+**Risk**: High - changes all factories, needs full testing
+
+#### 3.2 Engine Capability Registry
+**Change**: Instead of hardcoding recovery logic, engines register capabilities
+
+```python
+ENGINE_CAPABILITIES = {
+    "higgs_audio": {
+        "supports_cuda_graphs": True,
+        "recovery_handler": reset_higgs_cuda_graphs,
+        "can_corrupt": True
+    },
+    "f5tts": {
+        "supports_cuda_graphs": False,
+        "recovery_handler": None,
+        "can_corrupt": False
+    }
+}
+```
+
+Generic recovery logic uses registry.
+
+**Benefit**:
+- Adding new engine with special needs is data-driven, not code-driven
+- model_manager.py doesn't need modification for new engines
+- Scales to 20+ engines
+
+**Risk**: High - changes error recovery pattern, needs extensive testing
+
+#### 3.3 Unified Fallback Handler
+**Change**: Extract ChatterBox's fallback logic to generic utility
+
+ChatterBox has sophisticated fallback chain (local → HF → fallback language). This is valuable for any engine.
+
+```python
+def intelligent_model_loader(
+    primary_loader: Callable,
+    fallback_languages: List[str],
+    final_fallback: Callable
+) -> Any:
+    """Generic intelligent loader with fallback chain"""
+    try:
+        return primary_loader()
+    except ImportError as e:
+        if "401" in str(e):
+            # Handle gated models
+            pass
+        # Try fallback languages
+        for lang in fallback_languages:
+            try:
+                return primary_loader_with_lang(lang)
+            except:
+                continue
+        # Final fallback
+        return final_fallback()
+```
+
+**Benefit**:
+- Other engines get robust fallback behavior
+- Consistent error recovery across all engines
+- ChatterBox logic is maintainable, not scattered
+
+**Risk**: Medium - fallback patterns must be generic enough
+
+---
+
+## PART 6: IMPLEMENTATION PRIORITY MATRIX
+
+```
+RISK ↑
+│
+H │   Tier 3.1 (Factory)      Tier 3.2 (Registry)    Tier 3.3 (Fallback)
+│   Signature                 Engine Capabilities    Unified Handler
+│       [2-3 days]                [2-3 days]            [1-2 days]
+│
+M │   Tier 2.1 (Cache Unify)   Tier 2.2 (Key Hash)    Tier 2.3 (Errors)
+│   [4-6 hours]                  [2 hours]              [2 hours]
+│
+L │                                                      Tier 1.1-1.3
+│                                                    Device & Resolution
+│                                                        [1-2 hours]
+│
+└──────────────────────────────────────────────────────────→ EFFORT/REWARD
+
+RECOMMENDATION ORDER:
+1. Tier 1 (all three) - Highest ROI, lowest risk
+2. Tier 2.1 - Unify caching (critical for maintenance)
+3. Tier 2.2, 2.3 - Polish improvements
+4. Tier 3 - Only if planning major expansion (10+ more engines)
+```
+
+---
+
+## PART 7: RECOMMENDATION FOR THIS PROJECT
+
+### Do you need to fix this?
+
+**For current state (7 engines)**: NO
+- System works without errors
+- Legacy fallbacks cover edge cases
+- Not blocking any functionality
+
+**For future (planning to add 10+ engines)**: YES
+- Complexity compounds with each engine
+- New contributors will struggle with inconsistent patterns
+- Technical debt becomes operational risk
+
+### Suggested Approach
+
+**Phase 1** (Next 1-2 weeks): Implement Tier 1 fixes
+- Solves "auto" device and comparison issues
+- Takes ~2 hours
+- Zero risk
+
+**Phase 2** (Optional, Next month): Implement Tier 2.1
+- Unifies caching
+- Removes triple-system confusion
+- Takes ~6 hours
+- Requires testing
+
+**Phase 3** (Only if adding multiple new engines): Implement Tier 3
+- Registers as architectural upgrade
+- Plan it as part of new engine addition
+- Amortize effort across multiple additions
+
+### Decision Point
+
+**Ask yourself**:
+1. Are you planning to add 10+ engines in the next 2 years? → Do Tier 3
+2. Is cache management causing issues in user reports? → Do Tier 2.1
+3. Is device handling causing issues? → Do Tier 1 immediately
+
+If answers are NO, NO, NO → **Current system is fine. Monitor for issues.**
+
+---
+
+## PART 8: SPECIFIC FILE CHANGES SUMMARY
+
+### Files to Modify (Tier 1)
+
+1. **utils/device/torch_device_resolver.py**
+   - Add caching for "auto" resolution
+   - 10-15 lines changed
+
+2. **utils/models/unified_model_interface.py**
+   - Use resolved device in cache keys (line 259)
+   - Fix device comparison (line 180)
+   - 5-10 lines changed
+
+3. **nodes/base/base_node.py**
+   - Remove resolve_device() method, use torch_device_resolver
+   - 3-5 lines changed
+
+### Files to Modify (Tier 2.1)
+
+1. **utils/models/comfyui_model_wrapper/model_manager.py**
+   - Remove engine-specific recovery code
+   - Add generic recovery registration
+
+2. **utils/models/smart_loader.py**
+   - Deprecate in favor of unified interface
+   - Route through ComfyUI manager
+
+3. **utils/models/manager.py**
+   - Remove fallback to SmartModelLoader
+   - Always use unified interface
+
+---
+
+## CONCLUSION
+
+**The system is architecturally sound for current scope** (7 engines), but accumulated technical debt will become problematic at larger scale.
+
+**Recommended action**:
+- Implement Tier 1 fixes now (low effort, immediate benefit)
+- Monitor for real-world issues
+- Plan Tier 2.1 if caching becomes problematic
+- Consider Tier 3 if project adds 10+ engines
+
+The improvements would take a skilled developer 2-3 weeks for full implementation, spread across phases. Each phase independently provides value.
+
+---
+
+**Report prepared by**: Claude Code Analysis
+**Analysis method**: Deep codebase examination + cross-reference testing
+**Confidence level**: High (based on 633 lines of detailed code analysis)

docs/Dev reports/SRT_Edit_Tag_Batch_Processing_Pattern.md

@@ -0,0 +1,87 @@
+# SRT Edit Tag Batch Processing Pattern
+
+## Problem
+Edit post-processing was happening per-subtitle, causing the editing engine to load/unload multiple times per SRT generation, wasting time and VRAM.
+
+## Solution
+Batch all edit processing at the END of SRT generation, after ALL subtitles are generated.
+
+## Implementation Pattern
+
+### 1. Adapter Layer (e.g., `vibevoice_adapter.py`)
+In `generate_with_pause_tags()`:
+- Extract edit tags even when NO pause tags present
+- Return segment dict format `[{waveform, text, edit_tags, ...}]` when edit tags exist
+- Return plain tensor when NO edit tags (backwards compatibility)
+
+### 2. Regular Processor (e.g., `step_audio_editx_processor.py`)
+In `_process_character_switching()`:
+```python
+# Apply edit post-processing ONLY when NOT in SRT mode
+if not self._srt_mode:
+    audio_segments = apply_edit_post_processing(
+        audio_segments,
+        self.config,
+        pre_loaded_engine=self.adapter.engine
+    )
+```
+
+### 3. SRT Processor (e.g., `vibevoice_srt_processor.py`, `step_audio_editx_srt_processor.py`)
+
+In `_process_all_subtitles()`:
+
+```python
+# Initialize collection for batch processing
+all_segments_for_editing = []
+
+# Loop through subtitles
+for i, subtitle in enumerate(subtitles):
+    segments = generate_subtitle(subtitle)
+
+    # Check if segments have edit tags
+    if isinstance(segments, list) or has_edit_tags(segments):
+        # Tag with subtitle index for later reassembly
+        for seg in segments:
+            seg['subtitle_index'] = i
+        all_segments_for_editing.extend(segments)
+        audio_segments[i] = None  # Placeholder
+    else:
+        # No edit tags - store directly
+        audio_segments[i] = combine_segments(segments)
+
+# BATCH PROCESS all edits at once
+if all_segments_for_editing:
+    print(f"🎨 Applying edit post-processing to {len(all_segments_for_editing)} segment(s) from all subtitles...")
+    processed = apply_edit_post_processing(
+        all_segments_for_editing,
+        config,
+        pre_loaded_engine=engine
+    )
+
+    # Group back by subtitle_index
+    segments_by_subtitle = group_by_subtitle_index(processed)
+
+    # Combine and store
+    for sub_idx, segs in segments_by_subtitle.items():
+        audio_segments[sub_idx] = combine_segments(segs)
+```
+
+### 4. Subtitle Processing Functions
+In functions like `_process_custom_character_switching_subtitle_with_params()`:
+- Return segment dicts when edit tags present (don't apply edits)
+- Return plain tensor when no edit tags
+- Let caller handle batch processing
+
+## Benefits
+- Edit engine loads ONCE instead of once per subtitle
+- Faster SRT generation
+- Less VRAM thrashing
+- Same quality output
+
+## Engines Implemented
+- ✅ VibeVoice SRT
+- ✅ Step Audio EditX SRT
+- ✅ Higgs Audio SRT
+
+## TODO
+- ChatterBox 23-Lang (needs edit tag support first)

docs/Dev reports/Step_Audio_EditX_Implementation_Plan.md

@@ -0,0 +1,464 @@
+# Step Audio EditX Engine Implementation Plan
+
+## Executive Summary
+
+**Implementation Approach**: Bundle Step Audio EditX code (like IndexTTS pattern) with TTS Suite integration
+
+**Key Decisions:**
+- ✅ **Bundle code** in `step_audio_editx_impl/` (NOT pip install)
+- ✅ **Skip Whisper** dependency (only needed for external audio transcription)
+- ✅ **Unpinned transformers** (patch if needed via TTS Suite system)
+- ✅ **Progress bars** via ComfyUI (UX improvement)
+- ✅ **Manual transcripts** for Audio Editor (Whisper integration later)
+- ✅ **Phased implementation**: Clone → Edit → SRT
+
+**What Step Audio EditX Is:**
+- 3B parameter QWen-based LLM for semantic generation
+- Custom dual-codebook tokenizer (VQ02 + VQ06)
+- CosyVoice vocoder for final audio generation (NOT full CosyVoice)
+- Unique emotion/style/speed editing capabilities
+- Worth implementing alongside future CosyVoice engine
+
+---
+
+## Reference Implementations
+
+**Local copies available for reference:**
+- **Original (stepfun-ai)**: `IgnoredForGitHubDocs/For_reference/Step-Audio-EditX/`
+- **Saganaki22's ComfyUI**: `IgnoredForGitHubDocs/For_reference/ComfyUI-Step_Audio_EditX_TTS/`
+
+---
+
+## Analysis of Both Implementations
+
+### Original Step Audio EditX (stepfun-ai)
+- **Core Features**:
+  - Zero-shot voice cloning (3-30s reference audio)
+  - Audio editing: emotion, style, speed, paralinguistics
+  - Supports Mandarin, English, Sichuanese, Cantonese, Japanese, Korean
+  - Iterative editing (n_edit_iter parameter)
+  - Denoise and VAD
+  - Polyphone pronunciation control
+  - 3B parameter model with dual-codebook tokenizer
+
+- **Architecture**:
+  - `tokenizer.py` (StepAudioTokenizer) - Dual-codebook tokenizer
+  - `tts.py` (StepAudioTTS) - Main TTS engine with clone() and edit() methods
+  - `model_loader.py` - Model loading from HuggingFace/ModelScope/local
+  - `config/edit_config.py` - Edit type configurations
+  - Uses bundled `funasr_detach/` for FunASR models (paraformer tokenizer)
+  - CosyVoice vocoder for audio generation (24kHz output)
+
+### Saganaki22's ComfyUI Implementation
+- **Native Features**: All of the above
+- **Custom Additions (NON-NATIVE)**:
+  1. **Longform chunking system** (`core/longform_chunker.py`):
+     - Splits long text at sentence boundaries
+     - Estimates tokens (1 token ≈ 4 chars)
+     - Creates overlapping chunks
+     - This is a ComfyUI-specific optimization NOT in original
+
+  2. **Audio stitching** (`core/audio_stitcher.py`):
+     - Concatenates audio chunks from longform generation
+     - Crossfade support
+     - NOT part of original implementation
+
+  3. **Progress bar integration**:
+     - ComfyUI progress tracking during clone generation
+     - NOT in original (original has no progress callbacks)
+
+  4. **Model wrapper architecture**:
+     - `StepAudioModelWrapper` with to_cpu()/to_cuda() methods
+     - CUDA graph cleanup logic
+     - More sophisticated VRAM management than original
+
+**CRITICAL**: Speed control (faster/slower) IS native - it's in the original edit_config.py, so this is NOT a custom addition.
+
+---
+
+## Recommended Implementation Approach
+
+### Use Original as Base (NOT Saganaki22)
+
+Saganaki22's implementation is excellent for ComfyUI but adds non-native features. Since you want faithful implementation, we start fresh.
+
+### Phase 1: Core Engine Implementation
+
+#### Files to Create:
+
+1. **`engines/step_audio_editx/step_audio_editx.py`**
+   - Main engine wrapper
+   - Wraps original `StepAudioTTS` class
+   - Implements `.to()` method for ComfyUI VRAM management
+   - Device checking before generation
+   - Sample rate: 24000 Hz (CosyVoice native)
+
+2. **`engines/step_audio_editx/step_audio_downloader.py`**
+   - Uses `utils/downloads/unified_downloader.py`
+   - Downloads Step-Audio-EditX and Step-Audio-Tokenizer
+   - Organized structure: `models/TTS/step_audio_editx/`
+
+3. **`engines/step_audio_editx/__init__.py`**
+   - Engine initialization
+   - Imports
+
+4. **`engines/step_audio_editx/original_impl/`**
+   - Copy entire original Step Audio EditX implementation here
+   - Includes: `tts.py`, `tokenizer.py`, `model_loader.py`, `config/`, `funasr_detach/`, etc.
+   - Keep it isolated so it doesn't conflict with existing dependencies
+
+#### Key Implementation Notes:
+
+**Audio Format**:
+- Original outputs: `torch.Tensor` shape `[samples]` or `[1, samples]`
+- Sample rate: 24000 Hz
+- Return format: TTS Audio Suite expects `[1, samples]`
+
+**Generation Methods**:
+```python
+def clone(self, prompt_wav_path, prompt_text, target_text, temperature=0.7, do_sample=True, max_new_tokens=4096):
+    # Call original implementation
+    pass
+
+def edit(self, input_audio_path, audio_text, edit_type, edit_info=None, text=None):
+    # Edit mode - temperature/do_sample/max_new_tokens are HARDCODED in original
+    # Parameters: edit_type (emotion/style/speed/paralinguistic/denoising)
+    # Returns audio_tensor, sample_rate
+    pass
+```
+
+**VRAM Management** (CRITICAL):
+- Implement `.to(device)` method
+- Move `self.llm` (3B parameter model) between devices
+- Move `self.cosy_model` (vocoder) between devices
+- Check device before generation and reload if needed
+- Follow patterns from IndexTTS implementation
+
+---
+
+### Phase 2: Node Implementation
+
+#### Files to Create:
+
+1. **`nodes/engines/step_audio_editx_engine_node.py`**
+   - Configuration node for Step Audio EditX
+   - Parameters:
+     - `model_path` - path to Step-Audio-EditX
+     - `device` - cuda/cpu
+     - `torch_dtype` - bfloat16/float16/float32
+     - `quantization` - none/int4/int8
+     - `attention_mechanism` - sdpa/eager/flash_attn
+   - Returns: ENGINE config
+
+2. **`nodes/step_audio_editx/step_audio_editx_processor.py`**
+   - Main TTS processor (internal, used by Unified TTS Text node)
+   - Handles:
+     - Character switching with `[CharacterName]` tags
+     - Pause tags with `[pause:1.5s]` syntax
+     - Caching via `utils/audio/cache.py`
+     - Voice discovery via `utils/voice/discovery.py`
+     - **Longform chunking via `utils/audio/chunk_combiner.py`** (TTS Suite unified)
+   - Sequential generation with interrupt handling
+
+3. **`nodes/step_audio_editx_special/step_audio_editx_audio_editor_node.py`** (Special Feature)
+   - Dedicated "🎨 Step Audio EditX - Audio Editor" node
+   - This is NOT voice conversion - it's specialized audio manipulation
+   - **CRITICAL**: Requires `audio_text` parameter (transcript of input audio)
+     - User must provide transcript manually (no Whisper transcription)
+     - Future: When Whisper engine is added, can auto-feed transcripts
+   - Parameters:
+     - `input_audio` - audio to edit (0.5-30s limit)
+     - `audio_text` - REQUIRED transcript of input audio
+     - `edit_type` - emotion/style/speed/paralinguistic/denoising
+     - `emotion` - happy/sad/angry/excited/etc (14 options)
+     - `style` - whisper/serious/child/older/etc (32 options)
+     - `speed` - faster/slower/more faster/more slower
+     - `paralinguistic` - [Laughter]/[Breathing]/[Sigh]/etc (10 options)
+     - `denoising` - denoise/vad
+     - `n_edit_iterations` - 1-5 (strength through iteration)
+     - `paralinguistic_text` - where to insert effect
+   - Returns edited audio
+
+---
+
+### Phase 3: Unified Systems Integration
+
+#### Integration Points:
+
+1. **Unified Model Loading**:
+   ```python
+   from utils.models.unified_model_interface import UnifiedModelInterface
+
+   # Register Step Audio EditX
+   unified_model_interface.register_model(
+       engine_type="step_audio_editx",
+       model=step_audio_engine,
+       model_info=ModelInfo(...)
+   )
+   ```
+
+2. **Character System**:
+   - Use `utils/text/character_parser.py` for `[CharacterName]` tags
+   - Use `utils/voice/discovery.py` for voice file loading
+   - Support both .wav and .txt voice references
+
+3. **Pause Tags**:
+   - Use `utils/text/pause_processor.py` for `[pause:xx]` syntax
+   - Integrate with audio generation pipeline
+
+4. **Caching**:
+   ```python
+   from utils.audio/cache import UnifiedCacheManager
+   from utils.audio.audio_hash import create_content_hash
+
+   cache_key = create_content_hash(
+       text=text,
+       voice_path=voice_path,
+       engine="step_audio_editx",
+       parameters={...}
+   )
+   ```
+
+5. **Interrupt Handling** (CRITICAL):
+   ```python
+   import comfy.model_management as model_management
+
+   # Before each generation
+   if model_management.interrupt_processing:
+       raise InterruptedError("Step Audio EditX generation interrupted")
+   ```
+
+---
+
+### Phase 4: Dependencies
+
+#### Add to `requirements.txt`:
+```txt
+# Step Audio EditX dependencies
+librosa>=0.10.2
+soundfile>=0.12.1  # Critical for ComfyUI audio preview
+onnxruntime>=1.17.0
+transformers>=4.30.0  # Unpinned - use TTS Suite transformer patches if needed
+sentencepiece>=0.1.99
+omegaconf>=2.3.0
+hyperpyyaml>=1.2.2
+# NOTE: openai-whisper NOT required for TTS (only for external audio transcription)
+# Future: When Whisper transcription engine is added, it can feed Audio Editor node
+```
+
+**Dependency Strategy:**
+- ✅ **Bundle code** (like IndexTTS) - not pip installable
+- ✅ **Skip Whisper** - only needed for external audio → edit workflow
+- ✅ **Unpinned transformers** - patch via TTS Suite system if 4.54+ breaks
+- ✅ **Windows-friendly** - no complex model downloads for basic TTS
+
+---
+
+### Phase 5: SRT Implementation (Later)
+
+**Strategy**: Sequential processing with language grouping
+- Group SRT segments by language (Mandarin/English/Japanese/Korean/etc)
+- Process each language group sequentially
+- No parallel processing (model doesn't support it well)
+- Support character switching within segments
+- Use timing modes from `utils/timing/`
+
+**File**: `nodes/step_audio_editx/step_audio_editx_srt_processor.py`
+
+---
+
+## Key Differences from Saganaki22 Implementation
+
+| Feature | Original | Saganaki22 | Our Implementation |
+|---------|----------|------------|-------------------|
+| Longform chunking | ❌ No | ✅ Yes (custom) | ✅ Yes (TTS Suite unified) |
+| Progress bars | ❌ No | ✅ Yes (clone only) | ✅ Yes (ComfyUI standard) |
+| Whisper transcription | ⚠️ Optional (ASR) | ⚠️ Optional (ASR) | ❌ Skip (future: Whisper engine) |
+| Audio stitching | ❌ No | ✅ Yes (custom) | ❌ No (not native) |
+| Iterative editing | ✅ Yes | ✅ Yes | ✅ Yes (native) |
+| Speed control | ✅ Yes | ✅ Yes | ✅ Yes (native) |
+| Character switching | ❌ No | ❌ No | ✅ Yes (TTS Suite feature) |
+| Pause tags | ❌ No | ❌ No | ✅ Yes (TTS Suite feature) |
+| Caching | ❌ No | ❌ No | ✅ Yes (TTS Suite feature) |
+| Interrupt handling | ❌ No | ❌ No | ✅ Yes (TTS Suite feature) |
+
+---
+
+## Critical Implementation Requirements
+
+### 1. VRAM Management (from guide lines 96-264)
+- Implement `.to(device)` method
+- Move LLM (self.llm) and vocoder (self.cosy_model) between devices
+- Check device before generation
+- Reload from CPU to CUDA if offloaded
+- Search for existing wrapper before creating new ones
+
+### 2. Clear VRAM Integration
+- Register with ComfyUI model management
+- Support "Clear VRAM" button functionality
+- Prevent device mismatch errors
+
+### 3. Interrupt Handling (CRITICAL)
+- Check `model_management.interrupt_processing` before each generation
+- In SRT mode: check before each segment
+- Allow users to cancel long operations
+
+### 4. Audio Limitations
+- **Clone mode**: No strict length limit (depends on max_new_tokens)
+- **Edit mode**: 0.5-30 seconds HARD LIMIT
+- Validate audio duration before processing
+
+### 5. Language Support
+- Mandarin (primary)
+- English
+- Sichuanese dialect
+- Cantonese dialect
+- Japanese
+- Korean
+- Add `[Sichuanese]`, `[Cantonese]`, `[Japanese]`, `[Korean]` tag support
+
+### 6. Edit Types and Options
+All options are from original `config/edit_config.py`:
+
+**Emotions** (14):
+- happy, sad, angry, excited, calm, fearful, surprised, disgusted
+- confusion, empathy, embarrass, depressed, coldness, admiration
+
+**Styles** (32):
+- whisper, serious, child, older, girl, pure, sister, sweet
+- exaggerated, ethereal, generous, recite, act_coy, warm, shy
+- comfort, authority, chat, radio, soulful, gentle, story, vivid
+- program, news, advertising, roar, murmur, shout, deeply, loudly
+- arrogant, friendly
+
+**Speed** (4):
+- faster, slower, more faster, more slower
+
+**Paralinguistics** (10):
+- [Breathing], [Laughter], [Surprise-oh], [Confirmation-en]
+- [Uhm], [Surprise-ah], [Surprise-wa], [Sigh]
+- [Question-ei], [Dissatisfaction-hnn]
+
+**Denoising** (2):
+- denoise, vad
+
+---
+
+## File Structure
+
+```
+engines/step_audio_editx/
+├── __init__.py
+├── step_audio_editx.py (main wrapper - follows IndexTTS pattern)
+├── step_audio_editx_downloader.py (unified downloader integration)
+└── step_audio_editx_impl/ (bundled original codebase - like IndexTTS)
+    ├── tts.py (StepAudioTTS - main engine)
+    ├── tokenizer.py (StepAudioTokenizer - dual-codebook)
+    ├── model_loader.py (ModelSource, model_loader)
+    ├── config/
+    │   ├── edit_config.py (edit types/options)
+    │   └── prompts.py
+    ├── funasr_detach/ (bundled FunASR - custom fork)
+    └── stepvocoder/cosyvoice2/ (CosyVoice vocoder)
+
+nodes/engines/
+└── step_audio_editx_engine_node.py (config node)
+
+nodes/step_audio_editx/
+├── step_audio_editx_processor.py (TTS processor - internal)
+└── step_audio_editx_srt_processor.py (SRT processor - internal, later)
+
+nodes/step_audio_editx_special/
+└── step_audio_editx_audio_editor_node.py (🎨 Audio Editor - specialized manipulation)
+
+```
+
+---
+
+## Testing Checklist
+
+### Phase 1 Testing (Basic TTS):
+- [ ] Model downloads correctly
+- [ ] Basic clone generation works
+- [ ] Audio format is correct (24kHz, mono)
+- [ ] Character switching works
+- [ ] Pause tags work
+- [ ] Caching works
+- [ ] VRAM management works (Clear VRAM button)
+- [ ] Device reload works after offload
+- [ ] Interrupt handling works
+
+### Phase 2 Testing (Audio Editing):
+- [ ] Emotion editing works (all 14 emotions)
+- [ ] Style editing works (all 32 styles)
+- [ ] Speed editing works (4 speed levels)
+- [ ] Paralinguistic editing works (all 10 effects)
+- [ ] Denoising works (denoise + vad)
+- [ ] Iterative editing works (n_edit_iter 1-5)
+- [ ] Audio length validation works (0.5-30s)
+- [ ] Edit node integrates with voice changer workflows
+
+### Phase 3 Testing (SRT Mode):
+- [ ] SRT parsing works
+- [ ] Language grouping works
+- [ ] Character switching in SRT works
+- [ ] Timing assembly works
+- [ ] Multiple languages in same SRT works
+- [ ] Interrupt handling in SRT works
+
+---
+
+## Questions for User
+
+Before we proceed with implementation, I need clarification on:
+
+1. **Longform chunking**: Saganaki22 added sentence-based text chunking for long content. This is NOT in the original. Do you want this feature, or should we stick to the original behavior (which may truncate long text at max_new_tokens)?
+
+2. **Progress bars**: Saganaki22 added ComfyUI progress tracking during clone generation. This is NOT in the original. Should we add this for better UX, or skip it?
+
+3. **Separate edit node**: Should we create a dedicated "Step Audio EditX - Edit" node for audio editing (emotion/style/speed), or integrate everything through the unified voice changer node?
+
+4. **Model location**: Should models go in `models/TTS/step_audio_editx/` or `models/step_audio_editx/` (Saganaki22 used `models/Step-Audio-EditX/`)?
+
+5. **Implementation priority**: What should we implement first?
+   - Option A: Clone mode only (basic TTS) → test → then edit mode → then SRT
+   - Option B: Clone + Edit modes together → test → then SRT
+   - Option C: All features at once
+
+---
+
+## Implementation Strategy (User Approved)
+
+**Phase 1**: Clone mode with TTS Suite integration
+- Full TTS Suite integration (character switching, pause tags, caching)
+- Use `utils/audio/chunk_combiner.py` for longform text
+- **Progress bars** via `comfy.utils.ProgressBar` during generation
+- VRAM management with `.to()` method (follow IndexTTS pattern)
+- Interrupt handling via `model_management.interrupt_processing`
+- Bundle code in `step_audio_editx_impl/` (like IndexTTS)
+- Test thoroughly before proceeding
+
+**Phase 2**: Audio Editor (specialized node)
+- Dedicated "🎨 Step Audio EditX - Audio Editor" node
+- NOT integrated with voice changer (different purpose)
+- **Requires manual transcript input** (audio_text parameter)
+  - Skip Whisper transcription for now
+  - Future: Auto-feed from Whisper transcription engine
+- All editing capabilities (emotion/style/speed/paralinguistic/denoising)
+- Iterative refinement support (n_edit_iter 1-5)
+- Audio length validation (0.5-30s)
+- Test all editing modes
+
+**Phase 3**: SRT support
+- Sequential processing with language grouping
+- Character switching within segments
+- Timing assembly
+- Interrupt handling for cancellation
+
+This approach:
+- Allows incremental testing
+- Validates VRAM management early
+- Ensures unified systems work correctly
+- Follows TTS Suite best practices
+- Lets you test and approve each phase
+

docs/Dev reports/Step_Audio_EditX_Inline_Tags_Plan.md

@@ -0,0 +1,557 @@
+# Step Audio EditX Inline Tags - Implementation Plan
+
+## Overview
+
+Add inline Step Audio EditX edit tags to the segment parameter system, enabling automatic post-processing edits on TTS-generated audio from ANY engine.
+
+**Example:**
+```
+[Alice|seed:43] This is an example <Laughter:3>, I'm funny!
+```
+
+Result:
+1. TTS generates "This is an example, I'm funny!" (tag stripped)
+2. Step Audio EditX automatically edits the segment with paralinguistic `<Laughter>` in 3 iterations
+
+## Syntax Design
+
+### Current Segment Parameter Syntax
+```
+[Character|param:value|param2:value2]
+```
+
+### New Edit Tag Syntax (inline in text)
+
+**Paralinguistic tags** (insert non-verbal sounds):
+```
+Hello <Laughter> world           # Default 1 iteration
+Hello <Laughter:3> world         # 3 iterations
+Hello <Sigh:2> goodbye           # 2 iterations
+```
+
+**Other edit types** - need different syntax to avoid confusion:
+
+Option A - Angle brackets with type prefix:
+```
+<emotion:happy:2>      # emotion=happy, 2 iterations
+<style:whisper:3>      # style=whisper, 3 iterations
+<speed:faster:1>       # speed=faster, 1 iteration
+```
+
+Option B - Curly braces for non-paralinguistic:
+```
+{emotion:happy:2}
+{style:whisper:3}
+{speed:faster:1}
+```
+
+Option C - Unified angle brackets, context determines type:
+```
+<happy:2>              # detected as emotion
+<whisper:3>            # detected as style
+<faster:1>             # detected as speed
+<Laughter:2>           # detected as paralinguistic (capitalized)
+```
+
+**Recommendation:** Option A - explicit type prefix. Clear, unambiguous, no magic detection.
+
+### Multiple Tags on Same Segment
+
+**Two syntax options for multiple edits:**
+
+1. **Pipe-separated in single tag** (like segment parameters):
+```
+[Alice] Hello world! <Laughter:2|style:whisper:1|emotion:happy>
+```
+
+2. **Multiple separate tags:**
+```
+[Alice] Hello world! <Laughter:2><style:whisper:1><emotion:happy>
+```
+
+Both are equivalent and can be mixed.
+
+**Processing order:**
+
+- **Paralinguistic tags**: Position in text matters (where to insert sound)
+  - `Hello <Laughter> world` → Laughter inserted after "Hello"
+
+- **Other edit types** (emotion, style, speed): Position doesn't matter, gets stripped
+  - `Hello <style:whisper> world` = `Hello world <style:whisper>` = `<style:whisper> Hello world`
+  - All result in: Generate "Hello world", then apply whisper style
+
+**Execution order for mixed tags:**
+1. First: All non-paralinguistic edits (emotion → style → speed) in fixed order
+2. Last: Paralinguistic edits (preserves position info for sound insertion)
+
+**Example:**
+```
+[Alice] Hello <Laughter:2> world <style:whisper:1>
+```
+1. Generate TTS: "Hello world"
+2. Edit pass 1: Apply whisper style (1 iteration) - position irrelevant
+3. Edit pass 2: Add Laughter at position (2 iterations) - position matters
+
+## Processing Flow
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  Input: "[Alice] Hello <Laughter:2> world!"                 │
+└─────────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────┐
+│  1. Parse segment: character=Alice, text with tags          │
+│     Extract edit tags: [("Laughter", "paralinguistic", 2)]  │
+│     Clean text: "Hello world!"                              │
+└─────────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────┐
+│  2. Generate TTS with clean text (any engine)               │
+│     Output: audio segment                                   │
+└─────────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────┐
+│  3. Check duration: > 30s?                                  │
+│     YES → Log warning, skip edit, return original audio     │
+│     NO  → Continue to edit                                  │
+└─────────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────┐
+│  4. Load Step Audio EditX engine (lazy load on first use)   │
+└─────────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────┐
+│  5. For each edit tag in order:                             │
+│     - Save audio to temp file                               │
+│     - Call edit_single() with appropriate edit_type         │
+│     - Use output as input for next edit (if any)            │
+└─────────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────┐
+│  Output: Edited audio segment                               │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## Architecture Decision: Processing Order
+
+### Option A: Generate All TTS First, Then Edit (RECOMMENDED)
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  1. Parse ALL segments, extract edit tags, store metadata   │
+│  2. Generate ALL TTS segments (any engine) - clean text     │
+│  3. Post-process: Apply edits to segments that have tags    │
+│  4. Assemble final audio                                    │
+└─────────────────────────────────────────────────────────────┘
+```
+
+**Pros:**
+- Step Audio EditX engine loaded ONCE, only if needed
+- All TTS generation completes before edit engine loads (memory efficient)
+- Can batch edit operations
+- Cleaner separation of concerns
+- Progress reporting is clearer (TTS phase → Edit phase)
+
+**Cons:**
+- Need to track which segments have edit tags through the pipeline
+
+### Option B: Edit Each Segment Immediately After Generation
+
+```
+For each segment:
+  1. Parse segment, extract edit tags
+  2. Generate TTS
+  3. If has edit tags → load Step Audio EditX → edit
+  4. Store result
+Assemble final audio
+```
+
+**Pros:**
+- Simpler data flow
+
+**Cons:**
+- Step Audio EditX loaded/unloaded repeatedly if interleaved
+- Two heavy models potentially in memory at same time
+- Harder to optimize
+
+**Decision: Option A** - Generate all TTS first, then batch edit.
+
+---
+
+## Implementation Details - Modular Architecture
+
+### Core Principle: NO CODE DUPLICATION
+
+The edit tag system integrates with existing parsers. We don't modify adapters individually.
+
+### Processing Pipeline
+
+```
+┌──────────────────────────────────────────────────────────────────────────┐
+│                        EXISTING PIPELINE                                  │
+│                                                                          │
+│  Text Input → Character Parser → Segment Parameters → TTS Generation     │
+│                                                                          │
+└──────────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌──────────────────────────────────────────────────────────────────────────┐
+│                      NEW: EDIT TAG EXTRACTION                            │
+│                                                                          │
+│  Happens at parsing stage (segment_parameters.py or new unified parser)  │
+│  - Extract <tag:iter> from text                                          │
+│  - Store in segment metadata: segment.edit_tags = [...]                  │
+│  - Return clean text for TTS                                             │
+│                                                                          │
+└──────────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌──────────────────────────────────────────────────────────────────────────┐
+│                      EXISTING: TTS GENERATION                            │
+│                                                                          │
+│  All engines generate audio from clean text (no edit tags)               │
+│  No changes to adapters needed!                                          │
+│                                                                          │
+└──────────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌──────────────────────────────────────────────────────────────────────────┐
+│                      NEW: POST-PROCESSOR (single point)                  │
+│                                                                          │
+│  Called ONCE after all TTS generation, before assembly                   │
+│  - Check each segment for edit_tags metadata                             │
+│  - If any segments have tags → lazy load Step Audio EditX                │
+│  - Apply edits to those segments                                         │
+│  - Return modified segments for assembly                                 │
+│                                                                          │
+└──────────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌──────────────────────────────────────────────────────────────────────────┐
+│                      EXISTING: AUDIO ASSEMBLY                            │
+│                                                                          │
+│  Combine segments (unchanged logic)                                      │
+│                                                                          │
+└──────────────────────────────────────────────────────────────────────────┘
+```
+
+### Files to Modify/Create
+
+#### 1. Extend Existing Parser: `utils/text/segment_parameters.py`
+
+Add edit tag extraction alongside existing parameter parsing:
+
+```python
+# Existing: [Character|seed:42|temp:0.8]
+# New addition: Also extract <Laughter:2|style:whisper>
+
+class SegmentData:
+    character: str
+    text: str  # Clean text (tags stripped)
+    parameters: dict  # seed, temp, etc.
+    edit_tags: list  # NEW: [("Laughter", "paralinguistic", 2, position), ...]
+```
+
+**No duplication** - extend existing `parse_segment()` function.
+
+#### 2. Extend: `utils/text/step_audio_editx_special_tags.py`
+
+Add iteration parsing to existing tag functions:
+
+```python
+# Existing: convert_step_audio_editx_tags(), strip_paralinguistic_tags()
+# Add: parse_edit_tags_with_iterations()
+
+def parse_edit_tags_with_iterations(text: str) -> Tuple[str, List[EditTag]]:
+    """
+    Parse all edit tags from text, return clean text and tag list.
+
+    Supports:
+    - <Laughter> → paralinguistic, 1 iter
+    - <Laughter:3> → paralinguistic, 3 iter
+    - <style:whisper:2> → style, 2 iter
+    - <emotion:happy> → emotion, 1 iter
+    - <Laughter:2|style:whisper:1> → multiple in one tag
+    """
+```
+
+#### 3. NEW: `utils/audio/edit_post_processor.py`
+
+Single centralized post-processor:
+
+```python
+class EditPostProcessor:
+    """
+    Post-processes TTS segments with Step Audio EditX edits.
+    Lazy loads engine only when needed.
+    """
+
+    _engine = None  # Lazy loaded
+
+    @classmethod
+    def process_segments(cls, segments: List[SegmentResult]) -> List[SegmentResult]:
+        """
+        Process all segments, applying edits where edit_tags exist.
+        Called ONCE after all TTS generation.
+        """
+        # Find segments with edit tags
+        segments_to_edit = [s for s in segments if s.edit_tags]
+
+        if not segments_to_edit:
+            return segments  # No edits needed
+
+        # Lazy load engine
+        if cls._engine is None:
+            cls._engine = cls._load_engine()
+
+        # Process each segment with tags
+        for segment in segments_to_edit:
+            segment.audio = cls._apply_edits(segment)
+
+        return segments
+```
+
+#### 4. Integration Points (MINIMAL CHANGES)
+
+**`nodes/unified/tts_text_node.py`** - Add one call:
+```python
+# After TTS generation, before return
+segments = EditPostProcessor.process_segments(segments)
+```
+
+**`nodes/unified/tts_srt_node.py`** - Add one call:
+```python
+# After all segments generated, before assembly
+segments = EditPostProcessor.process_segments(segments)
+```
+
+**Adapters: NO CHANGES NEEDED**
+- Adapters receive clean text (tags already stripped at parse stage)
+- Adapters return audio as usual
+- Post-processor handles edits centrally
+
+### Data Flow
+
+```
+Input: "[Alice|seed:42] Hello <Laughter:2> world! <style:whisper>"
+
+Step 1 - Parse (segment_parameters.py):
+  → character: "Alice"
+  → parameters: {seed: 42}
+  → clean_text: "Hello world!"
+  → edit_tags: [
+      EditTag(type="paralinguistic", value="Laughter", iterations=2, position=6),
+      EditTag(type="style", value="whisper", iterations=1, position=None)
+    ]
+
+Step 2 - TTS Generation (any adapter):
+  → Input: "Hello world!"
+  → Output: audio tensor
+
+Step 3 - Post-process (edit_post_processor.py):
+  → Check: segment has edit_tags? YES
+  → Validate: duration < 30s? YES
+  → Apply style:whisper (1 iter) - position-independent first
+  → Apply Laughter at position 6 (2 iter) - paralinguistic last
+  → Output: edited audio tensor
+
+Step 4 - Assembly (unchanged):
+  → Combine all segments
+```
+
+### Tag Detection
+
+**Paralinguistic tags** (existing set):
+- Breathing, Laughter, Sigh, Uhm
+- Surprise-oh, Surprise-ah, Surprise-wa
+- Confirmation-en, Question-ei, Dissatisfaction-hnn
+
+**Emotion tags** (14 options):
+- happy, sad, angry, excited, calm, fearful, surprised, disgusted
+- confusion, empathy, embarrass, depressed, coldness, admiration
+
+**Style tags** (32 options):
+- whisper, serious, child, older, girl, pure, sister, sweet
+- exaggerated, ethereal, generous, recite, act_coy, warm, shy
+- comfort, authority, chat, radio, soulful, gentle, story, vivid
+- program, news, advertising, roar, murmur, shout, deeply, loudly
+- arrogant, friendly
+
+**Speed tags** (4 options):
+- faster, slower, more_faster, more_slower
+
+### Duration Validation
+
+```python
+def validate_for_edit(audio_tensor, sample_rate):
+    duration = audio_tensor.shape[-1] / sample_rate
+    if duration > 30.0:
+        print(f"⚠️ Segment too long for Step Audio EditX edit ({duration:.1f}s > 30s). "
+              f"Edit tags will be skipped. Split your text into shorter segments.")
+        return False
+    if duration < 0.5:
+        print(f"⚠️ Segment too short for Step Audio EditX edit ({duration:.1f}s < 0.5s). "
+              f"Edit tags will be skipped.")
+        return False
+    return True
+```
+
+### Transcript Handling
+
+For edit mode, we need the transcript (what was spoken). This is the clean text BEFORE TTS:
+
+```python
+# Original: "[Alice] Hello <Laughter:2> world!"
+# Clean text for TTS: "Hello world!"
+# Transcript for edit: "Hello world!"  (same as clean text)
+
+# For paralinguistic, target text has tags:
+# Target: "Hello [Laughter] world!"
+```
+
+## Edge Cases
+
+1. **No Step Audio EditX model downloaded**
+   - First use triggers download (existing behavior)
+   - User sees download progress
+
+2. **Multiple segments, only some with tags**
+   - Only segments with tags get post-processed
+   - Others pass through unchanged
+
+3. **Tag in middle of word** (malformed)
+   - `Hel<Laughter>lo` → Strip tag, get "Hello"
+   - Best effort, no error
+
+4. **Unknown tag**
+   - `<UnknownTag:2>` → Ignored, passed as plain text, stripped
+   - Or warn user?
+
+5. **ChatterBox 23-Lang v2 tag collision**
+   - Their tags use different format: `<giggle>`, `<whisper>`, etc.
+   - Our edit tags: `<Laughter>`, `<style:whisper>`
+   - Paralinguistic overlap: need to check if text already has v2 tags
+   - Solution: Process our tags first, before sending to ChatterBox
+
+## Example Usage
+
+### Simple paralinguistic
+```
+[Alice] I can't believe it <Laughter> that's hilarious!
+```
+
+### With iterations
+```
+[Alice] I can't believe it <Laughter:3> that's hilarious!
+```
+
+### Multiple effects - pipe syntax
+```
+[Bob] I'm so tired, goodnight <Sigh:2|style:whisper:1>
+```
+
+### Multiple effects - separate tags
+```
+[Bob] I'm so tired <Sigh:2> goodnight <style:whisper:1>
+```
+
+### Mixed syntax (equivalent to above)
+```
+[Bob] I'm so tired <Sigh:2|style:whisper:1> goodnight
+```
+
+### Combined with segment parameters
+```
+[Alice|seed:42|temperature:0.8] Hello <Laughter:3> friend!
+```
+
+### Full combo
+```
+[Alice|seed:42] Hello <Laughter:2|emotion:happy:1> my friend!
+```
+
+### SRT mode
+```
+1
+00:00:00,000 --> 00:00:03,000
+[Alice] Hello <Laughter> world!
+
+2
+00:00:03,500 --> 00:00:06,000
+[Bob|seed:123] Hi there <style:whisper:2|emotion:calm>
+```
+
+### Position matters for paralinguistics
+```
+# Different results:
+[Alice] Hello <Laughter> world!    # Laughter after "Hello"
+[Alice] Hello world <Laughter>!    # Laughter after "world"
+
+# Same result (style position irrelevant):
+[Alice] Hello <style:whisper> world!
+[Alice] <style:whisper> Hello world!
+[Alice] Hello world! <style:whisper>
+```
+
+## Decisions Made
+
+1. **Syntax for non-paralinguistic edits:** `<emotion:happy:2>` - explicit type prefix
+2. **Multiple tags:** Support both `<tag1:2|tag2:3>` pipe syntax AND `<tag1:2><tag2:3>` separate tags
+3. **Iteration default:** `<Laughter>` = 1 iteration (no explicit `:1` required)
+4. **Warning behavior for >30s:** Skip edit with warning in console, return original audio
+5. **Engine loading:** Lazy load ONCE after all TTS generation completes
+6. **Processing order:** Generate ALL TTS first, then batch edit (Option A)
+7. **Architecture:** Modular, no adapter changes, single post-processor
+
+## Open Questions
+
+1. **Position tracking for paralinguistics:**
+   - How to track character position after tag stripping?
+   - Store original position? Or store as "after word N"?
+
+2. **Progress reporting:**
+   - Show separate progress for TTS phase vs Edit phase?
+   - How to report which segments are being edited?
+
+3. **Cache interaction:**
+   - Should edited segments be cached separately?
+   - Cache key should include edit tags?
+
+4. **Error handling:**
+   - If one segment edit fails, continue with others?
+   - Return original audio for failed edits?
+
+## Implementation Order
+
+1. **Phase 1: Tag Parsing**
+   - Extend `step_audio_editx_special_tags.py` with iteration support
+   - Add `parse_edit_tags_with_iterations()` function
+   - Support pipe syntax `<tag1|tag2>` and separate tags
+
+2. **Phase 2: Segment Integration**
+   - Extend `segment_parameters.py` to extract and store edit_tags
+   - Ensure clean text (tags stripped) flows to TTS
+
+3. **Phase 3: Post-Processor**
+   - Create `utils/audio/edit_post_processor.py`
+   - Lazy engine loading
+   - Duration validation with warnings
+   - Sequential edit application
+
+4. **Phase 4: Integration**
+   - Add single call in `tts_text_node.py`
+   - Add single call in `tts_srt_node.py`
+   - Test with multiple engines
+
+5. **Phase 5: Testing**
+   - Single tag, multiple tags
+   - Mixed syntax (pipe + separate)
+   - Duration edge cases
+   - SRT mode
+   - Multiple engines

docs/Dev reports/TIER2-3_REFACTOR_PLAN.md

@@ -0,0 +1,241 @@
+# Model Loading Tier 2-3 Refactor Plan
+
+## Overview
+Complete the model loading architecture refactor by:
+1. **Tier 2**: Consolidate caching, standardize error handling
+2. **Tier 3**: Standardize factory signatures, implement capability registry
+
+## TIER 2: Consolidation & Standardization
+
+### 2.1 Remove SmartModelLoader Usage
+
+**Current State**: SmartModelLoader still used in:
+- `nodes/chatterbox/chatterbox_tts_node.py` (lines with smart_model_loader)
+- `engines/chatterbox/streaming_model_manager.py`
+- `engines/chatterbox_official_23lang/streaming_model_manager.py`
+- `engines/adapters/f5tts_streaming_adapter.py`
+
+**Why Remove**:
+- Creates third caching system (after unified interface + ComfyUI manager)
+- Causes cache synchronization issues
+- Delegates to manager.load_tts_model() anyway (redundant)
+
+**Approach**:
+1. ChatterBox TTS node: Replace smart_model_loader call with direct unified interface
+2. Streaming processors: Use unified interface for model loading
+3. Verify no other code depends on SmartModelLoader returns
+
+**Risk**: Medium - streaming processors have complex state management
+
+---
+
+### 2.2 Standardize Error Handling
+
+**Current State**: 5 different error patterns across factories
+
+**Target State**: Single exception hierarchy:
+```python
+class TTSModelLoadingError(Exception):
+    """Base exception for model loading"""
+    pass
+
+class TTSModelNotFoundError(TTSModelLoadingError):
+    """Model not found locally or on HuggingFace"""
+    pass
+
+class TTSDeviceError(TTSModelLoadingError):
+    """Device-related issues"""
+    pass
+
+class TTSModelFormatError(TTSModelLoadingError):
+    """Model format compatibility issues"""
+    pass
+```
+
+**Files to Update**:
+- Create `utils/models/exceptions.py` with exception classes
+- Update all factories to use consistent exceptions
+- Update node error handlers to catch standardized exceptions
+
+**Risk**: Low - backwards compatible
+
+---
+
+### 2.3 Unified Cache Key Generation
+
+**Current State**: Multiple cache key formats:
+- Unified interface: string concatenation
+- SmartModelLoader: MD5 hashing
+- Nodes: MD5 of dict items
+
+**Target State**: Single hash-based cache key function
+
+```python
+def generate_model_cache_key(
+    engine_name: str,
+    model_name: str,
+    device: str,
+    language: Optional[str] = None,
+    **extra_params
+) -> str:
+    """Generate standardized cache key"""
+```
+
+**Files to Create/Update**:
+- `utils/models/cache_key_generator.py` (new)
+- Update unified_model_interface.py to use it
+- Update all node caching to use it
+
+**Risk**: Low
+
+---
+
+## TIER 3: Architecture Standardization
+
+### 3.1 Factory Parameter Dataclass
+
+**Current State**: Each factory has different parameters
+
+**Target State**: All factories accept standardized dataclass:
+
+```python
+@dataclass
+class ModelLoadConfig:
+    """Standard configuration for all model factories"""
+    # Required
+    device: str  # "auto", "cuda", "cpu", etc.
+    model_name: str
+
+    # Optional (engine-specific)
+    language: Optional[str] = None
+    model_path: Optional[str] = None
+    repo_id: Optional[str] = None
+
+    # Additional params (for quantization, optimization, etc.)
+    additional_params: Optional[Dict[str, Any]] = None
+
+    def __post_init__(self):
+        # Resolve device before storing
+        from utils.device import resolve_torch_device
+        self.device = resolve_torch_device(self.device)
+```
+
+**Factory Signature Becomes**:
+```python
+def engine_factory(config: ModelLoadConfig) -> Any:
+    """All factories use same signature"""
+```
+
+**Files to Create/Update**:
+- `utils/models/factory_config.py` (new)
+- All factories in unified_model_interface.py
+- Unified interface load_model() method
+
+**Risk**: High - changes all factory signatures
+
+---
+
+### 3.2 Engine Capability Registry
+
+**Current State**: Hardcoded engine-specific logic in model_manager.py
+
+**Target State**: Data-driven engine capabilities:
+
+```python
+ENGINE_CAPABILITIES = {
+    "chatterbox": {
+        "supports_voice_conversion": True,
+        "has_fallback": True,
+        "can_corrupt": False,
+        "recovery_handler": None
+    },
+    "higgs_audio": {
+        "supports_voice_conversion": False,
+        "has_fallback": False,
+        "can_corrupt": True,
+        "recovery_handler": reset_higgs_cuda_graphs,
+        "requires_special_init": True
+    },
+    # ... all 7 engines
+}
+```
+
+**Files to Create**:
+- `utils/models/engine_registry.py` (new) - define capabilities
+- Remove hardcoded engine checks from model_manager.py
+- Use registry for generic recovery
+
+**Risk**: Medium - affects error recovery paths
+
+---
+
+### 3.3 Unified Fallback Handler
+
+**Current State**: ChatterBox has sophisticated fallback, others don't
+
+**Target State**: Generic fallback utility any engine can use:
+
+```python
+class ModelFallbackHandler:
+    """Generic fallback chain for model loading"""
+
+    def __init__(self, primary_loader: Callable):
+        self.primary_loader = primary_loader
+        self.fallback_chain = []
+
+    def add_fallback(self, fallback_loader: Callable, condition: Callable):
+        """Add fallback with condition"""
+        pass
+
+    def load(self) -> Any:
+        """Execute fallback chain"""
+        pass
+```
+
+**Files to Create**:
+- `utils/models/fallback_handler.py` (new)
+
+**Risk**: Low - new utility, no breaking changes
+
+---
+
+## Implementation Order
+
+1. **First**: Create exception classes (no breaking changes)
+2. **Second**: Create cache key generator (parallel with tier 1 utilities)
+3. **Third**: Create factory config dataclass
+4. **Fourth**: Update all factories to use ModelLoadConfig
+5. **Fifth**: Remove SmartModelLoader usage
+6. **Sixth**: Create engine registry
+7. **Seventh**: Implement unified fallback handler
+8. **Eighth**: Update model_manager.py to use registry
+
+---
+
+## Risk Assessment
+
+| Task | Risk | Dependencies | Effort |
+|------|------|--------------|--------|
+| Exception classes | Low | None | 1 hr |
+| Cache key generator | Low | Exceptions | 1 hr |
+| Factory config dataclass | Medium | None | 2 hrs |
+| Update factories | High | Config, exceptions | 4 hrs |
+| Remove SmartModelLoader | Medium | Updated factories | 2 hrs |
+| Engine registry | Medium | Updated factories | 2 hrs |
+| Fallback handler | Low | Exceptions | 2 hrs |
+| Integration testing | High | All above | 3 hrs |
+
+**Total**: ~17 hours of work
+
+---
+
+## Success Criteria
+
+- [ ] All factories use consistent error handling
+- [ ] Cache key generation centralized and consistent
+- [ ] SmartModelLoader completely removed
+- [ ] Engine capabilities defined in registry
+- [ ] All 7 engines working without issues
+- [ ] Cache hits work correctly across device switches
+- [ ] Error messages are consistent and helpful
+- [ ] No VRAM spikes or device mismatches

docs/Dev reports/TODO.md

@@ -0,0 +1,139 @@
+# Project Refactoring TODO
+
+This document tracks architectural issues and inconsistencies that need refactoring to improve modularity and reduce code duplication.
+
+## SRT Processing Architecture Issues
+
+### Problem: Inconsistent SRT Implementation Approaches
+Different engines use completely different patterns for SRT processing:
+
+1. **ChatterBox (Old)**: Uses `ChatterboxSRTTTSNode` class (should be processor)
+2. **VibeVoice**: Uses proper `VibeVoiceSRTProcessor` class with full implementation
+3. **Higgs Audio**: Uses clean modular approach with timing utilities
+4. **ChatterBox Official 23-Lang**: Follows Higgs Audio pattern (good)
+5. **F5-TTS**: Uses `F5TTSSRTNode` class (should be processor)
+
+**Impact**: Code duplication, inconsistent interfaces, hard to maintain
+**Solution**: Standardize on processor pattern with shared utilities
+
+### Problem: Node vs Processor Naming Inconsistency
+- ChatterBox uses `ChatterboxSRTTTSNode` but should be `ChatterboxSRTProcessor`
+- F5-TTS uses `F5TTSSRTNode` but should be `F5TTSSRTProcessor`
+- Only VibeVoice and ChatterBox Official 23-Lang use proper processor naming
+
+**Impact**: Confusing architecture, breaks expected patterns
+**Solution**: Rename all SRT classes to use `*Processor` pattern
+
+### Problem: Duplicated SRT Logic
+Each engine reimplements:
+- SRT parsing
+- Character switching within SRT segments  
+- Timing calculations
+- Audio assembly
+- Report generation
+
+**Impact**: Massive code duplication, bugs in some implementations
+**Solution**: Create shared SRT base class with common functionality
+
+## Voice Reference Handling Issues
+
+### Problem: Inconsistent Audio Reference Processing
+- ChatterBox: Uses file paths only, converts tensors to temp files via `handle_reference_audio`
+- F5-TTS: Uses both file paths and ComfyUI tensors directly
+- Higgs Audio: Uses ComfyUI tensors with reference text
+- VibeVoice: Uses ComfyUI tensors
+
+**Impact**: Different interfaces, conversion logic scattered
+**Solution**: Standardize voice reference handling in base class
+
+## Character Parser Integration Issues
+
+### Problem: Inconsistent Language Resolution
+- Some engines hardcode language mappings
+- Some use character parser properly
+- Some bypass language mapper entirely
+- ChatterBox Official 23-Lang now uses proper integration (good example)
+
+**Impact**: Language switching behaves differently per engine
+**Solution**: Mandate character parser usage in base class
+
+### Problem: Engine-Specific Logic in Universal Character Parser
+- **Issue**: Universal character parser contains ChatterBox-specific Italian prefix logic (`apply_italian_prefix_if_needed`)
+- **Problem**: Engine-specific text processing belongs in engine adapters, not universal parser
+- **Current State**: Italian `[it]` prefix filter has complex conditions but should only affect:
+  - Explicit `[it:Alice]` or `[italian:Bob]` tags
+  - Characters mapped to Italian in alias system
+  - Should NOT affect normal English ChatterBox, other engines, or global model selection
+- **Impact**: Violation of separation of concerns, coupling universal parser to specific engine quirks
+- **Solution**: Move Italian prefix logic to ChatterBox adapter, keep character parser engine-agnostic
+- **Priority**: Medium (works correctly but violates clean architecture)
+- **Files Affected**: 
+  - `utils/text/character_parser/` (remove Italian logic)
+  - `engines/adapters/chatterbox_adapter.py` (add Italian logic)
+
+## Cache System Issues
+
+### Problem: Engine-Specific Cache Keys
+- Each engine uses different cache identifiers
+- Cache collision was discovered between ChatterBox engines
+- No standardized cache invalidation
+
+**Impact**: Cross-engine cache pollution, missed cache opportunities  
+**Solution**: Implement unified cache system with proper namespacing
+
+## Base Class Architecture Issues
+
+### Problem: Inconsistent Base Class Usage
+- Some engines properly inherit from `BaseTTSNode`
+- Some engines duplicate base class functionality
+- Method signatures vary between engines for same functionality
+
+**Impact**: Code duplication, inconsistent interfaces
+**Solution**: Enforce proper base class inheritance and standardize method signatures
+
+## Import System Issues
+
+### Problem: Different Import Patterns
+- Some use direct imports
+- Some use `importlib.util.spec_from_file_location`
+- Some use robust_import
+- Inconsistent path resolution
+
+**Impact**: Import failures, maintenance overhead
+**Solution**: Standardize import patterns project-wide
+
+## Error Handling Inconsistencies
+
+### Problem: Different Error Handling Approaches
+- Some engines have comprehensive error handling
+- Some engines crash on edge cases
+- Different error message formats
+- Inconsistent fallback mechanisms
+
+**Impact**: Poor user experience, difficult debugging
+**Solution**: Implement standardized error handling framework
+
+## Configuration Management Issues
+
+### Problem: Scattered Configuration Handling
+- Engine configs stored differently
+- Parameter validation inconsistent
+- Default values scattered across files
+- No centralized config schema
+
+**Impact**: Bugs from invalid configs, hard to maintain defaults
+**Solution**: Implement unified configuration management system
+
+---
+
+## Refactoring Priority
+
+1. **High Priority**: SRT processing standardization (affects user experience directly)
+2. **Medium Priority**: Voice reference handling (causes confusion)
+3. **Medium Priority**: Cache system unification (performance impact)
+4. **Low Priority**: Import system standardization (maintenance burden)
+
+---
+
+*Last Updated: 2025-01-XX*
+*Add new issues to this file as they are discovered during development*

docs/Dev reports/VIBEVOICE_INTEGRATION_PLAN.md

@@ -0,0 +1,209 @@
+# VibeVoice Integration Plan - Final Version
+
+## Overview
+
+Integrate Microsoft VibeVoice (1.5B and 7B models) into TTS Audio Suite following the unified architecture, with full support for all existing features while exposing VibeVoice's unique multi-speaker capabilities.
+
+## Implementation Structure
+
+### 1. Engine Implementation (`engines/vibevoice/`)
+
+#### `vibevoice.py` (Main Engine - ~500 lines)
+
+```python
+class VibeVoiceEngine:
+    def __init__(self)
+    def initialize_engine(model_name, device)
+    def generate_speech(text, voice_samples, cfg_scale, seed, ...)
+    def generate_multi_speaker(segments, voice_samples, ...)
+    def cleanup()  # Memory management
+    def unload_models()  # Explicit unload (called by ComfyUI's unload button)
+```
+
+- Use `unified_model_interface` for model loading
+- Integrate with `utils.audio.cache` for caching
+- Support both models (1.5B and 7B)
+- Handle memory management via ComfyUI's unload system
+
+#### `vibevoice_downloader.py` (~200 lines)
+
+```python
+VIBEVOICE_MODELS = {
+    "vibevoice-1.5B": {
+        "repo": "microsoft/VibeVoice-1.5B",
+        "files": [...],  # 5.4GB total
+    },
+    "vibevoice-7B": {
+        "repo": "microsoft/VibeVoice-Large",
+        "files": [...],  # 9.3GB total
+    }
+}
+```
+
+- Use `unified_downloader` for all downloads
+- Download to `models/TTS/vibevoice/` structure
+- No HuggingFace cache duplication
+
+### 2. Engine Configuration Node (`nodes/engines/vibevoice_engine_node.py`) (~150 lines)
+
+```python
+class VibeVoiceEngineNode:
+    INPUT_TYPES:
+        Required:
+            - model: ["VibeVoice-1.5B", "VibeVoice-7B"]
+            - device: ["auto", "cuda", "cpu"]
+            - multi_speaker_mode: ["Custom Character Switching", "Native Multi-Speaker"]
+            - cfg_scale: (1.0-2.0, default 1.3)
+            - use_sampling: BOOLEAN
+            - temperature: (0.1-2.0, default 0.95)
+            - top_p: (0.1-1.0, default 0.95)
+            - chunk_minutes: INT (0=disabled, UI shows minutes, backend converts to chars)
+        Optional:
+            - speaker2_voice: AUDIO
+            - speaker3_voice: AUDIO
+            - speaker4_voice: AUDIO
+```
+
+- Returns TTS_ENGINE configuration dict
+- NO seed parameter (that's in unified nodes)
+- NO load/unload toggles (handled by ComfyUI system)
+- Uses same naming as Higgs: "Custom Character Switching" not "segment_per_character"
+- Follows ChatterBox/Higgs patterns exactly
+
+### 3. Engine Adapter (`engines/adapters/vibevoice_adapter.py`) (~400 lines)
+
+```python
+class VibeVoiceEngineAdapter:
+    def __init__(self, node_instance)
+    def get_model_for_language(lang_code, default_model)
+    def load_base_model(model_name, device)
+    def generate_segment(text, voice_ref, params)
+    def process_character_segments(segments, voice_mapping)
+    def convert_character_to_speaker_format(text)  # [Alice] → Speaker 0
+    def handle_pause_tags(text)  # Use pause_processor
+    def handle_language_tags(text)  # Parse [de:Alice] syntax
+```
+
+- Bridge unified nodes to VibeVoice engine
+- Two modes matching Higgs naming:
+  - `Custom Character Switching`: Generate per character (like ChatterBox/Higgs)
+  - `Native Multi-Speaker`: Single generation with multiple speakers
+- Character mapping: [Alice], [Bob] → Speaker 0, Speaker 1
+- Full integration with existing systems
+
+### 4. Internal Processors (`nodes/vibevoice/`)
+
+#### `vibevoice_processor.py` (~300 lines)
+
+- Internal TTS processor (called by unified TTS node)
+- Handles actual generation orchestration
+- Manages chunking (time-based UI, character-based backend)
+- Memory management coordination
+
+#### `vibevoice_srt_processor.py` (~400 lines)
+
+- Internal SRT processor (called by unified SRT node)
+- Language grouping optimization
+- Multi-speaker dialogue handling
+- Timing synchronization
+
+### 5. Model Factory Registration (~50 lines in existing files)
+
+In `utils/models/unified_model_interface.py`:
+
+```python
+# Register VibeVoice model factories
+interface.register_model_factory(
+    "vibevoice", "tts", 
+    lambda **kwargs: load_vibevoice_model(**kwargs)
+)
+```
+
+### 6. Dependencies Update
+
+`requirements.txt`:
+
+```
+# VibeVoice dependencies
+transformers>=4.44.0
+git+https://github.com/microsoft/VibeVoice.git
+```
+
+## Key Integration Points
+
+### Memory Management
+
+- Memory unloading handled by ComfyUI's "Unload Models" button
+- Engine implements `unload_models()` method
+- Uses ComfyUI's model management system
+- Auto-cleanup on node destruction
+
+### Unified Systems Integration
+
+- **Cache**: Use `utils.audio.cache` with content-based hashing
+- **Characters**: Full `utils.text.character_parser` support
+- **Pause Tags**: `utils.text.pause_processor` integration
+- **Downloads**: `utils.downloads.unified_downloader` only
+- **Model Loading**: `utils.models.unified_model_interface`
+- **Chunking**: `utils.text.chunking` with time-based UI
+
+### Two Generation Modes (Named Like Higgs)
+
+1. **Custom Character Switching** (default):
+   
+   - Preserves all TTS Audio Suite features
+   - Generate audio per character, then combine
+   - Supports pause tags, language switching per character
+   - More flexible but slower
+
+2. **Native Multi-Speaker**:
+   
+   - Uses VibeVoice's native multi-speaker
+   - Single generation pass
+   - More efficient but less flexible
+   - Auto-converts [Character] → Speaker N format
+
+### Time-Based Chunking
+
+- UI shows minutes: "Chunk every [5] minutes"
+- Backend converts: 5 min → ~3750 chars (150 wpm * 5 chars/word * 5)
+- More intuitive for long-form content
+- Optional (0 = no chunking)
+
+### Seed Parameter Location
+
+- Seed is in unified nodes (TTS Text, TTS SRT) NOT in engine node
+- Follows existing pattern where engine provides configuration, unified nodes handle generation parameters
+
+## File Size Limits
+
+- Each file stays under 500-600 lines
+- Modular design with clear separation
+- Reuses all existing utilities
+- No code duplication
+
+## Testing Plan
+
+1. Test both 1.5B and 7B models
+2. Validate character switching in both modes
+3. Test pause tag processing
+4. Verify memory management via ComfyUI unload
+5. Test long-form generation with chunking
+6. Validate multi-speaker native mode
+7. Test cache functionality
+8. Verify SRT synchronization
+
+## Benefits
+
+- Full integration with all TTS Audio Suite features
+- Choice between flexibility (Custom mode) and efficiency (Native mode)
+- Proper memory management via ComfyUI system
+- No code duplication
+- Follows established patterns exactly (Higgs Audio style)
+- Exposes unique 90-minute and 4-speaker capabilities
+
+## Implementation Notes
+
+- Created: 2025-08-29
+- Author: Claude (TTS Audio Suite Assistant)
+- Status: Ready for implementation

docs/Dev reports/fails_to_avoid_TTS_Engine_Implementation.md

@@ -0,0 +1,111 @@
+# TTS Engine Implementation - Fails & Lessons Learned
+
+## Step Audio EditX Implementation
+
+### Import Errors
+- **Missing import**: Always add `import folder_paths` when using `folder_paths.get_temp_directory()` in node files
+- **Bundled code imports**: For complex bundled packages with internal cross-imports, add `sys.path.insert(0, impl_dir)` at top of main files instead of converting all imports
+
+### Audio Utility Functions
+- **Temp file creation**: Use `AudioProcessingUtils.save_audio_to_temp_file()` not `save_audio()` (doesn't exist)
+- **ComfyUI audio format**: Extract waveform with `audio_tensor['waveform']` and `audio_tensor.get('sample_rate')` before passing to utilities
+
+### Download Configuration
+- **HuggingFace file list**: Always verify actual repo structure with `curl` before hardcoding file lists
+- **Downloader API**: Use dict format `[{"remote": f, "local": f}]` for unified_downloader, not plain strings
+- **File existence**: Check actual HuggingFace repo structure - don't assume file naming patterns (e.g., `model-00001.safetensors` not `model-00001-of-00002.safetensors`)
+- **CRITICAL - No cache downloads**: NEVER allow models to auto-download to cache directories. ALL models/weights must be downloaded via our downloader to organized `models/TTS/` folders. Disable auto-download and add to downloader instead.
+
+### Factory Registration
+- **Unified model interface**: Always implement `register_<engine>_factory()` when using `unified_model_interface.load_model()`
+- **Factory initialization**: Add factory to `initialize_all_factories()` or it won't be registered
+
+### Node Registration
+- **Engine branch**: Add engine-specific branch in `_create_engine_node_instance()` for each new engine
+- **Variable naming**: Use consistent variable names (`config` not `engine_config`) throughout node code
+
+### Voice References
+- **Reference text requirement**: F5-TTS and Step Audio EditX REQUIRE `prompt_text` (transcript), ChatterBox/VibeVoice/Higgs don't
+- **Narrator voice mapping**: Map narrator from TTS Text node input by saving `audio_tensor` to temp file with `reference_text`
+- **Voice discovery**: Use `get_character_mapping()` not `discover_voices_for_engine()` (latter doesn't exist)
+
+### Pause Tag Format
+- **Correct format**: Use `[pause:2]`, `[pause:1.5s]`, `[pause:500ms]` - NOT `<pause_X>`
+- **Parser usage**: Use `PauseTagProcessor.parse_pause_tags()` which returns `('text', content)` or `('pause', duration_seconds)` tuples
+
+### Seed Control
+- **Global torch state**: Some engines (Step Audio EditX) use global `torch.manual_seed()` for reproducibility, not function parameters
+- **CUDA seeds**: Always set both `torch.manual_seed()` and `torch.cuda.manual_seed_all()` for GPU reproducibility
+
+### Quantization Support
+- **Device movement**: Bitsandbytes quantized models (int4/int8) CANNOT be moved with `.to()` - wrap in try-except
+- **VRAM clearing**: Quantized models stay on GPU; "Clear VRAM" must unload completely (not move to CPU)
+- **Error handling**: Catch `ValueError` with "is not supported for" and "8-bit"/"4-bit" in message
+
+### Character Switching Implementation
+- **Missing voice fallback**: When `get_character_mapping()` returns `(None, None)` for a character, MUST still add entry to `voice_mapping` with fallback to narrator/default voice
+- **Voice mapping consistency**: All characters in parsed segments MUST have entries in `voice_mapping`, even if empty/None
+- **Adapter validation**: Adapters that require voice references (Step Audio EditX, F5-TTS) need graceful fallback when voice_ref is None
+- **Character parser interaction**: Character parser changes unknown characters to "narrator" - need to ensure original character names are preserved for voice mapping lookup
+- **Parser fix**: Add text tag characters to `available_characters` (lowercase) + set language defaults like IndexTTS
+- **Working pattern**: `all_available = set(get_available_characters())` + aliases + text tag chars (lowercase) + "narrator" → `set_available_characters(list(all_available))` + `set_character_language_default()` for each char
+
+### Caching Implementation
+- **CRITICAL - Planned but skipped**: Caching listed in Phase 1 but often omitted during processor/adapter implementation. Verify with testing (run generation twice on same text - should show cache hit on second)
+- **Required in adapters**: Generate stable audio hash, create cache key with all params, check cache BEFORE generation, store after with duration calc
+- **Cache key pattern**: `audio_cache.generate_cache_key(engine_type, text=..., audio_component=..., **all_params)`
+- **Duration calculation**: Update `_calculate_duration()` with engine sample rate (e.g., 24000 for Step Audio EditX, F5-TTS)
+
+### Model Lifecycle - __del__ Destructor
+- **CRITICAL**: Remove `__del__` from engine classes - causes automatic unload after generation ends (when object goes out of scope)
+- **Pattern**: F5-TTS and ChatterBox don't have `__del__`, only IndexTTS and StepAudio did (wrong)
+- **Correct behavior**: Models stay in VRAM for reuse; only unload on explicit Clear VRAM button, engine switch, or parameter change
+
+### Generation Info Reports - Modular Pattern
+- **CRITICAL - Consistency required**: ALL engines must use `ChunkCombiner` + `ChunkTimingHelper` for generation reports
+- **Correct pattern** (from ChatterBox 23-Lang, F5-TTS):
+  1. `combine_audio_segments()` method accepts `return_info=True` parameter
+  2. Uses `ChunkCombiner.combine_chunks()` with `return_info=True` to get `(audio, chunk_info)` tuple
+  3. In unified node: call `ChunkTimingHelper.enhance_generation_info(base_info, chunk_info)` for detailed timing
+- **Wrong**: Inline concatenation (`torch.cat`) without timing tracking, manual report formatting
+- **Example**: IndexTTS/StepAudio initially did inline concat - needed refactor to use ChunkCombiner
+- **Required info**: Duration, text length, segments/chunks count, chunk timing breakdown (start/end/text per chunk)
+
+### Segment Parameter Registration
+- **Forgot to add**: New engines to `PARAMETER_ENGINES` in `utils/text/segment_parameters.py` for `[seed:X]` tag support
+
+### SRT Processor Import Pattern
+- **Wrong**: Using `from nodes.xxx import` - fails with `'nodes' is not a package`
+- **Correct**: Use `importlib.util.spec_from_file_location()` for cross-module imports in dynamically loaded processors
+
+### ComfyUI Audio Dict Format
+- **Wrong**: Passing `audio_tensor` directly to functions expecting raw tensor
+- **Correct**: Extract waveform with `audio_tensor.get('waveform')` - ComfyUI audio is `{"waveform": tensor, "sample_rate": int}`
+- **Also**: Check `AudioProcessingUtils.save_audio_to_temp_file(audio, sample_rate)` signature - it returns path, doesn't take path as arg
+
+### ComfyUI Audio Tensor Shape
+- **CRITICAL**: ComfyUI expects **3D** `[batch, channels, samples]` NOT 2D `[channels, samples]`
+- **Error symptom**: `IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)` in `save_audio()`
+- **Correct shape conversion**:
+  - 1D `[samples]` → 3D `[1, 1, samples]` (two unsqueeze)
+  - 2D `[channels, samples]` → 3D `[1, channels, samples]` (one unsqueeze)
+- **Wrong**: Squeezing 3D to 2D, or only adding one dimension to 1D
+- **Pattern**: `tensor.unsqueeze(0)` adds batch dim to 2D; check VibeVoice processor for reference
+
+### Step Audio EditX Sample Rate
+- **Symptom**: Pitch shift in edit/clone modes with external audio (44100 Hz, 48000 Hz)
+- **Cause**: Saving audio with wrong sample rate metadata → `torchaudio.load()` misinterprets data
+- **Fix**: Resample to 24000 Hz before saving temp files (edit node, TTS text node, TTS SRT node)
+- **Prevention**: Always resample to engine's native rate (24000 Hz for Step Audio EditX)
+
+### Unified Model Interface - Adapter & Node Consistency
+- **CRITICAL**: ALL engine loading must use `unified_model_interface.load_model()` for smart caching
+- **Symptom**: Model loads twice (once for TTS, once for editing/other nodes) - wasting VRAM and time
+- **Wrong pattern**: Creating engine instances directly (`engine = StepAudioEditXEngine()`) in adapters or nodes
+- **Correct pattern**:
+  1. **Adapter** `load_base_model()` uses `unified_model_interface.load_model(config)`
+  2. **Audio Editor node** uses `unified_model_interface.load_model(config)`
+  3. **Both use same `model_type`** (e.g., "tts") so cache keys match → engine reused
+- **Cache key components**: `engine_name`, `model_type`, `model_name`, `device`, `quantization`, etc.
+- **Fix Step Audio EditX**: Updated adapter and Audio Editor node to use unified interface with `model_type="tts"`
+- **Prevention**: When adding new engines, ALWAYS use unified interface in ALL code paths (adapters, nodes, processors)

docs/INDEXTTS2_UPDATE_LOG.md

@@ -0,0 +1,92 @@
+# IndexTTS-2 Updates Log
+
+This document tracks updates applied to our bundled IndexTTS-2 code from the upstream repository.
+
+## Update Process
+1. **Check for upstream changes:** `cd IgnoredForGitHubDocs/For_reference/index_tts && git pull`
+2. **Review changes:** `git diff OLD_COMMIT..NEW_COMMIT -- indextts/infer_v2.py`
+3. **Apply relevant changes** to `engines/index_tts/indextts/infer_v2.py`
+4. **Test functionality** with our IndexTTS-2 nodes
+5. **Document changes** in this log
+
+---
+
+## 2025-09-18: Major Update - Cache & Emotion Improvements
+
+**Reference commit range:** `8336824..64cb31a` (September 11 → September 18, 2025)
+**Upstream commits reviewed:** 5 commits affecting `infer_v2.py`
+
+### 🔧 Changes Applied:
+
+#### 1. **Emotion Vector Normalization Function** (commit: 8aa8064)
+- **Added:** `normalize_emo_vec()` helper function
+- **Purpose:** Prevents voice identity loss from overly strong emotion settings
+- **Implementation:**
+  - Applies emotion bias factors to reduce problematic emotions
+  - Caps total emotion sum to 0.8 maximum
+  - Protects against extreme emotion vector values
+- **User benefit:** Better emotion control without losing speaker characteristics
+
+#### 2. **Audio Length Limiting** (commit: 0828dcb)
+- **Added:** `_load_and_cut_audio()` helper function
+- **Purpose:** Prevents memory/VRAM issues from overly long reference audio
+- **Implementation:**
+  - Automatically truncates audio to 15 seconds maximum
+  - Supports different sample rates (16kHz, 22kHz)
+  - Provides verbose logging of truncation
+- **User benefit:** More stable generation with long reference audio files
+
+#### 3. **Persistent Cache Buildup Fix** (commit: 64cb31a)
+- **Added:** Proper cache clearing with `torch.cuda.empty_cache()`
+- **Purpose:** Solves memory accumulation issues during multiple generations
+- **Implementation:**
+  - Clears old cache variables before loading new audio
+  - Calls `torch.cuda.empty_cache()` to free GPU memory
+  - Applied to both speaker and emotion cache invalidation
+- **User benefit:** Better memory management during batch processing
+
+#### 4. **BigVGAN CUDA Kernel Import Fix** (commits: ee23371, e409c4a)
+- **Fixed:** Corrected import path for BigVGAN custom CUDA kernel
+- **Changed:** `indextts.BigVGAN.alias_free_activation.cuda` → `indextts.s2mel.modules.bigvgan.alias_free_activation.cuda`
+- **Purpose:** Ensures proper CUDA acceleration when available
+- **User benefit:** Better performance with CUDA-enabled setups
+
+### 📋 Implementation Status:
+
+| Change | Applied to Bundled Code | Tested | Notes |
+|--------|------------------------|--------|-------|
+| `normalize_emo_vec()` | ✅ **Applied** | ⏳ Pending | Core emotion control improvement |
+| `_load_and_cut_audio()` | ✅ **Applied** | ⏳ Pending | Memory stability fix |
+| Cache clearing fixes | ✅ **Applied** | ⏳ Pending | Critical for batch processing |
+| BigVGAN import fix | ✅ **Applied** | ⏳ Pending | Performance improvement |
+
+### 🎯 User-Facing Improvements:
+- **Better emotion control:** Emotion vectors now auto-normalize to prevent voice identity loss
+- **Memory stability:** Long reference audio automatically truncated to prevent crashes
+- **Batch processing:** Improved cache management for multiple generations
+- **Performance:** Fixed CUDA kernel loading for faster inference
+
+### 🧪 Testing Required:
+- [ ] Emotion vector controls in radar chart widget
+- [ ] Long reference audio handling (>15 seconds)
+- [ ] Multiple sequential generations (cache clearing)
+- [ ] CUDA performance with FP16 enabled
+
+---
+
+## Next Update Check: 2025-10-18
+
+**Monitoring:** Watch for commits to `indextts/infer_v2.py` in https://github.com/index-tts/index-tts
+
+**Update command:**
+```bash
+cd IgnoredForGitHubDocs/For_reference/index_tts
+git pull origin main
+git log --oneline LAST_COMMIT..HEAD -- indextts/infer_v2.py
+```
+
+**Priority changes to watch for:**
+- Performance optimizations
+- Memory management improvements
+- Bug fixes in emotion processing
+- New model loading features

docs/INDEXTTS_IMPLEMENTATION_PLAN.md

@@ -0,0 +1,107 @@
+# IndexTTS-2 Implementation Plan
+
+## Overview
+IndexTTS-2 is a breakthrough emotionally expressive and duration-controlled autoregressive zero-shot TTS engine featuring emotion disentanglement and precise voice cloning.
+
+## Unique Features
+- **Duration Control**: First autoregressive TTS model with precise duration control
+- **Emotion Disentanglement**: Separates speaker identity from emotional expression
+- **Multi-Modal Emotion Control**: Audio references, text descriptions, manual vectors
+- **Zero-Shot Voice Cloning**: Single reference audio for voice cloning
+- **Advanced Character Switching**: `[character:emotion_ref]` syntax
+
+## Technical Specifications
+- **Sample Rate**: 22,050 Hz
+- **Audio Format**: `torch.Tensor` shape `[1, samples]`
+- **Dependencies**: `torch==2.8.*`, `transformers==4.52.1`, IndexTTS-2 source code
+- **Model Components**: GPT + Semantic codec + S2Mel + BigVGAN + CAMPPlus + QwenEmotion
+
+## Implementation Status
+
+### ✅ Completed
+1. **Core Engine** (`engines/index_tts/index_tts.py`)
+   - IndexTTSEngine class with unified model interface integration
+   - Proper UnifiedModelInterface integration for loading/unloading
+   - Factory registration in unified_model_interface.py
+   - All official parameters exposed (emotion vectors, text emotions, generation params)
+
+2. **Downloader** (`engines/index_tts/index_tts_downloader.py`)
+   - Uses folder_paths.models_dir for correct TTS/IndexTTS/ structure
+   - unified_downloader integration 
+   - Downloads both HuggingFace model files and GitHub source code
+   - Model verification system
+
+3. **Cache Integration** (`utils/audio/cache.py`)
+   - IndexTTSCacheKeyGenerator added to unified cache system
+   - Handles all IndexTTS-2 parameters for cache key generation
+   - Integrated with global audio_cache
+
+4. **Engine Adapter** (`engines/adapters/index_tts_adapter.py` - IN PROGRESS)
+   - Character switching support with emotion references
+   - Cache integration and parameter mapping
+   - Uses existing CharacterParser for tag parsing
+
+### 🚧 In Progress  
+5. **Character Tag Extension**
+   - Need to handle flexible syntax: `[de:]`, `[de:Alice]`, `[Alice]`, `[Alice:angry_bob]`, `[de:Alice:angry_bob]`
+   - Extend CharacterParser or create IndexTTS-specific parsing
+   - Avoid ambiguity between language codes and character names
+
+### 📋 Pending
+6. **Engine Configuration Node** (`nodes/engines/index_tts_engine_node.py`)
+   - UI node with all IndexTTS-2 parameters
+   - Model path selection with auto-download
+   - Emotion controls (audio references, vectors, text)
+   - Generation parameters (temperature, top_p, top_k, etc.)
+
+7. **TTS Text Integration** (`nodes/index_tts/`)
+   - `index_tts_processor.py` - Main TTS processor for unified TTS Text node
+   - `index_tts_srt_processor.py` - SRT subtitle processing with timing
+   - Integration with character switching system
+
+8. **Testing & Validation**
+   - Basic functionality testing
+   - Character switching validation  
+   - Emotion control verification
+   - Performance optimization
+
+## Character Switching Syntax Design
+
+### Proposed Flexible Syntax Resolution:
+```
+[de:]           → language="de", character="narrator", emotion=None
+[de:Alice]      → language="de", character="Alice", emotion=None  
+[Alice]         → language=None, character="Alice", emotion=None
+[Alice:angry_bob] → language=None, character="Alice", emotion="angry_bob"
+[de:Alice:angry_bob] → language="de", character="Alice", emotion="angry_bob"
+```
+
+### Resolution Strategy:
+1. Split by `:` and count parts
+2. 1 part: Check if it's a known language code, else treat as character
+3. 2 parts: Check if first part is language code, else treat as character:emotion
+4. 3 parts: Always language:character:emotion
+
+### Implementation Files:
+- Extend `utils/text/character_parser.py` with emotion support
+- Or create `engines/index_tts/emotion_parser.py` for IndexTTS-specific parsing
+
+## Model Management Integration
+
+### Unified Systems Used:
+- **UnifiedModelInterface**: Standardized loading/unloading
+- **unified_downloader**: HuggingFace + Git downloads  
+- **AudioCache**: Generation result caching
+- **folder_paths**: Correct model directory structure
+- **CharacterParser**: Tag parsing and language resolution
+
+### Memory Management:
+- ComfyUI model wrapper integration via UnifiedModelInterface
+- Automatic VRAM management and "Clear VRAM" button support
+- Model factory registration for proper lifecycle management
+
+## Next Steps:
+1. Resolve character tag ambiguity with flexible parsing
+2. Create IndexTTS engine configuration node
+3. Implement TTS Text processor integration
+4. Test full character switching + emotion control workflow

docs/INLINE_EDIT_TAGS_USER_GUIDE.md

@@ -0,0 +1,672 @@
+# Inline Edit Tags - User Guide
+
+## 💡 Inline vs Manual Node Workflow
+
+**ComfyUI is modular** - you can manually chain **TTS → 🎨 Step Audio EditX - Audio Editor** nodes for full control.
+
+**Why use inline tags instead?**
+- Apply edits to **specific segments** of text without multiple TTS→Edit→TTS chains
+- Convenient for **SRT files** with selective editing per subtitle
+- Automatic batch processing for efficiency
+
+**For maximum control and quality**, build manual workflows. **For convenience and selective segment editing**, use inline tags.
+
+---
+
+## Overview
+
+Inline edit tags allow you to apply **Step Audio EditX** post-processing effects to TTS-generated audio from ANY engine (F5-TTS, ChatterBox, Higgs Audio, VibeVoice, etc.). These tags are automatically stripped before TTS generation and applied as a second processing pass.
+
+**Example:**
+```
+[Alice] This is hilarious <Laughter:3>, I can't believe it!
+```
+
+**What happens:**
+1. TTS generates: "This is hilarious, I can't believe it!" (tag removed)
+2. Step Audio EditX edits the audio with 3 iterations of Laughter effect
+3. Final audio has natural laughter inserted
+
+---
+
+## Supported Edit Types
+
+### 1. Paralinguistic Tags (Sound Insertion)
+
+Insert non-verbal sounds at specific positions in the audio:
+
+| Tag | Effect | Example |
+|-----|--------|---------|
+| `<Laughter>` or `<Laughter:2>` | Insert laughter | `Hello <Laughter> world` |
+| `<Breathing>` | Insert breathing | `I'm tired <Breathing> let's rest` |
+| `<Sigh>` | Insert sigh | `Okay <Sigh> fine` |
+| `<Uhm>` | Insert "uhm" hesitation | `Well <Uhm> maybe` |
+| `<Surprise-oh>` | Surprised "oh!" | `<Surprise-oh> I didn't know!` |
+| `<Surprise-ah>` | Surprised "ah!" | `<Surprise-ah> That's amazing!` |
+| `<Surprise-wa>` | Surprised "wa!" | `<Surprise-wa> Incredible!` |
+| `<Confirmation-en>` | Confirmation sound | `<Confirmation-en> Yes` |
+| `<Question-ei>` | Question sound | `<Question-ei> Really?` |
+| `<Dissatisfaction-hnn>` | Dissatisfied "hnn" | `<Dissatisfaction-hnn> Not good` |
+
+**Position matters:** The sound is inserted where you place the tag.
+
+**💡 Pro Tip - Stronger Effects:** For more reliable and pronounced effects, include relevant text alongside the tag:
+```
+# Weaker effect
+I'm happy <Laughter>
+
+# Stronger, more guaranteed effect
+I'm laughing haha <Laughter:2>
+```
+
+The TTS-generated "haha" combined with the post-processing laughter creates a more natural and pronounced effect.
+
+### 2. Emotion Tags (Whole Segment)
+
+Apply emotional tone to the entire audio segment:
+
+`<emotion:VALUE>` or `<emotion:VALUE:ITERATIONS>`
+
+**Available emotions:** happy, sad, angry, excited, calm, fearful, surprised, disgusted, confusion, empathy, embarrass, depressed, coldness, admiration
+
+**Examples:**
+```
+[Alice] I'm so happy to see you! <emotion:happy>
+[Bob] This is terrible news. <emotion:sad:2>
+```
+
+**Position doesn't matter:** Tag can be anywhere in the text.
+
+### 3. Style Tags (Whole Segment)
+
+Apply speaking style to the entire audio segment:
+
+`<style:VALUE>` or `<style:VALUE:ITERATIONS>`
+
+**Available styles:** whisper, serious, child, older, girl, pure, sister, sweet, exaggerated, ethereal, generous, recite, act_coy, warm, shy, comfort, authority, chat, radio, soulful, gentle, story, vivid, program, news, advertising, roar, murmur, shout, deeply, loudly, arrogant, friendly
+
+**Examples:**
+```
+[Alice] Come closer, I have a secret <style:whisper>
+[Bob] Listen carefully to this. <style:serious:2>
+```
+
+**Position doesn't matter:** Tag can be anywhere in the text.
+
+### 4. Speed Tags (Whole Segment)
+
+Adjust speaking speed for the entire audio segment:
+
+`<speed:VALUE>` or `<speed:VALUE:ITERATIONS>`
+
+**Available speeds:** faster, slower, more_faster, more_slower
+
+**Examples:**
+```
+[Alice] Quick! We need to hurry! <speed:faster>
+[Bob] Let me think about this... <speed:slower:2>
+```
+
+**Position doesn't matter:** Tag can be anywhere in the text.
+
+### 5. Voice Restoration Tags (Whole Segment)
+
+Restore voice resemblance after edit effects using ChatterBox Official 23-Lang Voice Changer:
+
+`<restore>`, `<restore:N>`, or `<restore:N@M>`
+
+**Syntax:**
+- `<restore>` - 1 VC pass using original pre-edit audio as reference
+- `<restore:2>` - 2 VC passes using original pre-edit audio as reference (for stronger restoration)
+- `<restore:1@2>` - 1 VC pass using audio from iteration 2 as reference
+
+**⚠️ Why You Need This:**
+
+Edit effects (especially after multiple iterations) can **degrade voice quality and alter voice resemblance**:
+- Paralinguistic effects (`<Laughter:3>`) add sounds but can muddy the voice
+- Style effects (`<style:whisper:4>`) change timbre and can distort characteristics
+- Multiple iterations compound these effects
+
+The `<restore>` tag uses voice conversion to restore your original voice character while keeping the edit effects.
+
+**Examples:**
+
+**Basic restoration (use original voice):**
+```
+[Alice] I'm laughing so hard <Laughter:3> <restore>
+```
+- Applies 3 iterations of laughter (may degrade voice)
+- Restores original voice with 1 VC pass
+
+**Stronger restoration:**
+```
+[Alice] I'm laughing so hard <Laughter:3> <restore:2>
+```
+- Applies 3 iterations of laughter (may degrade voice)
+- Restores original voice with 2 VC passes (stronger effect)
+
+**Preserve partial effects (use intermediate iteration):**
+```
+[Bob] Listen carefully <style:whisper:2> <Laughter:3> <restore:1@2>
+```
+- Applies whisper 2 times (iterations 1-2)
+- Applies laughter 3 times (iterations 3-5)
+- Restores using iteration 2 audio (whispered version) with 1 VC pass
+- **Result:** Keeps whisper effect but removes laughter voice degradation
+
+**When to use iteration reference:**
+- `<restore>` or `<restore:N>` - Restore completely to original clean voice
+- `<restore:N@M>` - Keep some effects (like whisper) while removing others (like laughter degradation)
+
+**Processing order:** Restore tags ALWAYS run last, after all other edits complete.
+
+**Position doesn't matter:** Tag can be anywhere in the text.
+
+---
+
+## Syntax Guide
+
+### Basic Syntax
+
+**Default (1 iteration):**
+```
+<Laughter>
+<emotion:happy>
+<style:whisper>
+<speed:faster>
+```
+
+**Custom iterations:**
+```
+<Laughter:3>           # 3 iterations of laughter
+<emotion:happy:2>      # 2 iterations of happy emotion
+<style:whisper:4>      # 4 iterations of whisper style
+```
+
+### Multiple Tags
+
+**Option 1 - Pipe-separated (single tag):**
+```
+[Alice] Hello world! <Laughter:2|style:whisper|emotion:happy>
+```
+
+**Option 2 - Separate tags:**
+```
+[Alice] Hello world! <Laughter:2><style:whisper><emotion:happy>
+```
+
+**Option 3 - Mixed (both work):**
+```
+[Alice] Hello <Laughter:2|emotion:happy> world! <style:whisper>
+```
+
+### Processing Order
+
+When multiple tags are present:
+
+1. **First:** Non-paralinguistic tags in order: emotion → style → speed
+2. **Second:** Paralinguistic tags (preserves position for sound insertion)
+3. **Last:** Voice restoration (if present)
+
+**Example:**
+```
+[Alice] Hello <Laughter:2> world <style:whisper:1> <restore>
+```
+
+Execution:
+1. Generate TTS: "Hello world"
+2. Apply whisper style (1 iteration)
+3. Insert Laughter at position (2 iterations)
+4. Restore voice to original character (1 VC pass)
+
+---
+
+## Usage Examples
+
+### Simple Effects
+
+**Laughter:**
+```
+[Alice] That's the funniest thing I've heard <Laughter>!
+```
+
+**Multiple iterations:**
+```
+[Bob] I can't stop laughing <Laughter:3>, this is amazing!
+```
+
+### Combined Effects
+
+**Emotion + Paralinguistic:**
+```
+[Alice] I'm so excited <Laughter:2|emotion:happy> to see you!
+```
+
+**Style + Speed:**
+```
+[Bob] Listen carefully <style:whisper|speed:slower>, this is important.
+```
+
+### Full Example with Character Switching
+
+```
+[Narrator] The story begins on a dark night.
+
+[Alice|seed:42] Oh no <Surprise-oh>, what was that sound? <emotion:fearful>
+
+[Bob] Don't worry <style:comfort>, I'll check it out. <Breathing>
+
+[Alice] Please be careful! <style:whisper|emotion:worried>
+```
+
+### SRT Mode
+
+```
+1
+00:00:00,000 --> 00:00:03,000
+[Alice] Hello there <Laughter> my friend!
+
+2
+00:00:03,500 --> 00:00:06,000
+[Bob|seed:123] Hi! <style:excited|emotion:happy>
+
+3
+00:00:06,500 --> 00:00:09,000
+[Alice] Let me tell you a secret <style:whisper:2>
+```
+
+---
+
+## ChatterBox 23-Lang v2 Users: Native vs Post-Processing Tags
+
+### Understanding the Difference
+
+**ChatterBox Official 23-Lang v2** has **native special tokens** built into the model (e.g., `<laughter>`, `<giggle>`, `<whisper>`). Our **Step Audio EditX post-processing system** uses similar-looking tags but works differently.
+
+### Syntax Distinction
+
+| Syntax | System | When to Use |
+|--------|--------|-------------|
+| `<laughter>` | **ChatterBox v2 Native** | Model generates laughter during TTS (experimental, may not work well) |
+| `<laughter:1>` | **Step Audio EditX Post-Processing** | Applied AFTER TTS generation (more reliable) |
+
+**The colon (`:`) makes the difference!**
+
+### Examples
+
+```
+# Native ChatterBox v2 tag (model-generated during TTS)
+[Alice] I'm laughing <laughter> so hard!
+
+# Step Audio EditX tag (post-processing after TTS)
+[Alice] I'm laughing <Laughter:2> so hard!
+```
+
+### Which Should You Use?
+
+**For ChatterBox 23-Lang users:**
+
+✅ **Use Step Audio EditX tags** (`<Laughter:2>`, etc.) - More reliable, works consistently
+
+⚠️ **ChatterBox v2 native tags** (`<laughter>`, `<giggle>`) - Experimental, limited functionality:
+- Tokens exist in vocabulary but may produce minimal/no effect
+- No official documentation from ResembleAI
+- Results are inconsistent
+- See [ResembleAI/chatterbox #186](https://github.com/resemble-ai/chatterbox/issues/186)
+
+**Console Warning:**
+
+If you use ChatterBox v2 native tags without the colon, you may see:
+```
+⚠️ Using ChatterBox v2 native tag '<laughter>' - experimental feature with limited effect.
+   For more reliable results, use Step Audio EditX: '<Laughter:2>'
+```
+
+### Compatibility Note
+
+Step Audio EditX tags work with **ALL engines** (F5-TTS, ChatterBox, Higgs Audio, VibeVoice), not just ChatterBox. Native ChatterBox tags only work with ChatterBox 23-Lang v2.
+
+---
+
+## Important Limitations
+
+### ⚠️ Language Support Warning
+
+**Step Audio EditX supports ONLY:**
+- **Mandarin Chinese** (primary)
+- **English**
+- **Sichuanese** (Chinese dialect)
+- **Cantonese** (Chinese dialect)
+- **Japanese**
+- **Korean**
+
+**If using other languages** (French, German, Spanish, etc.):
+
+❌ **PROBLEM:** Step Audio EditX will process non-supported languages as English or Chinese, which will:
+- **Lose the original accent** - French audio may lose French pronunciation
+- **Distort the audio** - The model tries to "fix" unknown languages
+- **Generate poor quality** - Editing may sound unnatural or broken
+
+✅ **SOLUTION:**
+1. **Don't use edit tags** with unsupported languages
+2. **Use only supported languages** when edit tags are needed
+3. **Test carefully** if mixing languages - some distortion may be acceptable
+
+**Example - BAD:**
+```
+[Alice:fr] Bonjour mon ami <Laughter>     # French + edit tag = accent loss/distortion
+```
+
+**Example - GOOD:**
+```
+[Alice:fr] Bonjour mon ami                 # French without edit tag = preserves accent
+[Alice:en] Hello my friend <Laughter>      # English + edit tag = works perfectly
+```
+
+### Duration Limits
+
+**Minimum:** 0.5 seconds - Segments shorter than this will skip editing
+**Maximum:** 30 seconds - Segments longer than this will skip editing
+
+**Warning messages:**
+```
+⚠️ Segment too short (0.3s < 0.5s). Edit tags will be skipped.
+⚠️ Segment too long (35.2s > 30s). Edit tags will be skipped. Split your text into shorter segments.
+```
+
+**Solution for long segments:**
+```
+# BAD - Too long
+[Alice] [Very long paragraph that exceeds 30 seconds...] <Laughter>
+
+# GOOD - Split into multiple segments
+[Alice] First part of the paragraph. <Laughter>
+[Alice] Second part continues here.
+[Alice] Final part with the conclusion. <emotion:happy>
+```
+
+### Batch Processing (Automatic)
+
+Edit tags are **automatically batched** for efficiency:
+
+✅ **What happens:**
+1. ALL TTS audio is generated first (for all segments/subtitles)
+2. Step Audio EditX loads ONCE
+3. ALL segments with edit tags are processed together
+4. Engine unloads
+
+✅ **Benefits:**
+- Faster generation (engine loads once, not per segment)
+- Less VRAM thrashing
+- Better performance
+
+**You don't need to do anything** - batching happens automatically in both regular and SRT modes.
+
+---
+
+## Performance Tips
+
+### 1. Use Iterations Wisely
+
+More iterations = stronger effect BUT **can alter voice quality and resemblance**:
+
+```
+<Laughter:1>     # Subtle effect, minimal voice change (recommended)
+<Laughter:2>     # Moderate effect, slight voice alteration (acceptable)
+<Laughter:3>     # Strong effect, noticeable voice change (risky)
+<Laughter:4>     # Very strong effect, significant voice distortion (not recommended)
+```
+
+⚠️ **Voice Resemblance Warning:**
+- **1-2 iterations:** Generally safe, minimal voice alteration
+- **3+ iterations:** High risk of voice distortion and quality degradation
+- Each iteration processes the audio again, which can degrade the original voice characteristics
+
+**Recommendation:** Start with 1 iteration, only increase to 2 if needed. Avoid 3+ unless you're willing to accept voice changes.
+
+**✅ Voice Restoration Available:** Use the `<restore>` tag to restore original voice resemblance after editing. See the Voice Restoration Tags section above for details.
+
+### 2. Combine Tags Efficiently
+
+**Less efficient (multiple passes):**
+```
+[Alice] Text here <emotion:happy>
+[Alice] More text <emotion:happy>
+[Alice] Even more <emotion:happy>
+```
+
+**More efficient (apply once):**
+```
+[Alice|emotion:happy] Text here. More text. Even more.
+```
+
+Though with inline tags, segment-level parameters are cleaner for consistent effects across multiple sentences.
+
+### 3. Position Paralinguistic Tags Carefully
+
+**Position matters for sound insertion:**
+```
+Hello <Laughter> world    # Laughter after "Hello"
+Hello world <Laughter>    # Laughter at end
+<Laughter> Hello world    # Laughter at start
+```
+
+Choose the position that sounds most natural for your use case.
+
+---
+
+## Troubleshooting
+
+### Problem: Edit tags not applied
+
+**Possible causes:**
+1. **Segment too short/long** - Check duration (0.5s - 30s range)
+2. **Syntax error** - Verify tag format: `<type:value:iterations>`
+3. **Model not downloaded** - First use triggers download (be patient)
+
+**Check console output:**
+```
+🎨 Applying Step Audio EditX processing to 3 segments...
+📝 Segment 1 - Applying edit tags:
+  🎨 Applying paralinguistic: Laughter:2
+✅ Edit post-processing complete for 3 segments
+```
+
+### Problem: Audio quality degraded
+
+**Possible causes:**
+1. **Too many iterations** - Reduce iterations (try 1-2 instead of 3-4)
+2. **Incompatible tags** - Some emotion+style combinations may clash
+3. **Unsupported language** - Check language support (see warning above)
+
+**Solution:**
+- Use fewer iterations
+- Test individual tags separately
+- Use only supported languages with edit tags
+
+### Problem: "Segment too long" warning
+
+**Solution:** Split text into shorter segments:
+
+```
+# Instead of:
+[Alice] [300-word monologue that lasts 45 seconds] <Laughter>
+
+# Do this:
+[Alice] First paragraph of about 2-3 sentences.
+[Alice] Second paragraph continues. <Laughter>
+[Alice] Third paragraph concludes the thought. <emotion:happy>
+```
+
+---
+
+## Advanced Usage
+
+### Combining with Segment Parameters
+
+Edit tags work with ALL segment parameters:
+
+```
+[Alice|seed:42|temperature:0.8|speed:1.2] Hello <Laughter:2> friend! <emotion:happy>
+```
+
+Processing:
+1. TTS uses seed:42, temp:0.8, speed:1.2
+2. Edit tags applied after generation
+
+### Using with Language Switching
+
+**Supported languages ONLY:**
+```
+[Alice:en] Hello my friend <Laughter>              # English - works ✓
+[Bob:zh] 你好朋友 <Laughter>                         # Chinese - works ✓
+[Alice:ja] こんにちは友達 <Laughter>                  # Japanese - works ✓
+[Bob:ko] 안녕 친구 <Laughter>                        # Korean - works ✓
+```
+
+**Unsupported languages - AVOID:**
+```
+[Alice:fr] Bonjour mon ami <Laughter>              # French - will distort! ✗
+[Bob:de] Hallo mein Freund <Laughter>              # German - will distort! ✗
+[Alice:es] Hola mi amigo <Laughter>                # Spanish - will distort! ✗
+```
+
+### Creating Natural Conversations
+
+```
+[Narrator] The interview begins.
+
+[Host] Welcome to the show! <emotion:excited>
+
+[Guest] Thank you for having me. <style:warm>
+
+[Host] So tell us <style:chat>, what's your secret? <Laughter>
+
+[Guest] Well <Uhm> it's quite simple actually. <style:story>
+
+[Host] That's amazing <Surprise-oh>! <emotion:surprised>
+```
+
+---
+
+## Best Practices
+
+### 1. Start Simple
+
+Begin with single tags before combining:
+
+```
+# Step 1: Test basic tag
+[Alice] Hello <Laughter>
+
+# Step 2: Add iterations
+[Alice] Hello <Laughter:2>
+
+# Step 3: Combine effects
+[Alice] Hello <Laughter:2|emotion:happy>
+```
+
+### 2. Match Tags to Context
+
+Choose appropriate effects for the situation:
+
+```
+# Scary scene
+[Alice] What was that? <Surprise-oh|emotion:fearful>
+
+# Happy scene
+[Bob] We did it! <Laughter:2|emotion:excited>
+
+# Sad scene
+[Alice] I can't believe it's over. <Sigh|emotion:sad>
+
+# Secret conversation
+[Bob] Come closer. <style:whisper|emotion:calm>
+```
+
+### 3. Test and Iterate
+
+1. Generate with basic tag
+2. Listen to result
+3. Adjust iterations if needed
+4. Try different tag combinations
+5. Find what works best for your use case
+
+### 4. Use Language Hints Correctly
+
+**Always match edit language to TTS language:**
+
+```
+# GOOD - Language consistency
+[Alice:en] English text here <Laughter>           # Both English
+[Bob:zh] 中文文本 <Laughter>                        # Both Chinese
+
+# BAD - Language mismatch
+[Alice:fr] French text <Laughter>                 # French TTS + English edit model = distortion
+```
+
+---
+
+## Reference
+
+### All Paralinguistic Tags
+`<Breathing>`, `<Laughter>`, `<Sigh>`, `<Uhm>`, `<Surprise-oh>`, `<Surprise-ah>`, `<Surprise-wa>`, `<Confirmation-en>`, `<Question-ei>`, `<Dissatisfaction-hnn>`
+
+### All Emotions
+`<emotion:happy>`, `<emotion:sad>`, `<emotion:angry>`, `<emotion:excited>`, `<emotion:calm>`, `<emotion:fearful>`, `<emotion:surprised>`, `<emotion:disgusted>`, `<emotion:confusion>`, `<emotion:empathy>`, `<emotion:embarrass>`, `<emotion:depressed>`, `<emotion:coldness>`, `<emotion:admiration>`
+
+### All Styles
+`<style:whisper>`, `<style:serious>`, `<style:child>`, `<style:older>`, `<style:girl>`, `<style:pure>`, `<style:sister>`, `<style:sweet>`, `<style:exaggerated>`, `<style:ethereal>`, `<style:generous>`, `<style:recite>`, `<style:act_coy>`, `<style:warm>`, `<style:shy>`, `<style:comfort>`, `<style:authority>`, `<style:chat>`, `<style:radio>`, `<style:soulful>`, `<style:gentle>`, `<style:story>`, `<style:vivid>`, `<style:program>`, `<style:news>`, `<style:advertising>`, `<style:roar>`, `<style:murmur>`, `<style:shout>`, `<style:deeply>`, `<style:loudly>`, `<style:arrogant>`, `<style:friendly>`
+
+### All Speeds
+`<speed:faster>`, `<speed:slower>`, `<speed:more_faster>`, `<speed:more_slower>`
+
+### Voice Restoration
+`<restore>`, `<restore:N>`, `<restore:N@M>` (where N = VC passes 1-5, M = reference iteration number)
+
+---
+
+## FAQ
+
+**Q: Can I use edit tags with any TTS engine?**
+A: Yes! Edit tags work with F5-TTS, ChatterBox, Higgs Audio, VibeVoice, and any future engines.
+
+**Q: Do edit tags slow down generation?**
+A: Slightly. The TTS generates normally, then Step Audio EditX loads once and processes all tagged segments in batch. Total time depends on number of segments and iterations.
+
+**Q: Can I use edit tags in SRT mode?**
+A: Yes! Batch processing is automatically applied - all subtitles are generated first, then all edits are applied at once.
+
+**Q: What if I make a typo in a tag?**
+A: Unknown tags are ignored and stripped from the text. Check console for warnings.
+
+**Q: Can I nest tags?**
+A: No. Use pipe syntax or separate tags: `<tag1|tag2>` or `<tag1><tag2>`
+
+**Q: Do edit tags work with ChatterBox special tokens?**
+A: Yes, but ChatterBox v2 uses different syntax (`<giggle>`, `<whisper>` built into model). Our edit tags are post-processing, not model-native.
+
+**Q: Will edit tags affect my voice cloning?**
+A: Yes, they can. TTS generates with your voice reference normally, but each edit iteration can alter voice quality and resemblance:
+- **1 iteration:** Minimal impact, usually safe
+- **2 iterations:** Slight alteration, generally acceptable
+- **3+ iterations:** Noticeable voice change, may lose original voice characteristics
+
+Use the minimum iterations needed for your desired effect. To restore voice resemblance after editing, use the `<restore>` tag which applies ChatterBox Voice Changer to recover the original voice character.
+
+**Q: Can I disable edit tags temporarily?**
+A: Just remove them from your text. No special setting needed.
+
+---
+
+## Support
+
+For issues, feature requests, or questions:
+- GitHub Issues: https://github.com/diodiogod/TTS-Audio-Suite/issues
+- Check console output for detailed processing information
+- Test with simple examples first before complex combinations
+
+---
+
+**Last Updated:** 2025-12-08
+**Compatible with:** TTS Audio Suite v4.4.0+

docs/IndexTTS2_Emotion_Control_Guide.md

@@ -0,0 +1,214 @@
+# IndexTTS-2 Emotion Control Guide
+
+IndexTTS-2 features advanced emotion control capabilities that allow you to precisely control the emotional expression of generated speech. This guide covers all available emotion control methods and their applications.
+
+## Overview
+
+IndexTTS-2 supports multiple emotion control methods that can be combined for sophisticated emotional expression:
+
+- **Direct Audio Reference**: Use any audio file as an emotion reference
+- **Character Voices**: Use character audio references from the Character Voices node
+- **Emotion Vectors**: Manual 8-emotion slider control with precise values
+- **Text Emotion**: AI-powered QwenEmotion analysis from text descriptions with dynamic templates
+- **Character Tag Emotions**: Per-character emotion control using `[Character:emotion_ref]` syntax
+
+## Emotion Control Priority
+
+You can only connect to the Engine node one source of control emotion: Either audio, text, or vectors.
+
+When using tags on the text iself, **Character tag emotions** (highest priority) - `[Alice:angry_bob]` overrides all other emotion control settings for that character segment
+
+## Method 1: Direct Audio Reference
+
+Connect any audio file directly to the IndexTTS-2 Engine's `emotion_control` input.
+
+**How it works:**
+
+- IndexTTS-2 analyzes the emotional characteristics of your reference audio
+- The emotional style is applied to all generated speech
+- Works with any audio format (WAV, MP3, etc.)
+
+**Best practices:**
+
+- Use audio clips with clear emotional expression
+- Choose audio with consistent voice characteristics for best results
+- Avoid background music or noise
+
+**Example:**
+
+```
+AUDIO node → IndexTTS-2 Engine (emotion_control)
+```
+
+## Method 2: Character Voices Audio Reference
+
+Use the `opt_narrator` output from the 🎭 Character Voices node as an emotion reference.
+
+**Setup:**
+
+1. Add a 🎭 Character Voices node
+2. Select a voice with the desired emotional expression
+3. Connect `opt_narrator` output to IndexTTS-2 Engine `emotion_control` input
+
+**Advantages:**
+
+- Leverages your existing voice library
+- Consistent character-based emotions
+- Easy to manage and organize
+
+**Example workflow:**
+
+```
+🎭 Character Voices (David_Attenborough) → opt_narrator → IndexTTS-2 Engine (emotion_control)
+```
+
+## Method 3: Emotion Vectors
+
+Use the 🌈 IndexTTS-2 Emotion Vectors node for precise manual control over 8 different emotions.
+
+**Available emotions:**
+
+- **Happy**: Joy, excitement, positivity (0.0-1.2)
+- **Angry**: Aggression, frustration, intensity (0.0-1.2)
+- **Sad**: Melancholy, sorrow, downcast tone (0.0-1.2)
+- **Surprised**: Amazement, shock, wonder (0.0-1.2)
+- **Afraid**: Fear, anxiety, nervousness (0.0-1.2)
+- **Disgusted**: Revulsion, displeasure, rejection (0.0-1.2)
+- **Calm**: Peaceful, relaxed, steady (0.0-1.2)
+- **Melancholic**: Thoughtful sadness, wistfulness (0.0-1.2)
+
+**Usage tips:**
+
+- Values above 1.0 create more intense emotional expression BUT MAY interfear with the cloned voice resemblance
+- Combine multiple emotions for complex feelings (e.g., 0.8 Happy + 0.3 Surprised = excited joy)
+- Start with single emotions, then experiment with combinations
+- Use the `random` buttom to get a completely random emotion pattern. Might be too strong.
+
+## Method 4: Text Emotion (Dynamic Analysis)
+
+Use the 🌈 IndexTTS-2 Text Emotion node for AI-powered emotion analysis with dynamic templates.
+
+### Static Text Emotion
+
+Provide a simple emotion description that applies to all text segments:
+
+```
+Input: "angry and frustrated"
+Result: All speech generated with angry, frustrated emotion
+```
+
+### Dynamic Templates with {seg}
+
+Use the `{seg}` placeholder for contextual, per-segment emotion analysis:
+
+**Template examples:**
+
+- `"Happy character speaking: {seg}"` - Cheerful narrator
+- `"Angry boss yelling: {seg}"` - Aggressive authority figure
+- `"Calm meditation guide: {seg}"` - Peaceful instructor
+- `"Excited game show host: {seg}"` - Energetic presenter
+
+**How dynamic templates work:**
+
+1. IndexTTS-2 processes each text segment separately
+2. `{seg}` gets replaced with the actual segment text
+3. QwenEmotion analyzes the combined context + content
+4. Unique emotion vector generated for each segment
+
+**Example:**
+
+```
+Template: "Worried parent speaking: {seg}"
+Segment: "Where have you been?"
+Analysis: "Worried parent speaking: Where have you been?"
+Result: Anxious, concerned vocal expression
+```
+
+---
+
+## Character Tag Emotion Control
+
+Control emotions per character using inline tags in your text: `[Character:emotion_ref]`
+
+**Syntax:**
+
+```
+[CharacterName:emotion_reference]
+```
+
+**emotion_reference options:**
+
+- Any character name from your voices library (uses that character's voice as emotion)
+- Custom emotion references
+
+**Examples:**
+
+```
+Hello everyone! [Alice:happy_sarah] I'm so excited to be here today!
+[Bob:angry_tom] That's completely unacceptable behavior.
+[Narrator:David] Meanwhile, in a distant galaxy...
+
+*assuming happy_sarah, angry_tom and David are alias or character voices in yout folder with that name
+```
+
+\*assuming happy_sarah, angry_tom and David are alias or character voices in yout folder with that name
+
+
+
+**Character tag priority:**
+
+- Character tags override ALL other emotion settings for that specific character
+- Other characters use global emotion settings
+- Allows mixing different emotions in the same audio
+
+## Emotion Alpha Control
+
+The `emotion_alpha` parameter on the IndexTTS-2 Engine controls the intensity of emotion application:
+
+**Values:**
+
+- **0.0**: No emotion applied (neutral voice)
+- **0.5**: 50% emotion blend (subtle emotional influence)
+- **1.0**: Full emotion intensity (standard recommended setting)
+- **1.5**: 150% enhanced emotion (more dramatic)
+- **2.0**: Maximum emotion intensity (very dramatic)
+
+## Practical Workflow Examples
+
+### Example 1: Multi-Character Drama with Individual Emotions
+
+```text
+[Alice:happy_sarah] Welcome to our cooking show!
+[Bob:serious_narrator] Today we'll be making pasta.
+[Alice:excited_sarah] I can't wait to get started!
+```
+
+**Setup:**
+
+- No global emotion control needed
+- Each character gets individual emotion via tags
+- `emotion_alpha=1.0` for more expressiveness
+
+### Example 2: Mixed Emotion Control
+
+**Global setup:**
+
+- 🌈 IndexTTS-2 Text Emotion: `"Cheerful host presenting: {seg}"`
+- `emotion_alpha=0.8`
+
+**Text with overrides:**
+
+```text
+Welcome to our show! [Bob:serious_narrator] But first, a serious announcement.
+[Alice:excited_sarah] Now back to our regular programming!
+```
+
+**Result:**
+
+- Default segments use cheerful host emotion
+- Bob's line uses serious narrator emotion (overrides global)
+- Alice's line uses excited emotion (overrides global)
+
+---
+
+This comprehensive emotion control system gives you unprecedented flexibility in creating expressive, emotionally rich TTS audio for any application.

docs/NEW_ENGINE_IMPLEMENTATION_GUIDE.md

@@ -0,0 +1,619 @@
+# New TTS Engine Implementation Guide
+
+*Comprehensive guide for LLMs to implement new TTS engines in TTS Audio Suite*
+
+---
+
+Don't ever make PLACEHOLDERS during implementation.
+
+## Pre-Implementation Analysis
+
+### 1. Research the Original Implementation
+
+**📁 Reference Storage:**
+
+- Clone the original implementation to: `ComfyUI_TTS_Audio_Suite/IgnoredForGitHubDocs/For_reference/[ENGINE_NAME]/`
+- Study the original codebase thoroughly
+- Document all features, parameters, and capabilities
+
+**🔍 Key Areas to Analyze:**
+
+- **Audio Format**: Sample rate, bit depth, channels, tensor format
+- **Model Architecture**: Input/output requirements, tokenization, generation process
+- **Parameters**: All generation parameters, their ranges, default values
+- **Dependencies**: Required packages, versions, potential conflicts
+- **Unique Features**: Special capabilities not found in other engines
+- **Language Support**: Monolingual vs multilingual, language codes/tags
+- **Voice Control**: How voices are defined, selected, and applied
+
+### 2. Dependency Analysis
+
+*⚠️ Problematic Dependencies Handling:**
+
+- If dependencies conflict with existing packages → Add to `scripts/install.py`
+- If dependencies require specific versions → Document in engine requirements
+
+**Example problematic patterns:**
+
+- Downgrades numpy/librosa/transformers
+- Conflicts with other TTS engines
+
+---
+
+## Project Architecture Understanding
+
+### Core Architecture Pattern
+
+```
+⚙️ Engine Node (UI Layer real node on ComfyUI)
+    ↓
+🔌 Engine Adapter (Optional? - Parameter Translation)
+    ↓
+🏭 Engine Processor (Engine-Specific Logic)
+```
+
+### File Structure Template
+
+```
+engines/[ENGINE_NAME]/
+├── __init__.py                    # Engine initialization
+├── [engine_name].py              # Core engine implementation
+├── [engine_name]_downloader.py   # Model auto-download (optional, should use universal downloader code not it's own)
+├── stateless_wrapper.py          # Thread-safe wrapper (if needed)
+└── models/                       # Model-specific code (if needed)
+
+engines/adapters/
+└── [engine_name]_adapter.py      # Unified interface adapter
+
+nodes/engines/
+└── [engine_name]_engine_node.py  # UI configuration node
+
+nodes/[engine_name]/               # Engine-specific processors
+├── [engine_name]_processor.py    # Main TTS processor
+├── [engine_name]_srt_processor.py # SRT processor
+└── [engine_name]_vc_processor.py  # Voice conversion (if applicable)
+
+nodes/[engine_name]_special/       # Special features (if any)
+└── [engine_name]_special_node.py  # Special functionality nodes
+```
+
+---
+
+## Implementation Steps
+
+### Phase 1: Basic UI Engine Implementation
+
+#### Step 1: Create Core Engine Implementation
+
+**File:** `engines/[ENGINE_NAME]/[engine_name].py`
+
+**Key Implementation Notes:**
+
+- **Audio Format**: Always return `torch.Tensor` in shape `[1, samples]` or `[batch, samples]`
+- **Sample Rate**: Must match engine's native sample rate, handle conversion in adapter if needed
+- **Device Management**: Support "auto", "cuda", "cpu" device selection
+- **Clear VRAM Integration**: CRITICAL - Implement `.to()` method and device checking for ComfyUI model management (see Step 1b below)
+- **Error Handling**: Graceful fallbacks, informative error messages
+
+#### Step 1b: Implement Clear VRAM Integration (CRITICAL)
+
+**⚠️ REQUIRED FOR ALL NEW ENGINES**
+
+ComfyUI's "Clear VRAM" button offloads models to CPU. Your engine MUST support this or it will cause device mismatch errors.
+
+**Implementation Requirements:**
+
+**1. Add `.to()` Method to Engine Class**
+
+This method moves ALL model components between devices (CPU ↔ CUDA).
+
+```python
+def to(self, device):
+    """
+    Move all model components to the specified device.
+
+    Critical for ComfyUI model management - ensures all components move together
+    when models are detached to CPU and later reloaded to CUDA.
+    """
+    self.device = device
+
+    # Move the underlying model/engine if loaded
+    if self._model is not None:
+        # OPTION 1: Simple engines with single model
+        if hasattr(self._model, 'to'):
+            self._model = self._model.to(device)
+
+        # OPTION 2: Complex engines with multiple components
+        # Example: ChatterBox has model.s3gen, model.t3, model.voice_encoder
+        for component_name in ['s3gen', 't3', 'voice_encoder', 'tokenizer']:
+            if hasattr(self._model, component_name):
+                component = getattr(self._model, component_name)
+                if hasattr(component, 'to'):
+                    setattr(self._model, component_name, component.to(device))
+
+        # OPTION 3: Deeply nested engines (like Index-TTS)
+        # Call .to() on engine itself for recursive moving, with manual fallback
+        if hasattr(self._model, 'to'):
+            self._model = self._model.to(device)
+        else:
+            # Manually move known nested components
+            for attr_name in ['semantic_model', 'gpt_model', 'vocoder']:
+                if hasattr(self._model, attr_name):
+                    component = getattr(self._model, attr_name)
+                    if hasattr(component, 'to'):
+                        setattr(self._model, attr_name, component.to(device))
+
+        # Update device attribute on model if it exists
+        if hasattr(self._model, 'device'):
+            self._model.device = torch.device(device) if isinstance(device, str) else device
+
+    return self
+```
+
+**2. Add Device Checking Before Generation**
+
+Check if model was offloaded to CPU and reload to CUDA before generation.
+
+```python
+def generate(self, text, reference_audio, **kwargs):
+    """Main generation method"""
+
+    # CRITICAL: Check and reload model if offloaded to CPU
+    target_device = "cuda" if torch.cuda.is_available() else "cpu"
+
+    if self._model is not None:
+        # Check current device of model
+        # Strategy 1: Check first parameter device (most reliable)
+        if hasattr(self._model, 'parameters'):
+            try:
+                first_param = next(self._model.parameters())
+                current_device = str(first_param.device)
+
+                if current_device != target_device:
+                    print(f"🔄 Reloading {self.__class__.__name__} from {current_device} to {target_device}")
+
+                    # Try to find wrapper and call wrapper.model_load()
+                    wrapper_found = False
+                    try:
+                        from utils.models.unified_model_interface import unified_model_interface
+
+                        if hasattr(unified_model_interface, 'model_manager'):
+                            for cache_key, wrapper in unified_model_interface.model_manager._model_cache.items():
+                                model = wrapper.model if hasattr(wrapper, 'model') else None
+                                if model is self:
+                                    wrapper.model_load(target_device)
+                                    print(f"✅ Reloaded via wrapper - ComfyUI tracking in sync")
+                                    wrapper_found = True
+                                    break
+                                # Check through SimpleModelWrapper if present
+                                elif hasattr(model, 'model') and model.model is self:
+                                    wrapper.model_load(target_device)
+                                    print(f"✅ Reloaded via wrapper (unwrapped)")
+                                    wrapper_found = True
+                                    break
+
+                        if not wrapper_found:
+                            # Fallback: direct .to() and check if already registered
+                            self.to(target_device)
+
+                            # Check ComfyUI's current_loaded_models for existing wrapper
+                            try:
+                                import comfy.model_management as model_management
+
+                                already_registered = False
+                                if hasattr(model_management, 'current_loaded_models'):
+                                    for wrapper in model_management.current_loaded_models:
+                                        if hasattr(wrapper, 'model_info') and wrapper.model_info.engine == "[engine_name]":
+                                            wrapper.model_load(target_device)
+                                            print(f"✅ Reloaded via existing ComfyUI wrapper")
+                                            already_registered = True
+                                            break
+
+                                if not already_registered:
+                                    # Re-register with ComfyUI
+                                    from utils.models.comfyui_model_wrapper.base_wrapper import ComfyUIModelWrapper, ModelInfo, SimpleModelWrapper
+
+                                    model_size = ComfyUIModelWrapper.calculate_model_memory(self)
+                                    wrapped_model = SimpleModelWrapper(self)
+
+                                    model_info = ModelInfo(
+                                        model=wrapped_model,
+                                        model_type="tts",
+                                        engine="[engine_name]",
+                                        device=target_device,
+                                        memory_size=model_size,
+                                        load_device=target_device
+                                    )
+                                    new_wrapper = ComfyUIModelWrapper(wrapped_model, model_info)
+                                    model_management.current_loaded_models.append(new_wrapper)
+                                    print(f"✅ Re-registered with ComfyUI model management")
+                            except Exception as e:
+                                print(f"⚠️ Re-registration failed: {e}")
+
+                    except Exception as e:
+                        print(f"⚠️ Wrapper reload failed ({e}), using direct .to()")
+                        self.to(target_device)
+
+            except StopIteration:
+                pass
+
+        # Strategy 2: For models with nested components
+        elif hasattr(self._model, 'semantic_model') and hasattr(self._model.semantic_model, 'parameters'):
+            try:
+                first_param = next(self._model.semantic_model.parameters())
+                current_device = str(first_param.device)
+                # ... same reload logic as above
+            except StopIteration:
+                pass
+
+    # Continue with normal generation
+    # ... your generation code here
+```
+
+**3. Important Device Checking Notes:**
+
+- **Always check against INTENDED device** (`"cuda"` if available), NOT `self.device` which gets updated to "cpu" after offload
+- **Search for existing wrapper first** before creating new ones to avoid duplicates
+- **Use wrapper.model_load()** when possible to keep ComfyUI tracking in sync
+- **Fallback to direct .to()** if wrapper not found, then re-register
+- **Check both unified interface cache AND ComfyUI's current_loaded_models** for wrappers
+
+**4. Key Implementation Notes:**
+
+- **Always check against INTENDED device** (`"cuda"` if available), NOT `self.device`
+- **Search for existing wrapper first** to avoid duplicates
+- **Use wrapper.model_load()** when possible to keep ComfyUI tracking in sync
+- **Check both unified interface cache AND current_loaded_models**
+
+#### Step 2: Create Model Downloader
+
+**File:** `engines/[ENGINE_NAME]/[engine_name]_downloader.py`
+
+**Follow Unified Download Pattern:**
+
+```python
+from utils.downloads.unified_downloader import UnifiedDownloader
+```
+
+#### Step 3: Create Engine Adapter
+
+**File:** `engines/adapters/[engine_name]_adapter.py`
+
+#### Step 4: Create Engine Configuration Node
+
+**File:** `nodes/engines/[engine_name]_engine_node.py`
+
+### Phase 2: Unified Systems Integration
+
+#### Step 5: Integrate with Unified Model Loading
+
+**Use ComfyUI Model Management:**
+
+```python
+from utils.models.unified_model_interface import UnifiedModelInterface
+from utils.models.comfyui_model_wrapper import ComfyUIModelWrapper
+```
+
+#### Step 6: Implement Caching System
+
+**Cache Integration:**
+
+```python
+from utils.audio.cache import UnifiedCacheManager
+from utils.audio.audio_hash import create_content_hash
+```
+
+#### Step 7: Character Switching Integration
+
+**Use Unified Character System:**
+
+```python
+from utils.text.character_parser import CharacterParser
+from utils.voice.discovery import get_character_mapping
+```
+
+#### Step 8: Language Switching Integration
+
+**Use Unified Language System:**
+
+```python
+from utils.models.language_mapper import LanguageMapper
+
+class [EngineClass]Processor:
+    def __init__(self):
+        self.language_mapper = LanguageMapper()
+
+    def generate_with_language(self, text, language, **params):
+        # Map language code
+        engine_language = self.language_mapper.map_language(
+            language_code=language,
+            engine_type="[engine_name]"
+        )
+
+        # Apply language-specific model loading if needed
+        if self.supports_multiple_languages:
+            self.load_language_model(engine_language)
+
+        # Generate with language parameter
+        return self.engine.generate(text, language=engine_language, **params)
+```
+
+#### Step 9: Pause Tag Integration
+
+**Use Unified Pause System, search for the unified code**
+
+#### Step 9.5: Implement Interrupt Handling (CRITICAL)
+
+**⚠️ REQUIRED FOR ALL NEW ENGINES - Users Must Be Able to Cancel Operations**
+
+ComfyUI provides an interrupt mechanism that allows users to stop long-running operations (like SRT generation with many segments). Your processors MUST check for interruption signals.
+
+**Import and Check Pattern:**
+
+```python
+import comfy.model_management as model_management
+
+# In your main processing loops:
+def generate_srt_speech(self, srt_content: str, ...):
+    # ... setup code ...
+
+    for i, segment in enumerate(srt_segments):
+        # Check for interruption before processing each segment
+        if model_management.interrupt_processing:
+            raise InterruptedError(f"Engine SRT segment {i+1}/{len(srt_segments)} interrupted by user")
+
+        # Your generation code...
+```
+
+**Where to Add Interrupt Checks:**
+
+1. **SRT Processors** - Add checks in main subtitle/segment loops:
+   
+   - Before processing each subtitle/segment
+   - Before processing character segments within a subtitle
+   - Before timing assembly (after all audio generation)
+
+2. **Text Processors** - Add checks in main generation loops:
+   
+   - Before character setup/loading
+   - Before processing each character segment
+   - Before processing chunks (if chunking is enabled)
+
+3. **Special Feature Processors** - Add checks in long-running operations:
+   
+   - Before each major processing step
+   - At natural breakpoints (before/after audio generation)
+
+**Example SRT Processor Structure:**
+
+```python
+class EngineSRTProcessor:
+    def generate_srt_speech(self, srt_content, ...):
+        # Pre-generation check
+        if model_management.interrupt_processing:
+            raise InterruptedError("Setup interrupted")
+
+        # Main segment loop
+        for i, segment in enumerate(srt_segments):
+            if model_management.interrupt_processing:
+                raise InterruptedError(f"Segment {i+1}/{len(srt_segments)} interrupted")
+
+            # Character switching support
+            for char_segment in char_segments:
+                if model_management.interrupt_processing:
+                    raise InterruptedError(f"Character {char} interrupted")
+                # ... generate audio ...
+
+        # Pre-assembly check
+        if model_management.interrupt_processing:
+            raise InterruptedError("Assembly interrupted")
+
+        # Timing assembly...
+```
+
+**Real Examples:**
+
+- See `nodes/higgs_audio/higgs_audio_srt_processor.py` - SRT interrupts
+- See `nodes/vibevoice/vibevoice_processor.py` - TTS processor interrupts
+- See `nodes/chatterbox_official_23lang/chatterbox_official_23lang_srt_processor.py` - Complex interrupts
+
+#### Step 10: Create Main TTS Processor
+
+**File:** `nodes/[engine_name]/[engine_name]_processor.py`
+
+#### Step 11: Test TTS Text Implementation
+
+BUT rememeber to register all nodes so they load and work on comfyui.
+
+Also to test, requirements and dependencies need to be added.
+
+**Ask suer to Test Checklist:**
+
+- [ ] Basic text generation works
+- [ ] Character switching works with `[CharacterName] text`
+- [ ] Language switching works (if applicable)
+- [ ] Pause tags work with `[pause:1.5s]`
+- [ ] Caching works (same input = cached output)
+- [ ] Model auto-download works
+- [ ] VRAM management works (model unloads)
+- [ ] Different parameter combinations work
+- [ ] **Interrupt handling works** - User can stop SRT generation and it stops within ~1 segment
+
+### Phase 4: SRT Implementation
+
+#### Step 12: Analyze SRT Strategies
+
+**Study Existing SRT Implementations:**
+
+- **ChatterBox**: Sequential processing with language grouping
+- **F5-TTS**: Language grouping with chunking
+- **Higgs Audio**: Character-based processing
+- VibeVoice?
+
+**Choose Strategy:**
+
+1. **Sequential**: Process each subtitle line individually
+2. **Language Grouping**: Group by language, then process
+3. **Character Grouping**: Group by character, then batch process
+4. **Hybrid**: Combine multiple strategies
+
+#### Step 13: Create SRT Processor
+
+**File:** `nodes/[engine_name]/[engine_name]_srt_processor.py`
+
+### Phase 5: Special Features Implementation
+
+#### Step 14: Identify Special Features
+
+**Common Special Features:**
+
+- **Speech Editing** (F5-TTS): Edit specific words in audio
+- **Voice Conversion** - speech2speech (ChatterBox): Convert voice characteristics
+- **Multi-Speaker** (Some engines): Multiple speakers in one generation
+- **Style Control**: Emotion, speaking rate, emphasis
+
+#### Step 15: Implement Special Features
+
+Create Dedicated Nodes
+
+---
+
+## Unified Systems Integration
+
+### Character Voice System
+
+**Files to Study:**
+
+- `utils/voice/discovery.py` - Voice file discovery
+- `utils/text/character_parser.py` - Character tag parsing
+- `nodes/shared/character_voices_node.py` - Character voice management
+
+### Language System
+
+**Files to Study:**
+
+- `utils/models/language_mapper.py` - Language code mapping
+- Engine-specific language model files
+
+### Pause Tag System
+
+**Files to Study:**
+
+- `utils/text/pause_processor.py` - Pause tag parsing and generation
+
+**Integration Pattern:**
+
+### Model Management
+
+**Files to Study:**
+
+- `utils.models/unified_model_interface.py` - Unified model loading
+- `utils/models/comfyui_model_wrapper.py` - ComfyUI integration
+
+---
+
+## Documentation Updates
+
+### README.md Updates
+
+#### 1. Features Section
+
+Add engine to the features list with its unique capabilities.
+
+#### 2. What's New Section
+
+Add changelog entry for the new engine.
+
+#### 3. Model Download Section
+
+Add download instructions and model requirements.
+
+#### 4. Supported Engines Table
+
+Update the engines comparison table.
+
+### Example README Addition:
+
+```markdown
+## What's New in v4.X.X
+
+### 🚀 New [Engine Name] TTS Engine
+- High-quality text-to-speech with [unique feature]
+- Support for [languages/voices/special capabilities]
+- Integrated with unified interface (TTS Text, SRT, Voice Changer)
+- Auto-download models with one click
+
+## Features
+
+### 🎤 [Engine Name] TTS Engine
+- **[Unique Feature 1]**: Description of what makes this engine special
+- **[Unique Feature 2]**: Another special capability
+- **Multi-language support**: List of supported languages (if applicable)
+- **Voice cloning**: Description of voice capabilities (if applicable)
+```
+
+---
+
+## Implementation Phase Strategy
+
+### Phase 1: Foundation (Implement First)
+
+1. Core engine implementation
+2. Basic text generation (With character switching, 
+   - Language switching (if applicable)
+   - Pause tag support
+   - Caching system
+   - VRAM management)
+3. Model loading and downloading
+4. Engine configuration node
+5. Integration with TTS Text node
+
+**Stop here and test with user before proceeding**
+
+### Phase 2: SRT Support
+
+1. SRT processor implementation
+2. Timing and assembly integration
+3. Character switching in SRT
+4. Performance optimization
+
+### Phase 3: Special Features
+
+1. Engine-specific unique features
+2. Special nodes for unique capabilities
+3. Advanced parameter controls
+
+### Phase 4: Documentation and Polish
+
+1. README updates
+2. Example workflows
+3. Performance testing
+4. Error handling improvements
+
+---
+
+## Critical Integration Points
+
+### Must Use These Unified Systems
+
+#### ✅ Required Integrations
+
+- **UnifiedModelInterface** - Model loading
+- **ComfyUIModelWrapper** - VRAM management
+- **UnifiedCacheManager** - Audio caching
+- **CharacterParser** - Character tag parsing
+- **PauseTagProcessor** - Pause tag handling
+- **LanguageMapper** - Language code mapping
+- **UnifiedDownloader** - Model downloads
+- **model_management.interrupt_processing** - Interrupt signal checking (CRITICAL for cancellation)
+
+#### ❌ Never Duplicate These
+
+- Character parsing logic
+- Pause tag parsing logic
+- Language mapping logic
+- Cache key generation
+- Model management logic
+- Audio format conversion utilities
+
+---

docs/PARAMETER_SWITCHING_GUIDE.md

@@ -0,0 +1,280 @@
+# ⚙️ Per-Segment Parameter Switching Guide
+
+## Overview
+
+The TTS Audio Suite supports **per-segment parameter control** through inline tags. This allows you to override TTS generation parameters (seed, temperature, cfg, speed, etc.) on a per-segment basis, providing fine-grained control over individual audio segments without modifying node-level defaults.
+
+Parameters are applied **only to the current segment** and automatically revert to node defaults for subsequent segments.
+
+---
+
+## ✨ Key Features
+
+- **⚙️ Per-Segment Control** - Override parameters for individual text segments or SRT subtitle entries
+- **🔀 Order-Independent** - Specify parameters in any order: `[Alice|seed:42|temp:0.5]` or `[seed:42|Alice|temp:0.5]`
+- **📝 Alias Support** - Use shortcuts like `temp` (temperature), `cfg_weight` (cfg), `exag` (exaggeration)
+- **🔤 Case-Insensitive** - `[SEED:42]`, `[Seed:42]`, `[seed:42]` all work identically
+- **🚨 Smart Filtering** - Unsupported parameters are silently ignored with warnings (e.g., `[speed:2.0]` on ChatterBox)
+- **📺 SRT Support** - Apply parameters to individual SRT subtitle lines
+- **🔙 Backward Compatible** - Works alongside character switching: `[de:Alice|seed:42|temp:0.7]`
+
+---
+
+## 🚀 Quick Start
+
+### Basic Parameter Syntax
+
+**Format:** `[character|parameter:value]` or `[parameter:value|character]`
+
+```
+[Alice|seed:42] Hi there with a specific seed!
+[Bob|temperature:0.5] Bob speaks more deterministically.
+[Alice|seed:42|temperature:0.7] Alice with both seed and temperature.
+[seed:42|Alice] Order doesn't matter!
+```
+
+### Real-World Examples
+
+#### Consistent Character Voice
+```
+[Alice|seed:42] This is the first segment of Alice.
+[Alice|seed:42] This is the second segment, sounding identical.
+[Alice|seed:42] Third segment with same seed.
+```
+
+#### Varied Voice Characteristics
+```
+[Alice|temperature:0.3] Alice speaks carefully and precisely.
+[Alice|temperature:0.8] Alice speaks more creatively and varied.
+[Alice|temperature:0.3] Alice is precise again.
+```
+
+#### SRT Example with Parameters
+```srt
+1
+00:00:01,000 --> 00:00:04,000
+[Alice|seed:42] Hello! I'm Alice and this line has a specific seed.
+
+2
+00:00:04,500 --> 00:00:09,500
+[Bob|seed:100] And I'm Bob with my own unique seed.
+
+3
+00:00:10,000 --> 00:00:14,000
+[Alice|seed:42] Alice again with the same seed as segment 1.
+```
+
+---
+
+## 📋 Supported Parameters
+
+### Universal Parameters (All Engines)
+
+| Parameter | Alias | Type | Range | Description |
+|-----------|-------|------|-------|-------------|
+| `seed` | — | int | 0-4294967295 | Random seed for reproducible generation |
+| `temperature` | `temp` | float | 0.1-2.0 | Randomness/creativity (lower=more deterministic) |
+
+### Engine-Specific Parameters
+
+#### ChatterBox & ChatterBox Official 23-Lang
+| Parameter | Alias | Type | Range | Description |
+|-----------|-------|------|-------|-------------|
+| `cfg` | `cfg_weight` | float | 0.0-1.0 | CFG weight for speaker guidance |
+| `exaggeration` | `exag` | float | 0.0-2.0 | Voice emotion exaggeration |
+
+#### F5-TTS
+| Parameter | Alias | Type | Range | Description |
+|-----------|-------|------|-------|-------------|
+| `cfg` | — | float | 0.0-20.0 | Classifier-free guidance strength |
+| `speed` | — | float | 0.5-2.0 | Speech speed multiplier |
+
+#### Higgs Audio 2
+| Parameter | Alias | Type | Range | Description |
+|-----------|-------|------|-------|-------------|
+| `cfg` | — | float | 0.0-20.0 | CFG strength |
+| `top_p` | `topp` | float | 0.0-1.0 | Nucleus sampling probability |
+| `top_k` | `topk` | int | 1-100 | Top-k sampling |
+
+#### VibeVoice
+| Parameter | Alias | Type | Range | Description |
+|-----------|-------|------|-------|-------------|
+| `cfg` | — | float | 0.0-20.0 | CFG strength |
+| `top_p` | `topp` | float | 0.0-1.0 | Nucleus sampling probability |
+| `inference_steps` | `steps` | int | 1-100 | Number of inference steps |
+
+#### IndexTTS-2
+| Parameter | Alias | Type | Range | Description |
+|-----------|-------|------|-------|-------------|
+| `cfg` | — | float | 0.0-20.0 | CFG strength |
+| `top_p` | `topp` | float | 0.0-1.0 | Nucleus sampling probability |
+| `top_k` | `topk` | int | 1-100 | Top-k sampling |
+| `emotion_alpha` | — | float | 0.0-1.0 | Emotion control strength |
+
+---
+
+## 🎯 Advanced Usage
+
+### Combining with Character Switching
+
+Parameters work seamlessly with character and language tags:
+
+```
+[de:Alice|seed:42] German Alice with specific seed.
+[fr:Bob|temperature:0.7] French Bob with higher temperature.
+[seed:123|de:Alice] Order-independent syntax.
+```
+
+### Per-Segment Fine-Tuning in SRT
+
+Create dynamic SRT content with per-segment control:
+
+```srt
+1
+00:00:00,500 --> 00:00:03,000
+[Narrator|temp:0.4] The important introduction needs precision.
+
+2
+00:00:03,100 --> 00:00:06,500
+[Narrator|temp:0.4] Continuing with careful, consistent delivery.
+
+3
+00:00:06,600 --> 00:00:10,000
+[Narrator|temp:0.7] Now for the creative narrative section!
+
+4
+00:00:10,100 --> 00:00:14,000
+[Narrator|temp:0.4] Back to precise delivery for the conclusion.
+```
+
+### Alias Examples
+
+Use shorter aliases for convenience:
+
+```
+[Alice|temp:0.5]           # temperature
+[Alice|cfg_weight:0.7]     # Same as cfg:0.7
+[Alice|exag:1.5]           # exaggeration
+[Alice|steps:50]           # inference_steps (VibeVoice)
+[Alice|topk:20]            # top_k
+[Alice|topp:0.95]          # top_p
+```
+
+### Semantic Parameters for Control
+
+#### Temperature Examples
+```
+[Alice|temperature:0.2] This is spoken very carefully, word for word.
+[Alice|temperature:0.5] This is natural but somewhat consistent.
+[Alice|temperature:0.9] This is creative with varied expression.
+```
+
+#### Seed Examples
+```
+# Same seed = identical audio output
+[Alice|seed:42] First attempt.
+[Alice|seed:42] Identical to first attempt.
+
+# Different seeds = different variations
+[Alice|seed:42] Variation 1.
+[Alice|seed:100] Variation 2.
+[Alice|seed:999] Variation 3.
+```
+
+---
+
+## ⚠️ Important Notes
+
+### Parameter Validation
+
+- **Invalid values** are logged with warnings but don't cause errors
+- **Unsupported parameters** for your engine are silently ignored
+- **Out-of-range values** are clamped to valid ranges
+- **Type mismatches** (e.g., `temperature:abc`) are logged and rejected
+
+### Engine Compatibility
+
+Different engines support different parameters. The system automatically:
+- ✅ Applies supported parameters
+- ⚠️ Warns about unsupported parameters
+- ✅ Continues processing without errors
+
+Example:
+```
+[Alice|speed:0.8|temperature:0.5]  # ChatterBox: ignores speed, applies temperature
+[Alice|speed:0.8|temperature:0.5]  # F5-TTS: applies both
+```
+
+### Performance Considerations
+
+- Parameter extraction happens during text parsing (minimal overhead)
+- Parameters only affect the specific segment
+- No impact on caching or model loading
+
+---
+
+## 🔧 Implementation Notes
+
+### For Node Developers
+
+If you're adding parameter support to a new engine:
+
+1. Import the parameter system:
+   ```python
+   from utils.text.segment_parameters import apply_segment_parameters
+   ```
+
+2. When processing segments with parameters:
+   ```python
+   if segment.parameters:
+       # Apply segment parameters to override config
+       segment_config = apply_segment_parameters(base_config, segment.parameters, engine_type)
+   else:
+       segment_config = base_config
+   ```
+
+3. Generate audio with `segment_config` instead of `base_config`
+
+### Parameter Storage
+
+Parameters are stored in the `CharacterSegment.parameters` dictionary:
+```python
+segment.parameters = {
+    'seed': 42,
+    'temperature': 0.5,
+    'cfg': 0.7
+}
+```
+
+---
+
+## 📚 See Also
+
+- [Character Switching Guide](CHARACTER_SWITCHING_GUIDE.md) - Multi-character and language switching
+- [ChatterBox Special Tokens Guide](CHATTERBOX_V2_SPECIAL_TOKENS.md) - Emotion and sound tokens
+- README.md - Complete feature overview
+
+---
+
+## 🐛 Troubleshooting
+
+### Parameters Not Being Applied
+
+1. **Check engine compatibility** - Parameter might not be supported by your engine
+2. **Verify syntax** - Use `|` to separate parameters: `[Alice|seed:42|temp:0.5]`
+3. **Check for typos** - Parameter names are case-insensitive but must match known parameters
+4. **Look at console** - Warnings are logged when parameters are unsupported or invalid
+
+### "Parameter not supported" Warning
+
+This is normal! Different engines support different parameters. For example:
+- ChatterBox doesn't support `speed` (F5-TTS-only parameter)
+- F5-TTS doesn't support `exaggeration` (ChatterBox-only parameter)
+
+The system silently ignores unsupported parameters and continues processing.
+
+### Values Not Taking Effect
+
+- **Check range** - Values outside the valid range are clamped to min/max
+- **Verify engine** - Make sure you're using an engine that supports that parameter
+- **Check node defaults** - Parameters only override node-level defaults, they don't create new capabilities

docs/POST_V4.4.0_DEVELOPMENT_REVIEW.md

@@ -0,0 +1,201 @@
+# Post v4.4.0 Development Review
+
+*Analysis of 32 local commits since v4.4.0 release*
+
+## Executive Summary
+
+Since the v4.4.0 release featuring the Silent Speech Analyzer, development has focused on three major areas: **Higgs Audio 2 integration**, **architectural refinement**, and **smart processing optimization**. These 32 commits represent significant enhancements that maintain the project's momentum toward becoming the ultimate universal TTS platform.
+
+## Major New Features
+
+### 🎯 **Higgs Audio 2 Engine Integration** (MAJOR FEATURE)
+
+**Commits**: `7a7c205`, `fa0a6d8`, `c6d6efb`, `d750190`, `31b8aed`, `2c8f955`, `460da24`, `d94df09`
+
+**What's New**:
+- **Complete Higgs Audio 2 TTS engine** with state-of-the-art voice cloning capabilities
+- **Real-time voice cloning** from 30+ second reference audio samples
+- **Multi-speaker conversation processing** with native character switching
+- **8 premium voice presets** (belinda, en_woman, en_man, chadwick, vex, etc.)
+
+**Technical Implementation**:
+- **10,000+ lines of code** added for complete engine integration
+- **Modular architecture** following established F5-TTS/ChatterBox patterns
+- **Unified adapter integration** - works seamlessly with existing TTS Text/SRT nodes
+- **Automatic model downloading** to organized TTS/HiggsAudio/ structure
+- **Smart chunking system** for unlimited length text generation
+
+**Voice Cloning Capabilities**:
+- High-quality voice replication from short audio samples
+- Multi-language support (English, Chinese dialects)
+- Configurable generation parameters (temperature, top_p, top_k, max_new_tokens)
+- Custom reference audio support beyond preset voices
+
+### 🧠 **Smart Modular Chunk Combination System** (PERFORMANCE)
+
+**Commits**: `bda23d2`, `19323f1`
+
+**What's New**:
+- **ChunkCombiner utility** with intelligent auto-selection algorithms
+- **Smart analysis** of chunk split patterns (sentence boundaries, commas, forced splits)
+- **Per-junction analysis** for optimal audio segment combination
+- **Universal implementation** across all TTS engines (ChatterBox, F5-TTS, Higgs Audio)
+
+**Technical Innovation**:
+- **202+ lines** of sophisticated chunk analysis logic
+- **ChunkTiming utility** providing standardized timing info integration
+- **Auto-selection method** that analyzes text structure for optimal processing
+- **Cache optimization** maintained through post-processing approach
+
+**Performance Impact**:
+- Improved audio quality through intelligent segment boundaries
+- Reduced artifacts from poor chunk combinations
+- Consistent processing across all engines
+- Enhanced timing accuracy for SRT processing
+
+### 🏗️ **Modular Processor Architecture Completion** (ARCHITECTURE)
+
+**Commits**: `460da24`, `d94df09`, `c9f648f`
+
+**What's New**:
+- **HiggsAudioSRTProcessor** and **HiggsAudioTTSProcessor** - dedicated engine processors
+- **Modular SRT overlap detection** across all TTS engines
+- **Clean delegation pattern** - Unified nodes now 23 lines vs 267 lines of hardcoded logic
+- **Architectural consistency** achieved across all engines
+
+**Technical Benefits**:
+- **267 lines of hardcoded logic** extracted into dedicated processors
+- **Unified architecture** - all engines now follow identical delegation patterns
+- **Easier maintenance** and debugging through modular separation
+- **Consistent error handling** and processing across engines
+
+## Quality of Life Improvements
+
+### 📊 **Enhanced User Experience**
+
+**Progress Visualization**: 
+- **tqdm progress bars** added to Higgs Audio generation (`83c5274`)
+- Real-time feedback for long-running operations
+- Better user awareness of processing status
+
+**Improved Documentation**:
+- **Detailed tooltips** for Higgs Audio Engine parameters (`240fbb7`)
+- **Multi-speaker mode recommendations** for optimal usage (`1235faf`)
+- Clear guidance on parameter usage and expected results
+
+**Parameter Optimization**:
+- **Fixed max_new_tokens minimum** from 128 to 1 (`40c1da4`)
+- More flexible generation control for varied use cases
+- Better parameter validation and user guidance
+
+### 🔧 **Critical Bug Fixes**
+
+**Cache System Fixes**:
+- **Critical cache invalidation bug** fixed for Higgs Audio (`5ee6b10`, `ba8bace`)
+- **Centralized cache system** improvements for consistency
+- **Parameter flow fixes** for proper SRT node configuration (`083d865`)
+
+**Engine Integration Fixes**:
+- **F5-TTS bundled integration** improvements (`34db557`)
+- **ChatterBox model detection** enhancements
+- **Character switching** fixes for narrator voice priority (`9e11f12`)
+
+**Initialization and Parameter Flow**:
+- **Engine initialization bypass bug** fixed (`a676dad`, `ab90051`)
+- **SRT stretch_to_fit mode** parameter format corrections (`1912250`)
+- **Timing report generation** using modular calculation system (`762e472`)
+
+## Technical Architecture Improvements
+
+### Unified Processing Pipeline
+
+**Before**: Each engine had custom SRT processing with duplicated logic
+**After**: Clean delegation to dedicated engine processors
+
+**Benefits**:
+- Consistent behavior across all engines
+- Easier debugging and maintenance
+- Reduced code duplication
+- Modular testing capabilities
+
+### Enhanced Cache Management
+
+**Centralized Cache System**:
+- Universal cache invalidation across all engines
+- Content-based hashing for consistency
+- Engine-specific optimizations while maintaining unified interface
+
+### Voice Management Evolution
+
+**Higgs Audio Voice System**:
+- **Removed voice_preset system** in favor of flexible reference text handling (`a112ee5`)
+- **Improved reference text processing** for better voice cloning results
+- **Character switching compatibility** with existing voice management
+
+## Code Quality and Maintenance
+
+### Repository Management
+
+**Cleanup and Organization**:
+- **ComfyUI reference folders** properly gitignored (`fe9dcd6`, `24982e5`)
+- **Requirements.txt updates** for Higgs Audio dependencies (`00483cc`)
+- **README consolidation** for better feature presentation (`4111d64`)
+
+**Merge Management**:
+- **External contributions** properly integrated (`6f3452d`, `894692c`)
+- Clean merge handling without conflicts
+
+### Development Process
+
+**32 Commits** of focused development since v4.4.0:
+- **1 Major Feature** (Higgs Audio 2 integration)
+- **2 Performance Systems** (Smart chunking, modular processors)
+- **15+ Bug fixes** and stability improvements
+- **Multiple UX enhancements**
+
+## Impact Analysis
+
+### Feature Completeness
+
+The addition of Higgs Audio 2 completes the "Big Three" TTS engine integration:
+1. ✅ **ChatterBox** - Fast, reliable, multilingual
+2. ✅ **F5-TTS** - High-quality, speech editing
+3. ✅ **Higgs Audio 2** - Voice cloning, multi-speaker
+
+### Performance Optimization
+
+- **Smart chunking** reduces audio artifacts
+- **Modular processors** improve maintainability
+- **Enhanced caching** provides instant regeneration
+- **Progress feedback** improves user experience
+
+### Architecture Maturity
+
+- **Unified delegation pattern** completed across all engines
+- **Modular utilities** enable consistent behavior
+- **Clean separation** of concerns for easier development
+- **Future-proof design** for additional engine integration
+
+## Upcoming Version Readiness
+
+These 32 commits represent substantial improvements worthy of a **v4.5.0 release**:
+
+**Major Features**: Higgs Audio 2 integration with voice cloning
+**Performance**: Smart chunk combination system
+**Architecture**: Modular processor completion
+**Stability**: 15+ critical bug fixes
+**UX**: Enhanced progress feedback and documentation
+
+## Conclusion
+
+The post-v4.4.0 development cycle demonstrates continued innovation and architectural refinement. The **Higgs Audio 2 integration** brings cutting-edge voice cloning capabilities, while the **smart chunking system** and **modular processor architecture** ensure robust, maintainable code.
+
+These improvements maintain the project's position as the most comprehensive TTS platform available, with three world-class engines unified under a single, intuitive interface.
+
+---
+
+*Review Date*: August 21, 2025  
+*Commits Analyzed*: 32 local commits since v4.4.0  
+*Major Features Added*: 1 (Higgs Audio 2)  
+*Performance Systems*: 2 (Smart chunking, Modular processors)  
+*Bug Fixes*: 15+

docs/Upstream_Engine_Tracking/INDEXTTS2_UPDATE_LOG.md

@@ -0,0 +1,225 @@
+# IndexTTS-2 Updates Log
+
+This document tracks updates applied to our bundled IndexTTS-2 code from the upstream repository.
+
+## Update Process
+1. **Check for upstream changes:** `cd IgnoredForGitHubDocs/For_reference/index_tts && git pull`
+2. **Review changes:** `git diff OLD_COMMIT..NEW_COMMIT -- indextts/infer_v2.py`
+3. **Apply relevant changes** to `engines/index_tts/indextts/infer_v2.py`
+4. **Test functionality** with our IndexTTS-2 nodes
+5. **Document changes** in this log
+
+---
+
+## 2025-09-18: Major Update - Cache & Emotion Improvements
+
+**Reference commit range:** `8336824..64cb31a` (September 11 → September 18, 2025)
+**Upstream commits reviewed:** 5 commits affecting `infer_v2.py`
+
+### 🔧 Changes Applied:
+
+#### 1. **Emotion Vector Normalization Function** (commit: 8aa8064)
+- **Added:** `normalize_emo_vec()` helper function
+- **Purpose:** Prevents voice identity loss from overly strong emotion settings
+- **Implementation:**
+  - Applies emotion bias factors to reduce problematic emotions
+  - Caps total emotion sum to 0.8 maximum
+  - Protects against extreme emotion vector values
+- **User benefit:** Better emotion control without losing speaker characteristics
+
+#### 2. **Audio Length Limiting** (commit: 0828dcb)
+- **Added:** `_load_and_cut_audio()` helper function
+- **Purpose:** Prevents memory/VRAM issues from overly long reference audio
+- **Implementation:**
+  - Automatically truncates audio to 15 seconds maximum
+  - Supports different sample rates (16kHz, 22kHz)
+  - Provides verbose logging of truncation
+- **User benefit:** More stable generation with long reference audio files
+
+#### 3. **Persistent Cache Buildup Fix** (commit: 64cb31a)
+- **Added:** Proper cache clearing with `torch.cuda.empty_cache()`
+- **Purpose:** Solves memory accumulation issues during multiple generations
+- **Implementation:**
+  - Clears old cache variables before loading new audio
+  - Calls `torch.cuda.empty_cache()` to free GPU memory
+  - Applied to both speaker and emotion cache invalidation
+- **User benefit:** Better memory management during batch processing
+
+#### 4. **BigVGAN CUDA Kernel Import Fix** (commits: ee23371, e409c4a)
+- **Fixed:** Corrected import path for BigVGAN custom CUDA kernel
+- **Changed:** `indextts.BigVGAN.alias_free_activation.cuda` → `indextts.s2mel.modules.bigvgan.alias_free_activation.cuda`
+- **Purpose:** Ensures proper CUDA acceleration when available
+- **User benefit:** Better performance with CUDA-enabled setups
+
+### 📋 Implementation Status:
+
+| Change | Applied to Bundled Code | Tested | Notes |
+|--------|------------------------|--------|-------|
+| `normalize_emo_vec()` | ✅ **Applied** | ⏳ Pending | Core emotion control improvement |
+| `_load_and_cut_audio()` | ✅ **Applied** | ⏳ Pending | Memory stability fix |
+| Cache clearing fixes | ✅ **Applied** | ⏳ Pending | Critical for batch processing |
+| BigVGAN import fix | ✅ **Applied** | ⏳ Pending | Performance improvement |
+
+### 🎯 User-Facing Improvements:
+- **Better emotion control:** Emotion vectors now auto-normalize to prevent voice identity loss
+- **Memory stability:** Long reference audio automatically truncated to prevent crashes
+- **Batch processing:** Improved cache management for multiple generations
+- **Performance:** Fixed CUDA kernel loading for faster inference
+
+### 🧪 Testing Required:
+- [ ] Emotion vector controls in radar chart widget
+- [ ] Long reference audio handling (>15 seconds)
+- [ ] Multiple sequential generations (cache clearing)
+- [ ] CUDA performance with FP16 enabled
+
+---
+
+---
+
+## 2025-11-06: Performance & Stability Improvements
+
+**Reference commit range:** `64cb31a..1d5d079` (September 18 → November 6, 2025)
+**Upstream commits reviewed:** 6 commits affecting `infer_v2.py`
+
+### 🔧 New Changes Identified:
+
+#### 1. **Empty Generator → IndexError Fix** (commit: 750d9d9)
+- **Issue:** Non-streaming `infer()` would crash with IndexError on empty generator
+- **Fix:** Wrapped generator list conversion in try/except, returns `None` on error
+- **Changes to `infer_v2.py`:**
+  - Lines 354-364: Added exception handling
+  - Lines 680, 690: Changed `return` to `yield` for proper generator flow
+- **Priority:** 🔴 **Critical** - Prevents runtime crashes
+
+#### 2. **Streaming Implementation** (commit: b0c6ab8)
+- **Added:** `stream_return` and `more_segment_before` parameters for low-latency streaming
+- **Purpose:** Enable first-chunk audio output before full generation completes
+- **Changes:**
+  - `stream_return` (bool): Return generator instead of final result
+  - `more_segment_before` (int): Apply extra segmentation for faster first output (0-80 recommended)
+  - Modified `indextts/utils/front.py` for tokenization changes
+- **Priority:** 🟡 **Important** - Enables streaming use cases
+
+#### 3. **UNK Token Warnings** (commit: 34be9bf)
+- **Added:** Detection and warning for unknown tokens in input text
+- **Behavior:**
+  - Detects tokens the BPE model cannot encode
+  - Prints warnings with token IDs and suggested fixes
+  - Helps users debug encoding issues
+- **Changes to `infer_v2.py`:** Lines 450-459, added token validation logic
+- **Priority:** 🟢 **Nice-to-have** - Improves user experience with error diagnosis
+
+#### 4. **S2Mel Stage Optimization** (commit: 31e7e85)
+- **Focus:** Optimized mel-spectrogram generation pipeline
+- **Changes:**
+  - `indextts/s2mel/modules/commons.py`: 15 lines added/modified
+  - `indextts/s2mel/modules/diffusion_transformer.py`: 2 lines modified
+  - `indextts/s2mel/modules/flow_matching.py`: 15 lines added (new optimization code)
+  - `indextts/infer_v2.py`: 28 lines added/modified
+- **Expected Impact:** Faster mel-spectrogram generation (stage 2 of 3)
+- **Priority:** 🟡 **Important** - Performance improvement
+
+#### 5. **GPT2 Inference Acceleration** (commit: c1ef414)
+- **Focus:** Major performance optimization for GPT2 stage (stage 1)
+- **Changes:**
+  - **New module:** `indextts/accel/` directory (609 lines added):
+    - `accel_engine.py`: Core acceleration logic (609 lines)
+    - `attention.py`: Optimized attention mechanism (154 lines)
+    - `gpt2_accel.py`: GPT2-specific optimizations (181 lines)
+    - `kv_manager.py`: Key-value cache management (209 lines)
+  - Modified `indextts/gpt/model_v2.py`: 63 lines changed (+47 new)
+  - Modified `indextts/infer_v2.py`: 17 lines changed
+- **Expected Impact:** Significantly faster GPT2 inference (first stage bottleneck)
+- **Priority:** 🔴 **Critical** - Major performance gain, but new code complexity
+
+#### 6. **Merge Commit** (commit: 5d67f62 → 1d5d079)
+- Merge of acceleration branch into main
+- Contains all above changes integrated
+
+### 📊 Summary
+
+| Change | Files Modified | Lines Changed | Priority | Status |
+|--------|---------------|---------------|----------|--------|
+| Empty generator fix | 1 | +12/-9 | 🔴 Critical | ⏳ Pending |
+| Streaming support | 2 | +72/-18 | 🟡 Important | ⏳ Pending |
+| UNK token warnings | 1 | +7 | 🟢 Nice-to-have | ⏳ Pending |
+| S2Mel optimization | 4 | +54/-6 | 🟡 Important | ⏳ Pending |
+| GPT2 acceleration | 6 | +1228/-14 | 🔴 Critical | ⏳ Pending |
+
+### 🚨 Integration Concerns
+
+1. **New acceleration module complexity** - The `indextts/accel/` package adds significant complexity with KV-cache management and specialized attention mechanisms. Needs careful testing.
+
+2. **Potential API changes** - Streaming parameters (`stream_return`, `more_segment_before`) may require adapter updates if our wrapper doesn't expose these.
+
+3. **GPU memory implications** - KV-cache optimization might have different memory profiles; needs profiling on different GPU setups.
+
+4. **File structure expansion** - New `accel/` subdirectory adds to maintenance burden.
+
+### 🧪 Testing Required
+
+- [ ] Empty generator fix - test non-streaming inference with various inputs
+- [ ] Streaming latency - measure first-chunk output time with `stream_return=True`
+- [ ] GPT2 acceleration - benchmark inference speed (expect 1.5-3x speedup)
+- [ ] S2Mel optimization - verify audio quality unchanged
+- [ ] UNK token handling - test with multilingual/special character inputs
+- [ ] Memory profiling - GPU memory usage with new KV-cache manager
+- [ ] ComfyUI integration - ensure all changes work through our node interface
+
+### ✅ Applied Changes Summary (2025-11-06)
+
+**All upstream commits successfully integrated:**
+
+1. ✅ **Empty generator fix (750d9d9)** - Refactored `infer()` to use generator pattern:
+   - Created wrapper `infer()` method that delegates to `infer_generator()`
+   - Refactored monolithic method into `infer_generator()` with yield statements
+   - Changed final `return` statements to `yield` for generator compatibility
+   - Handles both streaming (`stream_return=True`) and non-streaming modes
+
+2. ✅ **Streaming implementation (b0c6ab8)** - Added streaming parameters:
+   - Added `stream_return` parameter to `infer()` and `infer_generator()`
+   - Added `more_segment_before` parameter for streaming segmentation
+   - Integrated with generator-based inference for low-latency streaming
+
+3. ✅ **UNK token warnings (34be9bf)** - Added token validation:
+   - Added token validation in `infer_generator()`
+   - Warns users about unknown tokens with suggestions
+
+4. ✅ **S2Mel optimization (31e7e85)** - Applied all module updates:
+   - `commons.py`: Updated `fused_add_tanh_sigmoid_multiply()` with torch.split
+   - `diffusion_transformer.py`: Fixed `sequence_mask()` call with max_length parameter
+   - `flow_matching.py`: Added `enable_torch_compile()` method to CFM class
+   - `infer_v2.py`: Added `use_torch_compile` parameter and setup
+
+5. ✅ **GPT2 acceleration (c1ef414)** - Applied complete acceleration module:
+   - Created `/accel/` directory with 5 files (init, accel_engine, attention, gpt2_accel, kv_manager)
+   - Modified `model_v2.py`: Added `use_accel` parameter to UnifiedVoice
+   - Modified `model_v2.py`: Added accel engine initialization in `post_init_gpt2_config()`
+   - Modified `model_v2.py`: Added accel-aware inference in `inference_speech()`
+
+### 🔧 Custom Modifications Preserved
+
+- QwenEmotion lazy loading with path fallbacks (lines 81-112 in infer_v2.py)
+- W2V-BERT model loading from TTS folder with unified downloader (lines 145-166)
+- All custom logging and error handling in infer_v2.py and throughout
+- Custom model initialization sequence
+- Custom emotion vector normalization and processing
+
+---
+
+## Next Update Check: 2025-11-20
+
+**Monitoring:** Watch for commits to `indextts/infer_v2.py` in https://github.com/index-tts/index-tts
+
+**Update command:**
+```bash
+cd /tmp/index_tts_upstream
+git pull origin main
+git log --oneline 1d5d079..HEAD -- indextts/infer_v2.py
+```
+
+**Priority changes to watch for:**
+- Additional performance optimizations
+- Bug fixes in new acceleration code
+- Breaking API changes
+- Model loading improvements

docs/Upstream_Engine_Tracking/UPDATE_LOG_TEMPLATE.md

@@ -0,0 +1,103 @@
+# [ENGINE_NAME] Upstream Update Log
+
+**Engine:** [ENGINE_NAME]
+**Official Repository:** [UPSTREAM_URL]
+**Bundled Location:** `engines/[engine_folder]/`
+**Last Checked:** YYYY-MM-DD
+**Last Checked By:** [username]
+
+---
+
+## Check History
+
+| Date | Commits Checked | New Features | Bug Fixes | Breaking Changes | Notes |
+|------|-----------------|--------------|-----------|------------------|-------|
+| YYYY-MM-DD | N/A | - | - | - | Initial setup |
+
+---
+
+## Latest Upstream Changes (Since Last Check)
+
+### New Features 🆕
+- Feature 1
+- Feature 2
+
+### Bug Fixes 🐛
+- Fix 1
+- Fix 2
+
+### Breaking Changes ⚠️
+- Change 1
+- Change 2
+
+### Performance Improvements ⚡
+- Improvement 1
+
+### Documentation Updates 📝
+- Update 1
+
+---
+
+## Integration Status
+
+| Item | Status | Priority | Notes |
+|------|--------|----------|-------|
+| Feature/Fix name | `pending`/`integrated`/`skipped` | 🔴/🟡/🟢 | Integration notes |
+
+### Legend
+- `pending` - Identified but not yet integrated
+- `integrated` - Applied to bundled code
+- `skipped` - Reviewed but decided not to integrate
+- `wontfix` - Not applicable to our implementation
+
+---
+
+## Integration Notes
+
+### Conflicts with Our Customizations
+- [List any conflicts between upstream changes and our custom modifications]
+
+### Testing Required
+- [ ] Test case 1
+- [ ] Test case 2
+
+### Version Bump Recommendation
+- Current bundled version: X.Y.Z
+- Latest upstream version: X.Y.Z
+- Recommended bump: `patch` / `minor` / `major` / `none`
+
+---
+
+## Known Customizations We Maintain
+
+List any intentional differences from upstream:
+- Customization 1
+- Customization 2
+
+---
+
+## Next Check
+
+**Scheduled for:** YYYY-MM-DD
+**Check interval:** Weekly / Bi-weekly / Monthly
+
+---
+
+## Previous Check Summary
+
+Link to previous check details if available.
+
+---
+
+## Quick Commands
+
+```bash
+# Clone/update upstream repository
+git clone [UPSTREAM_URL] /tmp/[engine_name]_upstream
+
+# Compare with our bundled version
+diff -r /tmp/[engine_name]_upstream/[source_folder] engines/[bundled_folder]/
+
+# Show recent commits from upstream
+git log --oneline -20 origin/main
+```

docs/Upstream_Engine_Tracking/UPSTREAM_ENGINES_INDEX.md

@@ -0,0 +1,196 @@
+# Upstream TTS Engines Tracking Index
+
+This document provides a centralized reference for monitoring updates to all bundled TTS engines. Each engine has its own update log file tracking changes, bug fixes, and new features from the official upstream repository.
+
+**Last Updated:** 2025-11-06
+
+---
+
+## Quick Reference Table
+
+| Engine | Bundled Source | Location | Official Repository | Update Log | Last Checked |
+|--------|---------|----------|---------------------|------------|--------------|
+| **IndexTTS-2** | ✅ Full | `engines/index_tts/indextts/` | [github.com/index-tts/index-tts](https://github.com/index-tts/index-tts) | [INDEXTTS2_UPDATE_LOG.md](./INDEXTTS2_UPDATE_LOG.md) | 2025-11-06 |
+| **ChatterBox** | ✅ Full | `engines/chatterbox/` | [github.com/resemble-ai/chatterbox](https://github.com/resemble-ai/chatterbox) | [CHATTERBOX_UPDATE_LOG.md](./CHATTERBOX_UPDATE_LOG.md) | TBD |
+| **ChatterBox Multilingual (23-Lang)** | ✅ Full | `engines/chatterbox_official_23lang/` | [github.com/resemble-ai/chatterbox](https://github.com/resemble-ai/chatterbox) | [CHATTERBOX_UPDATE_LOG.md](./CHATTERBOX_UPDATE_LOG.md) | TBD |
+| **F5-TTS** | ✅ Full | `engines/f5_tts/` | [github.com/SWivid/F5-TTS](https://github.com/SWivid/F5-TTS) | [F5TTS_UPDATE_LOG.md](./F5TTS_UPDATE_LOG.md) | TBD |
+| **Higgs Audio** | ✅ Full | `engines/higgs_audio/boson_multimodal/` | [github.com/boson-ai/higgs-audio](https://github.com/boson-ai/higgs-audio) | [HIGGS_AUDIO_UPDATE_LOG.md](./HIGGS_AUDIO_UPDATE_LOG.md) | TBD |
+| **RVC** | ✅ Full | `engines/rvc/` | [github.com/RVC-Project/Retrieval-based-Voice-Conversion](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion) | [RVC_UPDATE_LOG.md](./RVC_UPDATE_LOG.md) | TBD |
+| **VibeVoice** | ⚠️ Wrapper Only | `engines/vibevoice_engine/` | [github.com/microsoft/VibeVoice](https://github.com/microsoft/VibeVoice) ⚠️ Disabled | [VIBEVOICE_UPDATE_LOG.md](./VIBEVOICE_UPDATE_LOG.md) | TBD |
+
+---
+
+## Engine Status Details
+
+### ✅ Fully Bundled Engines (Source Code Included)
+
+These engines have their complete source code bundled in the TTS Audio Suite repository:
+
+1. **IndexTTS-2** - Full inference code and models bundled
+2. **ChatterBox** - Full model and inference code bundled
+3. **ChatterBox Multilingual** - Wrapper around same ChatterBox repo with language support
+4. **F5-TTS** - Full source code with configs, models, and training scripts
+5. **Higgs Audio** - Full boson_multimodal framework bundled
+6. **RVC** - Complete voice conversion implementation
+
+### ⚠️ Wrapper-Only Engines
+
+These are external dependencies integrated via wrappers:
+
+- **VibeVoice** - Wrapper around Microsoft's `vibevoice` PyPI package (⚠️ **Note**: Official repo is disabled by Microsoft as of Sept 2025)
+
+---
+
+## Update Checking Workflow
+
+### Manual Checking (via slash command)
+```bash
+/check_tts_bundled_updates [engine_name] [--verbose]
+```
+
+Examples:
+- `/check_tts_bundled_updates` - Check ALL engines
+- `/check_tts_bundled_updates IndexTTS-2` - Check specific engine
+- `/check_tts_bundled_updates ChatterBox --verbose` - Verbose output with detailed diffs
+
+### What Gets Checked
+
+For each engine, the process:
+1. Clones/updates the official upstream repository
+2. Compares commit history since last check
+3. Identifies new features, bug fixes, breaking changes
+4. Updates the engine's UPDATE_LOG.md file
+5. Flags items requiring integration
+
+### Priority Levels for Updates
+
+- 🔴 **Critical**: Security fixes, breaking changes, major bug fixes
+- 🟡 **Important**: New features, performance improvements, compatibility updates
+- 🟢 **Nice-to-have**: Documentation, minor tweaks, non-critical improvements
+
+---
+
+## Individual Engine Logs
+
+Each engine has its own update log tracking:
+- **Date checked**: When upstream was last reviewed
+- **New commits**: List of changes since last version
+- **Actionable items**: What needs integration/testing
+- **Integration status**: Whether changes have been applied
+
+See individual UPDATE_LOG.md files in this directory for details.
+
+---
+
+## Integration Checklist
+
+When integrating upstream updates:
+
+- [ ] Read upstream changelog/commits
+- [ ] Identify compatibility issues with our wrapper code
+- [ ] Create backup of current bundled code
+- [ ] Apply changes incrementally
+- [ ] Run full test suite
+- [ ] Document any conflicts or customizations we maintain
+- [ ] Update version in nodes.py if integration warrants version bump
+- [ ] Commit with reference to upstream changes
+
+---
+
+## Known Customizations We Maintain
+
+### IndexTTS-2
+- Audio normalization fix for Linux (v4.14.15) - **Not in upstream**
+- Custom processor wrapper for unified interface
+- Audio amplitude normalization to handle platform-specific torchaudio.load() behavior
+
+### ChatterBox (both versions)
+- Unified adapter pattern for consistent interface
+- Integration with unified node architecture
+
+### F5-TTS
+- Unified adapter pattern for consistent interface
+
+### Higgs Audio
+- Unified adapter pattern for consistent interface
+
+### RVC
+- Complete custom implementation for TTS Audio Suite
+
+### VibeVoice
+- Wrapper around external vibevoice package
+- ⚠️ **Status**: Microsoft disabled official repo in Sept 2025 - consider migration path
+
+---
+
+## Related Documentation
+
+- [NEW_ENGINE_IMPLEMENTATION_GUIDE.md](../NEW_ENGINE_IMPLEMENTATION_GUIDE.md) - How to integrate new engines
+- [INDEXTTS2_UPDATE_LOG.md](./INDEXTTS2_UPDATE_LOG.md) - IndexTTS-2 specific tracking
+- [../DEPENDENCY_MANAGEMENT_GUIDE.md](../DEPENDENCY_MANAGEMENT_GUIDE.md) - Dependency compatibility
+- [../BUMP_SCRIPT_INSTRUCTIONS.md](../BUMP_SCRIPT_INSTRUCTIONS.md) - Version management
+
+---
+
+## Source Code Location Details
+
+### Where to Find the Bundled Engine Source Code
+
+| Engine | Main Source Files | Config/Models | Processor/Adapter |
+|--------|-------------------|---------------|-------------------|
+| IndexTTS-2 | `engines/index_tts/indextts/` (infer.py, infer_v2.py, gpt/, s2mel/) | `engines/index_tts/indextts/` | `engines/processors/index_tts_processor.py` |
+| ChatterBox | `engines/chatterbox/chatterbox/` (models, inference) | `engines/chatterbox/models/` | `engines/adapters/chatterbox_adapter.py` |
+| ChatterBox Multilingual | `engines/chatterbox_official_23lang/` (tts.py, models/*, language_models.py) | `engines/chatterbox_official_23lang/models/` | `engines/processors/` |
+| F5-TTS | `engines/f5_tts/` (src/, model/, infer/) | `engines/f5_tts/configs/` | `engines/f5tts/f5tts.py` (adapter) |
+| Higgs Audio | `engines/higgs_audio/boson_multimodal/` (full framework) | `engines/higgs_audio/boson_multimodal/model/` | `engines/adapters/higgs_audio_adapter.py` |
+| RVC | `engines/rvc/impl/` (inference, models) | `engines/rvc/impl/lib/` | `engines/rvc/rvc_engine.py` |
+| VibeVoice | External package: `vibevoice` (PyPI) | Downloaded at runtime | `engines/vibevoice_engine/vibevoice_engine.py` |
+
+### Key File Patterns to Check When Updating
+
+**Inference/Generation Code:**
+- IndexTTS-2: `engines/index_tts/indextts/infer_v2.py`
+- ChatterBox: `engines/chatterbox/chatterbox/inference.py` or `tts.py`
+- F5-TTS: `engines/f5_tts/src/f5_tts/infer/`
+- Higgs Audio: `engines/higgs_audio/boson_multimodal/serve/serve_engine.py`
+- RVC: `engines/rvc/impl/infer_pack/`
+
+**Model Loading:**
+- IndexTTS-2: `engines/index_tts/indextts/infer_v2.py` (model initialization)
+- ChatterBox: `engines/chatterbox/chatterbox/models/`
+- F5-TTS: `engines/f5_tts/src/f5_tts/model/`
+- Higgs Audio: `engines/higgs_audio/boson_multimodal/model/higgs_audio.py`
+- RVC: `engines/rvc/impl/infer_pack/modules.py`
+
+**Audio Processing:**
+- IndexTTS-2: `engines/index_tts/indextts/infer_v2.py` (audio normalization, vocoding)
+- ChatterBox: Check audio post-processing in inference methods
+- F5-TTS: `engines/f5_tts/` audio handling
+- All: Look for clamp, normalize, scale operations
+
+---
+
+## Important Notes
+
+- **ChatterBox and ChatterBox Multilingual** both use the same official repository: `github.com/resemble-ai/chatterbox`
+  - The difference is in implementation: one focuses on single-language, the other on 23-language support
+  - They share the same upstream to track for updates
+
+- **VibeVoice** is NOT actively maintained - Microsoft disabled the official GitHub repository
+  - Consider evaluating alternatives or pinning to a specific version
+  - The wrapper depends on the external `vibevoice` PyPI package
+
+- **F5-TTS** vs **f5tts** directories:
+  - `f5_tts/` = Full bundled source code (official implementation from SWivid)
+  - `f5tts/` = Wrapper/adapter layer only (NOT the official source code)
+
+---
+
+## Next Steps
+
+1. ✅ Document all official upstream repositories
+2. ✅ Identify which code is bundled vs wrapped
+3. ✅ Implement `/check_tts_bundled_updates` slash command
+4. ⏳ Create UPDATE_LOG.md for each engine (in progress)
+5. ⏳ Establish regular checking schedule (weekly/monthly)
+6. ⏳ Create integration PR template for upstream changes

docs/VOCAL_REMOVAL_GUIDE.md

@@ -15,8 +15,10 @@ The **🤐 Noise or Vocal Removal** node is a powerful audio processing tool tha
 ### 🏆 BEST MODELS (2024-2025)
 
 **Top Tier - State of the Art:**
-- ★★★ **model_bs_roformer_ep_317_sdr_12.9755.ckpt** - BEST OVERALL (12.97 dB SDR, Transformer-based SOTA)
-- ★★★ **MDX23C-8KFFT-InstVoc_HQ.ckpt** - Highest quality, minimal artifacts (Karafan architecture)
+- ★★★ **MELBAND/denoise_mel_band_roformer_sdr_27.99.ckpt** - BEST DENOISING (27.99 dB SDR!) 
+- ★★★ **MELBAND/MelBandRoformer_fp32.safetensors** - Kijai's high-quality vocal separation
+- ★★★ **model_bs_roformer_ep_317_sdr_12.9755.ckpt** - Reliable transformer-based (12.97 dB SDR)
+- ★★☆ **MDX23C-8KFFT-InstVoc_HQ.ckpt** - Minimal artifacts (Karafan architecture)
 - ★★☆ **UVR-MDX-NET-vocal_FT.onnx** - Professional vocal extraction (MDX architecture)
 
 ### 📂 VR MODELS (Convolutional Neural Networks)
@@ -37,6 +39,18 @@ The **🤐 Noise or Vocal Removal** node is a powerful audio processing tool tha
 
 - **MDX23C-8KFFT-InstVoc_HQ.ckpt** - ⭐ HIGHEST QUALITY (8K FFT, minimal artifacts)
 
+### 🎯 MELBAND ROFORMER MODELS (State-of-the-Art 2025)
+
+**Kijai's Models (Vocal Separation):**
+- **MELBAND/MelBandRoformer_fp16.safetensors** - Fast processing (456MB, FP16 precision)
+- **MELBAND/MelBandRoformer_fp32.safetensors** - Maximum quality (913MB, FP32 precision)
+
+**ZFTurbo's Models (Denoising Specialists):**
+- **MELBAND/denoise_mel_band_roformer_sdr_27.99.ckpt** - ⭐ BEST DENOISING (27.99 dB SDR!)
+- **MELBAND/denoise_mel_band_roformer_aggressive_sdr_27.97.ckpt** - Aggressive denoising
+
+**Note:** MelBand models use advanced transformer architecture with rotary embeddings for superior separation quality
+
 ## Use Case Workflows
 
 ### 🎤 Karaoke Creation (Vocal Removal)
@@ -50,9 +64,10 @@ The **🤐 Noise or Vocal Removal** node is a powerful audio processing tool tha
 3. **Use:** "extracted voice/noise/echo" output (isolated vocals)
 
 ### 🔧 Audio Denoising
-1. **Primary:** `UVR-DeNoise.pth` (gentle: 5-8)
-2. **Use:** "remaining" output (clean audio)
-3. **Note:** "extracted" contains removed noise
+1. **Best Quality:** `MELBAND/denoise_mel_band_roformer_sdr_27.99.ckpt` (no aggressiveness setting)
+2. **Alternative:** `UVR-DeNoise.pth` (gentle: 5-8)
+3. **Use:** "remaining" output (clean audio)
+4. **Note:** "extracted" contains removed noise
 
 ### 💼 Professional Workflow
 1. **Separate:** `model_bs_roformer_ep_317_sdr_12.9755.ckpt`
@@ -71,7 +86,8 @@ The **🤐 Noise or Vocal Removal** node is a powerful audio processing tool tha
 - **VR (Vocal Remover):** Fast, magnitude-only processing, good for basic separation
 - **MDX (Multi-Dimensional eXtraction):** Hybrid spectrogram/waveform, better quality preservation  
 - **Karafan:** Multi-stage ensemble, state-of-the-art quality (highest CPU usage)
-- **RoFormer:** Transformer with rotary embeddings, current SOTA
+- **RoFormer:** Transformer with rotary embeddings, excellent quality
+- **MelBand RoFormer:** Advanced transformer with Mel-frequency bands, current SOTA (27+ dB SDR)
 
 ## Aggressiveness Settings (0-20)
 
@@ -122,12 +138,11 @@ The **🤐 Noise or Vocal Removal** node is a powerful audio processing tool tha
 
 ## Advanced Techniques
 
-### 🚀 NEWER MODELS (Consider adding)
-- **Mel-RoFormer** - Next-gen transformer architecture
-- **SCNet-XL** - Large-scale separation network
-- **VitLarge23** - Vision transformer adaptation
-- **Demucs v4 (htdemucs_ft)** - Hybrid transformer fine-tuned
-- **Kim Vocal models** - Highly regarded community favorites
+### 🚀 MODEL NOTES
+- **MelBand RoFormer** - ✅ NOW SUPPORTED! Best-in-class denoising (27+ dB SDR)
+- **SCNet models** - ⚠️ Architecture issues causing audio artifacts
+- **MDX23C models** - ⚠️ Tensor alignment errors in current implementation
+- **Note:** Aggressiveness setting only affects VR architecture models, not MelBand/MDX/Karafan
 
 ### 💡 PRO TIPS
 - **Use ensemble combinations** for best results

engines/adapters/chatterbox_adapter.py

@@ -283,21 +283,41 @@ class ChatterBoxEngineAdapter:
     
     def combine_audio_chunks(self, audio_segments: List[torch.Tensor], **params) -> torch.Tensor:
         """
-        Combine ChatterBox audio segments.
+        Combine ChatterBox audio segments using modular combination utility.
         
         Args:
             audio_segments: List of audio tensors to combine
             **params: Combination parameters
             
         Returns:
-            Combined audio tensor
+            Combined audio tensor, or tuple with timing info if requested
         """
         if len(audio_segments) == 1:
             return audio_segments[0]
         
-        # ChatterBox uses simple concatenation
-        from utils.audio.processing import AudioProcessingUtils
-        return AudioProcessingUtils.concatenate_audio_segments(audio_segments, "simple")
+        # Use modular chunk combiner with ChatterBox settings
+        from utils.audio.chunk_combiner import ChunkCombiner
+        
+        method = params.get("combination_method", "auto")
+        silence_ms = params.get("silence_ms", 100)
+        text_length = params.get("text_length", 0)
+        original_text = params.get("original_text", "")
+        text_chunks = params.get("text_chunks", None)
+        return_info = params.get("return_timing_info", False)
+        
+        print(f"🔗 Combining {len(audio_segments)} ChatterBox chunks using '{method}' method")
+        
+        return ChunkCombiner.combine_chunks(
+            audio_segments=audio_segments,
+            method=method,
+            silence_ms=silence_ms,
+            crossfade_duration=0.1,
+            sample_rate=44100,  # ChatterBox sample rate
+            text_length=text_length,
+            original_text=original_text,
+            text_chunks=text_chunks,
+            return_info=return_info
+        )
     
     def generate_audio(self, text: str, voice_preset: str, voice_settings: dict,
                       output_filename: str, language: str = "auto",

engines/adapters/chatterbox_official_23lang_adapter.py

@@ -0,0 +1,407 @@
+"""
+ChatterBox Official 23-Lang Engine Adapter - Engine-specific adapter for ChatterBox Official 23-Lang TTS
+Provides standardized interface for ChatterBox Official 23-Lang operations in multilingual engine
+With native batch processing support for parallel text generation
+"""
+
+import torch
+from typing import Dict, Any, Optional, List, Tuple
+import concurrent.futures
+import threading
+# Use absolute import to avoid relative import issues in ComfyUI
+import sys
+import os
+current_dir = os.path.dirname(__file__)
+parent_dir = os.path.dirname(current_dir)
+if parent_dir not in sys.path:
+    sys.path.insert(0, parent_dir)
+from utils.models.language_mapper import get_model_for_language
+
+
+class ChatterBoxOfficial23LangEngineAdapter:
+    """Engine-specific adapter for ChatterBox TTS with batch processing support."""
+    
+    def __init__(self, node_instance):
+        """
+        Initialize ChatterBox adapter with batch processing capabilities.
+        
+        Args:
+            node_instance: ChatterboxTTSNode or SRTTTSNode instance
+        """
+        self.node = node_instance
+        self.engine_type = "chatterbox_official_23lang"
+        self.batch_size = 4  # Default batch size for parallel processing
+        self.enable_batch_processing = True  # Enable batch processing by default
+        self._generation_lock = threading.Lock()  # Thread safety for model access
+    
+    def get_model_for_language(self, lang_code: str, default_model: str) -> str:
+        """
+        Get ChatterBox model name for specified language.
+        
+        Args:
+            lang_code: Language code (e.g., 'en', 'de', 'no')
+            default_model: Default model name (language)
+            
+        Returns:
+            ChatterBox model name (language) for the specified language code
+        """
+        return get_model_for_language(self.engine_type, lang_code, default_model)
+    
+    def load_base_model(self, language: str, device: str):
+        """
+        Load base ChatterBox model.
+        
+        Args:
+            language: Language model to load (e.g., "English", "German")
+            device: Device to load model on
+        """
+        self.node.load_tts_model(device, language)
+    
+    def load_language_model(self, language: str, device: str):
+        """
+        Load language-specific ChatterBox model.
+        
+        Args:
+            language: Language model to load (e.g., "German", "Norwegian")
+            device: Device to load model on
+        """
+        self.node.load_tts_model(device, language)
+    
+    def generate_segment_audio(self, text: str, char_audio: str, 
+                             character: str = "narrator", **params) -> torch.Tensor:
+        """
+        Generate ChatterBox audio for a text segment.
+        
+        Args:
+            text: Text to generate audio for
+            char_audio: Reference audio file path
+            character: Character name for caching
+            **params: Additional ChatterBox parameters
+            
+        Returns:
+            Generated audio tensor
+        """
+        # Extract ChatterBox Official 23-Lang specific parameters
+        exaggeration = params.get("exaggeration", 1.0)
+        temperature = params.get("temperature", 0.8)
+        cfg_weight = params.get("cfg_weight", 1.0)
+        repetition_penalty = params.get("repetition_penalty", 2.0)
+        min_p = params.get("min_p", 0.05)
+        top_p = params.get("top_p", 1.0)
+        language_id = params.get("language_id", "en")
+        seed = params.get("seed", 0)
+        enable_cache = params.get("enable_audio_cache", True)
+        
+        # Create cache function if caching is enabled
+        cache_fn = None
+        if enable_cache:
+            from utils.audio.cache import create_cache_function
+            
+            # Get current language/model for cache key
+            current_language = params.get("current_language", params.get("model", "English"))
+            audio_component = params.get("stable_audio_component", "main_reference")
+            if character != "narrator":
+                audio_component = f"char_file_{character}"
+            
+            # Get model source
+            model_source = params.get("model_source")
+            if not model_source and hasattr(self.node, 'model_manager'):
+                model_source = self.node.model_manager.get_model_source("tts")
+            
+            cache_fn = create_cache_function(
+                engine_type="chatterbox_official_23lang",
+                character=character,
+                exaggeration=exaggeration,
+                temperature=temperature,
+                cfg_weight=cfg_weight,
+                seed=seed,
+                audio_component=audio_component,
+                model_source=model_source or "unknown",
+                device=params.get("device", "auto"),
+                language=current_language
+            )
+        
+        # Handle caching externally for consistency with F5-TTS
+        if cache_fn:
+            # Check cache first
+            cached_audio = cache_fn(text)
+            if cached_audio is not None:
+                return cached_audio
+        
+        # CRITICAL FIX: Handle caching based on pause tag presence
+        # If pause tags detected, let internal pause processor handle caching of text segments
+        # If no pause tags, use external caching as normal
+        enable_pause_tags = params.get("enable_pause_tags", True)
+        has_pause_tags = enable_pause_tags and "[pause:" in text
+        
+        if has_pause_tags:
+            # Let internal pause processor handle caching of individual text segments
+            audio_result = self.node._generate_tts_with_pause_tags(
+                text=text,
+                audio_prompt=char_audio,
+                exaggeration=exaggeration,
+                temperature=temperature,
+                cfg_weight=cfg_weight,
+                language=params.get("current_language", params.get("model", "English")),
+                enable_pause_tags=True,
+                character=character,
+                seed=seed,
+                enable_cache=True,  # Enable internal caching for pause tag processing
+                crash_protection_template=params.get("crash_protection_template", "hmm ,, {seg} hmm ,,"),
+                stable_audio_component=params.get("stable_audio_component", "main_reference")
+            )
+            # Don't use external cache for pause tag segments
+        else:
+            # No pause tags, use external caching as normal
+            audio_result = self.node._generate_tts_with_pause_tags(
+                text=text,
+                audio_prompt=char_audio,
+                exaggeration=exaggeration,
+                temperature=temperature,
+                cfg_weight=cfg_weight,
+                language=params.get("current_language", params.get("model", "English")),
+                enable_pause_tags=False,
+                character=character,
+                seed=seed,
+                enable_cache=False,  # Disable internal caching since we handle it externally
+                crash_protection_template=params.get("crash_protection_template", "hmm ,, {seg} hmm ,,"),
+                stable_audio_component=params.get("stable_audio_component", "main_reference")
+            )
+            
+            # Cache the result if external caching is enabled
+            if cache_fn:
+                cache_fn(text, audio_result)
+        
+        return audio_result
+    
+    def generate_batch_audio(self, text_segments: List[str], char_audio: str,
+                            character: str = "narrator", batch_size: Optional[int] = None,
+                            **params) -> List[torch.Tensor]:
+        """
+        Generate ChatterBox audio for multiple text segments using true batch processing.
+        
+        Args:
+            text_segments: List of text segments to generate audio for
+            char_audio: Reference audio file path
+            character: Character name for caching
+            batch_size: Optional batch size override (uses self.batch_size if None)
+            **params: Additional ChatterBox parameters
+            
+        Returns:
+            List of generated audio tensors
+        """
+        if not self.enable_batch_processing or len(text_segments) == 1:
+            # Fall back to sequential processing if batch processing is disabled or single segment
+            return [self.generate_segment_audio(text, char_audio, character, **params)
+                   for text in text_segments]
+        
+        batch_size = batch_size or self.batch_size
+        print(f"🚀 ChatterBox batch processing: {len(text_segments)} segments in batches of {batch_size}")
+        
+        audio_results = []
+        
+        # Process segments in batches using the engine's generate_batch method
+        for i in range(0, len(text_segments), batch_size):
+            batch_texts = text_segments[i:i + batch_size]
+            print(f"🔄 Processing batch {i//batch_size + 1}: {len(batch_texts)} segments")
+            
+            try:
+                # Extract ChatterBox specific parameters
+                exaggeration = params.get("exaggeration", 1.0)
+                temperature = params.get("temperature", 0.8)
+                cfg_weight = params.get("cfg_weight", 1.0)
+                
+                # Use the ChatterBox engine's generate_batch method for true batch processing
+                if hasattr(self.node, 'tts_model') and hasattr(self.node.tts_model, 'generate_batch'):
+                    print(f"🔧 Using overlapping batch processing with max_workers={self.batch_size}")
+                    batch_audio = self.node.tts_model.generate_batch(
+                        texts=batch_texts,
+                        audio_prompt_path=char_audio,
+                        exaggeration=exaggeration,
+                        cfg_weight=cfg_weight,
+                        temperature=temperature,
+                        batch_size=len(batch_texts),
+                        max_workers=self.batch_size  # Use the UI batch_size as max_workers
+                    )
+                    
+                    print(f"✅ Batch {i//batch_size + 1} completed: {len(batch_audio)} audio segments generated")
+                    audio_results.extend(batch_audio)
+                    
+                else:
+                    # Fallback to sequential generation if batch method not available
+                    print(f"⚠️ Batch method not available, falling back to sequential processing")
+                    for text in batch_texts:
+                        audio = self.generate_segment_audio(text, char_audio, character, **params)
+                        audio_results.append(audio)
+                        
+            except Exception as e:
+                print(f"⚠️ Batch processing failed for batch {i//batch_size + 1}: {e}")
+                print(f"🔄 Falling back to sequential processing for this batch")
+                # Fall back to sequential generation for failed batch
+                for text in batch_texts:
+                    try:
+                        audio = self.generate_segment_audio(text, char_audio, character, **params)
+                        audio_results.append(audio)
+                    except Exception as fallback_error:
+                        print(f"❌ Sequential fallback also failed for '{text[:30]}...': {fallback_error}")
+                        # Generate silence as last resort
+                        from utils.audio.processing import AudioProcessingUtils
+                        silence = AudioProcessingUtils.create_silence(1.0, 44100)
+                        audio_results.append(silence)
+        
+        print(f"🎉 Batch processing complete: {len(audio_results)} total segments generated")
+        return audio_results
+    
+    def _generate_single_with_lock(self, text: str, char_audio: str,
+                                  character: str = "narrator", **params) -> torch.Tensor:
+        """
+        Generate audio for a single segment with thread-safe model access.
+        
+        Args:
+            text: Text to generate audio for
+            char_audio: Reference audio file path
+            character: Character name
+            **params: Additional parameters
+            
+        Returns:
+            Generated audio tensor
+        """
+        # Use lock to ensure thread-safe model access
+        with self._generation_lock:
+            return self.generate_segment_audio(text, char_audio, character, **params)
+    
+    def set_batch_processing(self, enabled: bool, batch_size: Optional[int] = None):
+        """
+        Configure batch processing settings.
+        
+        Args:
+            enabled: Whether to enable batch processing
+            batch_size: Optional new batch size (1-32 supported, uses overlapping parallel processing)
+        """
+        self.enable_batch_processing = enabled
+        if batch_size is not None:
+            self.batch_size = max(1, min(32, batch_size))  # Allow up to 32 parallel workers
+        
+        status = "enabled" if enabled else "disabled"
+        print(f"🔧 ChatterBox batch processing {status} (batch_size={self.batch_size})")
+    
+    def combine_audio_chunks(self, audio_segments: List[torch.Tensor], **params) -> torch.Tensor:
+        """
+        Combine ChatterBox audio segments using modular combination utility.
+        
+        Args:
+            audio_segments: List of audio tensors to combine
+            **params: Combination parameters
+            
+        Returns:
+            Combined audio tensor, or tuple with timing info if requested
+        """
+        if len(audio_segments) == 1:
+            return audio_segments[0]
+        
+        # Use modular chunk combiner with ChatterBox settings
+        from utils.audio.chunk_combiner import ChunkCombiner
+        
+        method = params.get("combination_method", "auto")
+        silence_ms = params.get("silence_ms", 100)
+        text_length = params.get("text_length", 0)
+        original_text = params.get("original_text", "")
+        text_chunks = params.get("text_chunks", None)
+        return_info = params.get("return_timing_info", False)
+        
+        print(f"🔗 Combining {len(audio_segments)} ChatterBox chunks using '{method}' method")
+        
+        return ChunkCombiner.combine_chunks(
+            audio_segments=audio_segments,
+            method=method,
+            silence_ms=silence_ms,
+            crossfade_duration=0.1,
+            sample_rate=44100,  # ChatterBox sample rate
+            text_length=text_length,
+            original_text=original_text,
+            text_chunks=text_chunks,
+            return_info=return_info
+        )
+    
+    def generate_audio(self, text: str, voice_preset: str, voice_settings: dict,
+                      output_filename: str, language: str = "auto",
+                      enable_batch_processing: bool = False, batch_size: int = 4) -> dict:
+        """
+        Generate audio using ChatterBox adapter with batch processing support.
+        
+        This method provides the interface expected by the unified TTS system.
+        """
+        # For single text generation, use the existing segment generation
+        if not enable_batch_processing:
+            print(f"🎤 ChatterBox single generation: '{text[:50]}...'")
+            
+            # Use the existing node functionality for single generation
+            audio_result = self.node.generate_speech(
+                text=text,
+                language=language,
+                device=self.node.device if hasattr(self.node, 'device') else 'auto',
+                exaggeration=voice_settings.get('exaggeration', 0.5),
+                temperature=voice_settings.get('temperature', 0.8),
+                cfg_weight=voice_settings.get('cfg_weight', 0.5),
+                seed=voice_settings.get('seed', 0),
+                reference_audio=voice_settings.get('reference_audio'),
+                audio_prompt_path=voice_settings.get('audio_prompt_path', ''),
+                batch_size=1 if not enable_batch_processing else batch_size
+            )
+            
+            return {
+                'audio': audio_result[0],  # Audio tensor
+                'info': audio_result[1] if len(audio_result) > 1 else 'Generated with ChatterBox'
+            }
+        else:
+            # For batch processing, split text into chunks and use batch generation
+            print(f"🚀 ChatterBox batch generation enabled: batch_size={batch_size}")
+            
+            # Split text into reasonable chunks for batch processing
+            from utils.text.chunking import ImprovedChatterBoxChunker
+            chunker = ImprovedChatterBoxChunker()
+            
+            # Use chunking to split text into segments
+            max_chars_per_chunk = voice_settings.get('max_chars_per_chunk', 400)
+            if len(text) > max_chars_per_chunk:
+                text_segments = chunker.split_into_chunks(text, max_chars_per_chunk)
+                print(f"📝 Text split into {len(text_segments)} chunks for batch processing")
+            else:
+                text_segments = [text]
+            
+            # Generate audio for all segments using batch processing
+            audio_segments = self.generate_batch_audio(
+                text_segments=text_segments,
+                char_audio=voice_settings.get('audio_prompt_path', ''),
+                character=voice_preset or 'narrator',
+                batch_size=batch_size,
+                exaggeration=voice_settings.get('exaggeration', 0.5),
+                temperature=voice_settings.get('temperature', 0.8),
+                cfg_weight=voice_settings.get('cfg_weight', 0.5),
+                seed=voice_settings.get('seed', 0),
+                current_language=language,
+                device=self.node.device if hasattr(self.node, 'device') else 'auto'
+            )
+            
+            # Combine audio segments
+            if len(audio_segments) == 1:
+                combined_audio = audio_segments[0]
+            else:
+                combined_audio = self.combine_audio_chunks(audio_segments)
+            
+            return {
+                'audio': combined_audio,
+                'info': f'Generated {len(text_segments)} chunks using ChatterBox batch processing (batch_size={batch_size})'
+            }
+
+    def _get_audio_duration(self, audio_tensor: torch.Tensor) -> float:
+        """Calculate audio duration in seconds."""
+        # ChatterBox uses 44.1kHz sample rate
+        if audio_tensor.dim() == 1:
+            num_samples = audio_tensor.shape[0]
+        elif audio_tensor.dim() == 2:
+            num_samples = audio_tensor.shape[1]
+        else:
+            num_samples = audio_tensor.numel()
+        
+        return num_samples / 44100  # ChatterBox sample rate

engines/adapters/chatterbox_streaming_adapter.py

@@ -103,6 +103,7 @@ class ChatterBoxStreamingAdapter(StreamingEngineAdapter):
             if has_streaming_method:
                 # Build inputs dict like the old system
                 inputs = {
+                    "character": segment.character,  # CRITICAL: Character is needed for caching
                     "exaggeration": kwargs.get("exaggeration", 0.5),
                     "temperature": kwargs.get("temperature", 0.8),
                     "cfg_weight": kwargs.get("cfg_weight", 0.5),
@@ -113,6 +114,15 @@ class ChatterBoxStreamingAdapter(StreamingEngineAdapter):
                     "device": kwargs.get("device", "auto"),
                     "reference_audio": kwargs.get("reference_audio", None)
                 }
+
+                # Apply segment-level parameters if available
+                segment_params = segment.metadata.get('parameters', {}) if segment.metadata else {}
+                if segment_params:
+                    # Import parameter application function
+                    from utils.text.segment_parameters import apply_segment_parameters
+                    segment_config = apply_segment_parameters(inputs, segment_params, "chatterbox_official_23lang")
+                    inputs.update(segment_config)
+                    print(f"  📊 Streaming: Applying segment parameters: {segment_params}")
                 
                 # Call exactly like the old working system
                 audio = self.node._process_single_segment_for_streaming(
@@ -295,35 +305,25 @@ class ChatterBoxStreamingAdapter(StreamingEngineAdapter):
         # Check if node has generation method
         if hasattr(self.node, '_generate_tts_with_pause_tags'):
             print(f"🏷️ ADAPTER: Calling _generate_tts_with_pause_tags for '{text[:30]}...'")
-            
-            # Temporarily replace stateless wrapper with underlying model for pause tag processing
-            original_model = getattr(self.node, 'tts_model', None)
-            try:
-                # Load the correct model for this language and extract underlying model if needed
-                self.load_model_for_language(language, kwargs.get('device', 'auto'))
-                
-                if hasattr(self.node, 'tts_model') and hasattr(self.node.tts_model, 'model'):
-                    self.node.tts_model = self.node.tts_model.model
-                    print(f"🔓 ADAPTER: Extracted underlying model for pause tag processing")
-                
-                # Generate stable audio component for cache consistency
-                from utils.audio.audio_hash import generate_stable_audio_component
-                stable_audio_component = generate_stable_audio_component(
-                    kwargs.get("reference_audio"), voice_path
-                )
-                
-                # Use pause tag-aware generation
-                return self.node._generate_tts_with_pause_tags(
-                    text, voice_path, exaggeration, temperature, cfg_weight,
-                    language, True, character=character, seed=seed,
-                    enable_cache=enable_cache,
-                    crash_protection_template=crash_protection,
-                    stable_audio_component=stable_audio_component
-                )
-            finally:
-                # Restore original model
-                if original_model is not None:
-                    self.node.tts_model = original_model
+
+            # Load the correct model for this language
+            self.load_model_for_language(language, kwargs.get('device', 'auto'))
+
+            # Generate stable audio component for cache consistency
+            from utils.audio.audio_hash import generate_stable_audio_component
+            stable_audio_component = generate_stable_audio_component(
+                kwargs.get("reference_audio"), voice_path
+            )
+
+            # Use pause tag-aware generation
+            # Note: tts_model is now a wrapper with __getattr__ forwarding, no unwrapping needed
+            return self.node._generate_tts_with_pause_tags(
+                text, voice_path, exaggeration, temperature, cfg_weight,
+                language, True, character=character, seed=seed,
+                enable_cache=enable_cache,
+                crash_protection_template=crash_protection,
+                stable_audio_component=stable_audio_component
+            )
         elif hasattr(self.node, 'tts_model') and self.node.tts_model:
             # Direct model generation
             return self.node.tts_model.generate(
@@ -352,12 +352,13 @@ class ChatterBoxStreamingAdapter(StreamingEngineAdapter):
         Returns:
             True if segment can be processed
         """
-        # Check language support
+        # Check language support - fallback to English for unsupported languages
         if segment.language not in self.supported_languages:
             # Check if it's a local model
             if not segment.language.startswith('local:'):
-                print(f"❌ ChatterBox doesn't support language: {segment.language}")
-                return False
+                print(f"⚠️ ChatterBox: Language '{segment.language}' not supported, falling back to English model")
+                # Modify segment language to English for fallback
+                segment.language = 'en'
         
         # Basic validation
         if not segment.text.strip():

engines/adapters/f5tts_adapter.py

@@ -152,14 +152,14 @@ class F5TTSEngineAdapter:
     
     def combine_audio_chunks(self, audio_segments: List[torch.Tensor], **params) -> torch.Tensor:
         """
-        Combine F5-TTS audio segments.
+        Combine F5-TTS audio segments using modular combination utility.
         
         Args:
             audio_segments: List of audio tensors to combine
             **params: Combination parameters
             
         Returns:
-            Combined audio tensor
+            Combined audio tensor, or tuple with timing info if requested
         """
         if len(audio_segments) == 1:
             return audio_segments[0]
@@ -167,9 +167,13 @@ class F5TTSEngineAdapter:
         method = params.get("combination_method", "auto")
         silence_ms = params.get("silence_ms", 100)
         text_length = params.get("text_length", 0)
+        original_text = params.get("original_text", "")
+        text_chunks = params.get("text_chunks", None)
+        return_info = params.get("return_timing_info", False)
         
         return self.node.combine_f5tts_audio_chunks(
-            audio_segments, method, silence_ms, text_length
+            audio_segments, method, silence_ms, text_length,
+            original_text=original_text, text_chunks=text_chunks, return_info=return_info
         )
     
     def _get_audio_duration(self, audio_tensor: torch.Tensor) -> float:

engines/adapters/f5tts_streaming_adapter.py

@@ -159,30 +159,18 @@ class F5TTSStreamingAdapter(StreamingEngineAdapter):
             if self._current_model == model_name and self._current_language == language:
                 return True
             
-            # Use universal smart model loader
-            from utils.models.smart_loader import smart_model_loader
-            
-            def f5tts_load_callback(device: str, model: str):
-                """Callback for F5-TTS model loading"""
-                if hasattr(self.node, 'load_f5tts_model'):
-                    # Use node's existing model loading (which now uses smart loader internally)
-                    return self.node.load_f5tts_model(model, device)
-                else:
-                    # Fallback: try to load directly
-                    from engines.f5tts.f5tts import F5TTSEngine
-                    engine = F5TTSEngine()
-                    engine.load_model(model, device)
-                    return engine
-            
-            # Use smart loader to get or load the model
-            model_instance, was_cached = smart_model_loader.load_model_if_needed(
-                engine_type="f5tts",
+            # Use unified interface for model loading
+            from utils.models.unified_model_interface import load_tts_model
+
+            # Load model via unified interface
+            model_instance = load_tts_model(
+                engine_name="f5tts",
                 model_name=model_name,
-                current_model=getattr(self.node, 'f5tts_model', None),
                 device=device,
-                load_callback=f5tts_load_callback
+                language=language,
+                force_reload=False
             )
-            
+
             # Update node's model reference and our tracking
             if hasattr(self.node, 'f5tts_model'):
                 self.node.f5tts_model = model_instance
@@ -190,16 +178,13 @@ class F5TTSStreamingAdapter(StreamingEngineAdapter):
                 self.node.f5tts = model_instance
             else:
                 self.node.f5tts = model_instance
-            
+
             # Cache the model for local reuse and update tracking
             self._loaded_models[model_name] = model_instance
             self._current_model = model_name
             self._current_language = language
-            
-            if was_cached:
-                print(f"♻️ F5TTS STREAMING: Reused {model_name} from smart loader")
-            else:
-                print(f"✅ F5TTS STREAMING: Loaded {model_name} via smart loader")
+
+            print(f"✅ F5TTS STREAMING: Loaded {model_name} via unified interface")
             
             return True
                 
@@ -312,8 +297,8 @@ class F5TTSStreamingAdapter(StreamingEngineAdapter):
                 reference_audio, reference_text = self.node._load_reference_audio(voice_path)
             else:
                 # Simple loading fallback
-                import torchaudio
-                reference_audio, _ = torchaudio.load(voice_path)
+                from utils.audio.processing import AudioProcessingUtils
+                reference_audio, _ = AudioProcessingUtils.safe_load_audio(voice_path)
                 
                 # Check for text file
                 text_path = voice_path.replace('.wav', '.txt').replace('.mp3', '.txt')

engines/adapters/higgs_audio_adapter.py

@@ -0,0 +1,442 @@
+"""
+Higgs Audio Engine Adapter - Engine-specific adapter for Higgs Audio 2
+Provides standardized interface for Higgs Audio operations in unified engine
+"""
+
+import torch
+from typing import Dict, Any, Optional, List, Tuple
+# Use absolute import to avoid relative import issues in ComfyUI
+import sys
+import os
+current_dir = os.path.dirname(__file__)
+parent_dir = os.path.dirname(current_dir)
+if parent_dir not in sys.path:
+    sys.path.insert(0, parent_dir)
+
+from utils.models.language_mapper import get_model_for_language
+from engines.higgs_audio.higgs_audio import HiggsAudioEngine
+from engines.higgs_audio.higgs_audio_downloader import HIGGS_AUDIO_MODELS
+
+
+class HiggsAudioEngineAdapter:
+    """Engine-specific adapter for Higgs Audio 2."""
+    
+    def __init__(self, node_instance):
+        """
+        Initialize Higgs Audio adapter.
+        
+        Args:
+            node_instance: HiggsAudioNode or HiggsAudioSRTNode instance
+        """
+        self.node = node_instance
+        self.engine_type = "higgs_audio"
+        self.higgs_engine = HiggsAudioEngine()
+    
+    def get_model_for_language(self, lang_code: str, default_model: str) -> str:
+        """
+        Get Higgs Audio model name for specified language.
+        
+        Args:
+            lang_code: Language code (e.g., 'en', 'zh')
+            default_model: Default model name
+            
+        Returns:
+            Higgs Audio model name for the language
+            
+        Note: Higgs Audio v2 supports multiple languages with the same model
+        """
+        # Higgs Audio v2 3B model supports multiple languages
+        # Map all language codes to the default model for now
+        supported_languages = ['en', 'zh', 'zh-cn']  # Add more as confirmed
+        
+        if lang_code.lower() in supported_languages:
+            return "higgs-audio-v2-3B"
+        else:
+            print(f"⚠️ Language '{lang_code}' not specifically tested with Higgs Audio, using default model")
+            return "higgs-audio-v2-3B"
+    
+    def load_base_model(self, model_name: str, device: str, enable_cuda_graphs: bool = True):
+        """
+        Load base Higgs Audio model.
+        
+        Args:
+            model_name: Model name to load
+            device: Device to load model on
+            enable_cuda_graphs: Whether to enable CUDA Graph optimization
+        """
+        # Check if the model is already loaded AND engine is initialized
+        # Also check for corruption flags that require recreation
+        current_model = getattr(self.node, 'current_model_name', None)
+        cuda_graph_corrupted = getattr(self.higgs_engine, '_cuda_graph_corrupted', False)
+        needs_recreation = getattr(self.higgs_engine, '_needs_recreation', False)
+        
+        if (current_model == model_name and
+            self.higgs_engine.engine is not None and
+            not cuda_graph_corrupted and
+            not needs_recreation):
+            print(f"💾 Higgs Audio adapter: Model '{model_name}' already loaded - ensuring device and clearing other models")
+            # Even if model is loaded, ensure it's on the right device and clear other models
+            from utils.models.comfyui_model_wrapper.model_manager import tts_model_manager
+            # This will clear other TTS models (like Step Audio EditX) from VRAM
+            tts_model_manager.ensure_device("higgs_audio", device)
+            return
+        
+        if cuda_graph_corrupted or needs_recreation:
+            print(f"🔥 Higgs Audio adapter: Engine corruption detected - forcing complete recreation")
+            # Clear the corruption flags
+            self.higgs_engine._cuda_graph_corrupted = False
+            self.higgs_engine._needs_recreation = False
+            # Force new engine creation by clearing current engine
+            self.higgs_engine.engine = None
+        
+        # Determine model paths based on model name
+        if model_name in HIGGS_AUDIO_MODELS:
+            model_config = HIGGS_AUDIO_MODELS[model_name]
+            generation_model = model_config["generation_repo"]
+            tokenizer_model = model_config["tokenizer_repo"]
+        else:
+            # Default fallback
+            generation_model = "bosonai/higgs-audio-v2-generation-3B-base"
+            tokenizer_model = "bosonai/higgs-audio-v2-tokenizer"
+        
+        # Initialize the Higgs Audio engine with CUDA Graph setting
+        self.higgs_engine.initialize_engine(
+            model_path=generation_model,
+            tokenizer_path=tokenizer_model,
+            device=device,
+            enable_cuda_graphs=enable_cuda_graphs
+        )
+        
+        # Store current model for caching
+        self.node.current_model_name = model_name
+        print(f"✅ Higgs Audio adapter: Loaded model '{model_name}' on {device}")
+    
+    def load_language_model(self, model_name: str, device: str):
+        """
+        Load language-specific Higgs Audio model.
+        
+        Args:
+            model_name: Language-specific model name
+            device: Device to load model on
+            
+        Note: Higgs Audio v2 uses the same model for all languages
+        """
+        # For Higgs Audio, language models are the same as base models
+        self.load_base_model(model_name, device)
+    
+    def generate_segment_audio(self, text: str, char_audio: str, char_text: str, 
+                             character: str = "narrator", **params) -> torch.Tensor:
+        """
+        Generate Higgs Audio audio for a text segment.
+        
+        Args:
+            text: Text to generate audio for
+            char_audio: Reference audio file path or audio dict
+            char_text: Reference text
+            character: Character name for caching
+            **params: Additional Higgs Audio parameters
+            
+        Returns:
+            Generated audio tensor
+        """
+        # Ensure engine is initialized with current model/device
+        model = params.get("model", "higgs-audio-v2-3B")
+        device = params.get("device", "auto")
+        enable_cuda_graphs = params.get("enable_cuda_graphs", True)  # Get CUDA Graph setting
+        self.load_base_model(model, device, enable_cuda_graphs)
+        
+        # Extract Higgs Audio specific parameters  
+        system_prompt = params.get("system_prompt", "Generate audio following instruction.")
+        temperature = params.get("temperature", 1.0)
+        top_p = params.get("top_p", 0.95)
+        top_k = params.get("top_k", 50)
+        max_new_tokens = params.get("max_new_tokens", 2048)
+        force_audio_gen = params.get("force_audio_gen", False)
+        ras_win_len = params.get("ras_win_len", 7)  # Default from boson_multimodal
+        ras_max_num_repeat = params.get("ras_max_num_repeat", 2)  # Default from boson_multimodal
+        enable_cuda_graphs = params.get("enable_cuda_graphs", True)  # CUDA Graph toggle
+        seed = params.get("seed", -1)
+        enable_cache = params.get("enable_audio_cache", True)
+        model_name = params.get("model", "higgs-audio-v2-3B")
+        device = params.get("device", "auto")
+        
+        # Native multi-speaker mode parameters
+        multi_speaker_mode = params.get("multi_speaker_mode", "Custom Character Switching")
+        second_narrator_audio = params.get("second_narrator_audio")
+        second_narrator_text = params.get("second_narrator_text", "")
+        
+        # Chunking parameters
+        max_chars_per_chunk = params.get("max_chars_per_chunk", 400)
+        silence_between_chunks_ms = params.get("silence_between_chunks_ms", 100)
+        enable_chunking = len(text) > max_chars_per_chunk  # Auto-enable chunking for long text
+        
+        # Convert chars to tokens for engine (Higgs Audio uses ~3.5 chars per token)
+        max_tokens_per_chunk = int(max_chars_per_chunk / 3.5)
+        
+        # Initialize engine if not already done
+        if not self.higgs_engine.engine:
+            print(f"🚀 Initializing Higgs Audio engine with model: {model_name}")
+            self.load_base_model(model_name, device)
+        
+        # Prepare reference audio
+        reference_audio = None
+        if char_audio:
+            if isinstance(char_audio, str) and os.path.exists(char_audio):
+                # Load audio file
+                try:
+                    from utils.audio.processing import AudioProcessingUtils
+                    waveform, sample_rate = AudioProcessingUtils.safe_load_audio(char_audio)
+                    if waveform.dim() == 2 and waveform.size(0) > 1:
+                        # Convert to mono if stereo
+                        waveform = torch.mean(waveform, dim=0, keepdim=True)
+                    
+                    # Move to same device as the model to prevent device mismatch
+                    from utils.device import resolve_torch_device
+                    target_device = resolve_torch_device(device)
+                    waveform = waveform.float().to(target_device)
+
+                    reference_audio = {"waveform": waveform, "sample_rate": sample_rate}
+                    print(f"  📁 Loaded reference audio on {target_device}")
+                except Exception as e:
+                    print(f"⚠️ Failed to load reference audio {char_audio}: {e}")
+            elif isinstance(char_audio, dict):
+                # Already in ComfyUI format - ensure tensors are on correct device
+                reference_audio = {}
+                from utils.device import resolve_torch_device
+                target_device = resolve_torch_device(device)
+                
+                for key, value in char_audio.items():
+                    if torch.is_tensor(value):
+                        reference_audio[key] = value.to(target_device)
+                        print(f"  📦 Moved ComfyUI {key} tensor to {target_device}")
+                    else:
+                        reference_audio[key] = value
+        
+        # Generate audio using Higgs Audio engine with stateless methods
+        try:
+            if multi_speaker_mode in ["Native Multi-Speaker (System Context)", "Native Multi-Speaker (Conversation)"]:
+                # Use stateless native multi-speaker mode with multiple reference audios
+                if hasattr(self.higgs_engine, 'generate_native_multispeaker_stateless'):
+                    audio_result, generation_info = self.higgs_engine.generate_native_multispeaker_stateless(
+                        text=text,
+                        primary_reference_audio=reference_audio,
+                        primary_reference_text=char_text or "",
+                        secondary_reference_audio=second_narrator_audio,
+                        secondary_reference_text=second_narrator_text,
+                        use_system_context=(multi_speaker_mode == "Native Multi-Speaker (System Context)"),
+                        system_prompt=system_prompt,
+                        max_new_tokens=max_new_tokens,
+                        temperature=temperature,
+                        top_p=top_p,
+                        top_k=top_k,
+                        force_audio_gen=force_audio_gen,
+                        ras_win_len=ras_win_len,
+                        ras_max_num_repeat=ras_max_num_repeat,
+                        enable_cache=enable_cache,
+                        character=character,
+                        seed=seed
+                    )
+                else:
+                    # Fallback to standard method if wrapper not available
+                    audio_result, generation_info = self.higgs_engine.generate_native_multispeaker(
+                        text=text,
+                        primary_reference_audio=reference_audio,
+                        primary_reference_text=char_text or "",
+                        secondary_reference_audio=second_narrator_audio,
+                        secondary_reference_text=second_narrator_text,
+                        use_system_context=(multi_speaker_mode == "Native Multi-Speaker (System Context)"),
+                        system_prompt=system_prompt,
+                        max_new_tokens=max_new_tokens,
+                        temperature=temperature,
+                        top_p=top_p,
+                        top_k=top_k,
+                        force_audio_gen=force_audio_gen,
+                        ras_win_len=ras_win_len,
+                        ras_max_num_repeat=ras_max_num_repeat,
+                        enable_cache=enable_cache,
+                        character=character,
+                        seed=seed
+                    )
+            else:
+                # Use stateless custom character switching mode (existing behavior)
+                if hasattr(self.higgs_engine, 'generate_stateless'):
+                    audio_result, generation_info = self.higgs_engine.generate_stateless(
+                        text=text,
+                        reference_audio=reference_audio,
+                        reference_text=char_text or "",
+                        system_prompt=system_prompt,
+                        max_new_tokens=max_new_tokens,
+                        temperature=temperature,
+                        top_p=top_p,
+                        top_k=top_k,
+                        force_audio_gen=force_audio_gen,
+                        ras_win_len=ras_win_len,
+                        ras_max_num_repeat=ras_max_num_repeat,
+                        enable_chunking=enable_chunking,
+                        max_tokens_per_chunk=max_tokens_per_chunk,  # Now properly converted from chars to tokens
+                        silence_between_chunks_ms=silence_between_chunks_ms,
+                        enable_cache=enable_cache,
+                        character=character,
+                        seed=seed
+                    )
+                else:
+                    # Fallback to standard method if wrapper not available
+                    audio_result, generation_info = self.higgs_engine.generate(
+                        text=text,
+                        reference_audio=reference_audio,
+                        reference_text=char_text or "",
+                        system_prompt=system_prompt,
+                        max_new_tokens=max_new_tokens,
+                        temperature=temperature,
+                        top_p=top_p,
+                        top_k=top_k,
+                        force_audio_gen=force_audio_gen,
+                        ras_win_len=ras_win_len,
+                        ras_max_num_repeat=ras_max_num_repeat,
+                        enable_chunking=enable_chunking,
+                        max_tokens_per_chunk=max_tokens_per_chunk,  # Now properly converted from chars to tokens
+                        silence_between_chunks_ms=silence_between_chunks_ms,
+                        enable_cache=enable_cache,
+                        character=character,
+                        seed=seed
+                    )
+            
+            # Return the waveform tensor with correct dimensions for ComfyUI
+            waveform = audio_result["waveform"]
+            
+            # Ensure correct dimensions: ComfyUI expects [channels, samples]
+            if waveform.dim() == 3:
+                # Handle [batch, channels, samples] -> [channels, samples]
+                if waveform.size(0) == 1:  # batch size of 1
+                    waveform = waveform.squeeze(0)  # Remove batch dimension
+                else:
+                    waveform = waveform[0]  # Take first item from batch
+            elif waveform.dim() == 1:
+                waveform = waveform.unsqueeze(0)  # Add channel dimension
+            
+            return waveform
+            
+        except Exception as e:
+            print(f"❌ Error generating Higgs Audio segment: {e}")
+            raise e
+    
+    def get_available_models(self) -> List[str]:
+        """
+        Get list of available Higgs Audio models.
+        
+        Returns:
+            List of model names
+        """
+        return self.higgs_engine.get_available_models()
+    
+    def get_model_info(self, model_name: str) -> Optional[Dict[str, Any]]:
+        """
+        Get information about a Higgs Audio model.
+        
+        Args:
+            model_name: Name of the model
+            
+        Returns:
+            Model information dict or None
+        """
+        return self.higgs_engine.downloader.get_model_info(model_name)
+    
+    def cleanup(self):
+        """Clean up resources"""
+        if hasattr(self, 'higgs_engine'):
+            self.higgs_engine.cleanup()
+        
+        # Clear current model reference
+        if hasattr(self.node, 'current_model_name'):
+            self.node.current_model_name = None
+    
+    def supports_streaming(self) -> bool:
+        """
+        Check if Higgs Audio supports streaming.
+        
+        Returns:
+            False for now - streaming not implemented yet
+        """
+        return False
+    
+    def get_sample_rate(self) -> int:
+        """
+        Get the sample rate used by Higgs Audio.
+        
+        Returns:
+            Sample rate (24000 Hz for Higgs Audio v2)
+        """
+        return 24000  # Higgs Audio v2 uses 24kHz sample rate
+    
+    def supports_voice_cloning(self) -> bool:
+        """
+        Check if Higgs Audio supports voice cloning.
+        
+        Returns:
+            True - Higgs Audio v2 supports voice cloning
+        """
+        return True
+    
+    
+    def validate_parameters(self, **params) -> Dict[str, Any]:
+        """
+        Validate and normalize Higgs Audio parameters.
+        
+        Args:
+            **params: Parameters to validate
+            
+        Returns:
+            Validated parameters dict
+        """
+        validated = {}
+        
+        # Temperature validation
+        temperature = params.get("temperature", 1.0)
+        validated["temperature"] = max(0.0, min(2.0, float(temperature)))
+        
+        # Top-p validation
+        top_p = params.get("top_p", 0.95)
+        validated["top_p"] = max(0.1, min(1.0, float(top_p)))
+        
+        # Top-k validation
+        top_k = params.get("top_k", 50)
+        validated["top_k"] = max(-1, min(100, int(top_k)))
+        
+        # Max tokens validation
+        max_tokens = params.get("max_new_tokens", 2048)
+        validated["max_new_tokens"] = max(1, min(4096, int(max_tokens)))
+        
+        # Force audio gen validation
+        force_audio_gen = params.get("force_audio_gen", False)
+        validated["force_audio_gen"] = bool(force_audio_gen)
+        
+        # RAS parameters validation
+        ras_win_len = params.get("ras_win_len", 7)
+        if ras_win_len is not None and ras_win_len > 0:
+            validated["ras_win_len"] = max(1, min(20, int(ras_win_len)))
+        else:
+            validated["ras_win_len"] = None  # Disable RAS
+        
+        ras_max_num_repeat = params.get("ras_max_num_repeat", 2)
+        validated["ras_max_num_repeat"] = max(1, min(5, int(ras_max_num_repeat)))
+        
+        # Audio priority validation
+        audio_priority = params.get("audio_priority", "auto")
+        valid_priorities = ["auto", "preset_dropdown", "reference_input", "force_preset"]
+        if audio_priority not in valid_priorities:
+            print(f"⚠️ Invalid audio priority '{audio_priority}', using 'auto'")
+            validated["audio_priority"] = "auto"
+        else:
+            validated["audio_priority"] = audio_priority
+        
+        # System prompt
+        validated["system_prompt"] = params.get("system_prompt", "Generate audio following instruction.")
+        
+        # Seed validation
+        seed = params.get("seed", -1)
+        validated["seed"] = int(seed) if seed >= 0 else -1
+        
+        # Cache settings
+        validated["enable_audio_cache"] = params.get("enable_audio_cache", True)
+        
+        return validated

engines/adapters/index_tts_adapter.py

@@ -0,0 +1,586 @@
+"""
+IndexTTS-2 Engine Adapter
+
+Provides standardized interface for IndexTTS-2 integration with TTS Audio Suite.
+Handles parameter mapping, character switching, and emotion control.
+"""
+
+import os
+import torch
+from typing import Dict, Any, Optional, List, Union
+
+from engines.index_tts.index_tts import IndexTTSEngine
+from engines.index_tts.index_tts_downloader import index_tts_downloader
+from utils.text.character_parser import character_parser
+from utils.voice.discovery import get_character_mapping, get_available_characters
+from utils.audio.cache import get_audio_cache
+
+
+class IndexTTSAdapter:
+    """
+    Adapter for IndexTTS-2 engine providing unified interface compatibility.
+    
+    Handles:
+    - Parameter mapping between unified interface and IndexTTS-2
+    - Character switching with [character:emotion_ref] syntax 
+    - Emotion control via audio references or vectors
+    - Caching integration
+    - Model management
+    """
+    
+    def __init__(self):
+        """Initialize the IndexTTS-2 adapter."""
+        self.engine = None
+        self.audio_cache = get_audio_cache()
+    
+    def initialize_engine(self,
+                         model_path: Optional[str] = None,
+                         device: str = "auto",
+                         use_fp16: bool = True,
+                         use_cuda_kernel: Optional[bool] = None,
+                         use_deepspeed: bool = False,
+                         use_torch_compile: bool = False,
+                         use_accel: bool = False):
+        """
+        Initialize IndexTTS-2 engine.
+
+        Args:
+            model_path: Path to model directory (auto-downloaded if None)
+            device: Target device
+            use_fp16: Use FP16 for inference
+            use_cuda_kernel: Use BigVGAN CUDA kernels
+            use_deepspeed: Use DeepSpeed optimization
+            use_torch_compile: Enable torch.compile optimization for S2Mel stage
+            use_accel: Enable GPT2 acceleration with FlashAttention
+        """
+        # Auto-download model if not provided or if "auto-download" is specified
+        if model_path is None or model_path == "auto-download":
+            if not index_tts_downloader.is_model_available():
+                print("📥 IndexTTS-2 model not found, downloading...")
+                model_path = index_tts_downloader.download_model()
+            else:
+                model_path = index_tts_downloader.get_model_path()
+
+        # Initialize engine
+        self.engine = IndexTTSEngine(
+            model_dir=model_path,
+            device=device,
+            use_fp16=use_fp16,
+            use_cuda_kernel=use_cuda_kernel,
+            use_deepspeed=use_deepspeed,
+            use_torch_compile=use_torch_compile,
+            use_accel=use_accel
+        )
+        
+    
+    def generate(self,
+                text: str,
+                speaker_audio: Optional[str] = None,
+                emotion_audio: Optional[str] = None,
+                emotion_alpha: float = 1.0,
+                emotion_vector: Optional[List[float]] = None,
+                use_emotion_text: bool = False,
+                emotion_text: Optional[str] = None,
+                use_random: bool = False,
+                interval_silence: int = 200,
+                max_text_tokens_per_segment: int = 120,
+                # Generation parameters
+                temperature: float = 0.8,
+                top_p: float = 0.8,
+                top_k: int = 30,
+                length_penalty: float = 0.0,
+                num_beams: int = 3,
+                repetition_penalty: float = 10.0,
+                max_mel_tokens: int = 1500,
+                # Streaming parameters
+                stream_return: bool = False,
+                more_segment_before: int = 0,
+                **kwargs) -> torch.Tensor:
+        """
+        Generate speech with IndexTTS-2.
+        
+        Args:
+            text: Text to synthesize (supports [character:emotion] tags)
+            speaker_audio: Speaker reference audio file path
+            emotion_audio: Emotion reference audio file path
+            emotion_alpha: Emotion blend factor (0.0-1.0)
+            emotion_vector: Manual emotion vector [happy, angry, sad, afraid, disgusted, melancholic, surprised, calm]
+            use_emotion_text: Extract emotions from text
+            emotion_text: Custom emotion description text
+            use_random: Enable random sampling
+            interval_silence: Silence between segments (ms)
+            max_text_tokens_per_segment: Max tokens per segment
+            temperature: Sampling temperature
+            top_p: Nucleus sampling parameter
+            top_k: Top-k sampling parameter
+            length_penalty: Length penalty for beam search
+            num_beams: Number of beams for beam search
+            repetition_penalty: Repetition penalty
+            max_mel_tokens: Maximum mel tokens to generate
+            **kwargs: Additional parameters
+            
+        Returns:
+            Generated audio tensor [1, samples] at 22050 Hz
+        """
+        if self.engine is None:
+            self.initialize_engine()
+
+        # Check if text actually contains character tags before processing
+        has_character_tags = character_parser.CHARACTER_TAG_PATTERN.search(text) is not None
+
+        if has_character_tags:
+            # Parse character switching tags with emotion support
+            processed_segments = self._process_character_tags_with_emotions(text)
+            
+            if len(processed_segments) > 1:
+                # Multi-segment character switching - process each segment separately
+                return self._generate_multi_character_segments(processed_segments, speaker_audio, emotion_audio, **kwargs)
+            elif processed_segments:
+                # Single character segment
+                first_segment = processed_segments[0]
+                processed_text = first_segment.get('text', '').strip()
+                character_name = first_segment.get('character')
+                emotion_ref = first_segment.get('emotion')
+            else:
+                processed_text = text
+                character_name = None
+                emotion_ref = None
+        else:
+            # No character tags - use text as is
+            processed_text = text
+            character_name = None
+            emotion_ref = None
+        
+        # Determine final speaker and emotion audio
+        final_speaker_audio = speaker_audio
+
+        # Handle Character Voices emotion_audio format
+        if emotion_audio and isinstance(emotion_audio, dict):
+            if "audio_path" in emotion_audio:
+                # Character Voices format: {'audio': {...}, 'audio_path': 'path', ...}
+                final_emotion_audio = emotion_audio["audio_path"]
+                print(f"🎭 Using Character Voices emotion audio: {emotion_audio.get('character_name', 'unknown')} -> {final_emotion_audio}")
+            elif "waveform" in emotion_audio:
+                # Direct AUDIO format: {'waveform': tensor, 'sample_rate': rate}
+                final_emotion_audio = emotion_audio
+            else:
+                final_emotion_audio = emotion_audio
+        else:
+            final_emotion_audio = emotion_audio
+        
+        # Only do character mapping if we actually have character tags
+        if has_character_tags:
+            # Collect all characters that might be needed
+            all_characters = []
+            if character_name:
+                all_characters.append(character_name)
+            if emotion_ref:
+                all_characters.append(emotion_ref)
+                
+            # Get character mapping for IndexTTS
+            if all_characters:
+                character_mapping = get_character_mapping(all_characters, engine_type="index_tts")
+            
+            if character_name and character_name in character_mapping:
+                # IndexTTS returns (audio_path, reference_text) tuples, we need audio_path
+                character_audio_path = character_mapping[character_name][0]
+                if character_audio_path is not None:
+                    final_speaker_audio = character_audio_path
+                    print(f"🎭 Using character voice: {character_name} -> {final_speaker_audio}")
+                else:
+                    print(f"⚠️ Character '{character_name}' mapping returned None, using original speaker audio: {final_speaker_audio}")
+            elif character_name:
+                print(f"⚠️ Character '{character_name}' not found in character mapping, using original speaker audio: {final_speaker_audio}")
+                        
+            if emotion_ref and emotion_ref in character_mapping:
+                # IndexTTS returns (audio_path, reference_text) tuples, we need audio_path
+                final_emotion_audio = character_mapping[emotion_ref][0]
+                print(f"😊 Using emotion reference: {emotion_ref} -> {final_emotion_audio}")
+                    
+        # For caching, we need stable identifiers for audio references
+        # Convert file paths to content hashes if they're temp files
+        cache_speaker_audio = self._get_stable_audio_identifier(final_speaker_audio)
+        cache_emotion_audio = self._get_stable_audio_identifier(final_emotion_audio) if final_emotion_audio else final_emotion_audio
+
+        # Check cache before generation
+        cache_key = self._generate_cache_key(
+            text=processed_text,
+            speaker_audio=cache_speaker_audio,
+            emotion_audio=cache_emotion_audio,
+            emotion_alpha=emotion_alpha,
+            emotion_vector=emotion_vector,
+            use_emotion_text=use_emotion_text,
+            emotion_text=emotion_text,
+            use_random=use_random,
+            temperature=temperature,
+            top_p=top_p,
+            top_k=top_k,
+            length_penalty=length_penalty,
+            repetition_penalty=repetition_penalty,
+            max_mel_tokens=max_mel_tokens,
+            max_text_tokens_per_segment=max_text_tokens_per_segment,
+            interval_silence=interval_silence,
+            stream_return=stream_return,
+            more_segment_before=more_segment_before,
+            **kwargs  # Include seed and other kwargs in cache key
+        )
+
+        # print(f"🐛 IndexTTS-2 cache key params: text='{processed_text[:30]}...', seed={kwargs.get('seed', 'missing')}, speaker='{final_speaker_audio}', emotion='{final_emotion_audio}'")
+        # print(f"🐛 IndexTTS-2 full cache key: {cache_key}")
+        # print(f"🐛 IndexTTS-2 cache stats: {self.audio_cache.get_cache_stats()}")
+
+        cached_audio = self.audio_cache.get_cached_audio(cache_key)
+        if cached_audio:
+            character_desc = character_name or 'narrator'
+            print(f"💾 Using cached IndexTTS-2 audio for '{character_desc}': '{processed_text[:30]}...'")
+            return cached_audio[0]
+        # else:
+        #     print(f"❌ IndexTTS-2 cache MISS for key: {cache_key}")
+        #     # Show first few existing keys for comparison
+        #     # Access the global cache properly
+        #     from utils.audio.cache import GLOBAL_AUDIO_CACHE
+        #     existing_keys = list(GLOBAL_AUDIO_CACHE.keys())
+        #     if existing_keys:
+        #         print(f"🔍 Existing cache keys (first 3): {existing_keys[:3]}")
+        #     else:
+        #         print(f"🔍 Cache is empty")
+            
+        # Handle seed for deterministic generation
+        seed = kwargs.get('seed', 0)
+        if seed != 0:
+            torch.manual_seed(seed)
+            import random
+            import numpy as np
+            random.seed(seed)
+            np.random.seed(seed)
+
+        # Filter out seed from kwargs (used for caching but not supported by IndexTTS engine)
+        engine_kwargs = {k: v for k, v in kwargs.items() if k != 'seed'}
+        # Add streaming parameters to engine kwargs
+        engine_kwargs['stream_return'] = stream_return
+        engine_kwargs['more_segment_before'] = more_segment_before
+
+        # Apply consistent emotion priority: emotion_audio takes precedence over other emotion controls
+        # This ensures consistent behavior whether using character tags or direct engine inputs
+        if final_emotion_audio:
+            # emotion_audio connected - disable other emotion controls
+            final_emotion_vector = None
+            final_use_emotion_text = False
+            final_emotion_text = None
+        else:
+            # No emotion_audio - use provided emotion controls
+            final_emotion_vector = emotion_vector
+            final_use_emotion_text = use_emotion_text
+            final_emotion_text = emotion_text
+
+        # Generate audio with OOM protection
+        try:
+            audio = self.engine.generate(
+                text=processed_text,
+                speaker_audio=final_speaker_audio,
+                emotion_audio=final_emotion_audio,
+                emotion_alpha=emotion_alpha,
+                emotion_vector=final_emotion_vector,
+                use_emotion_text=final_use_emotion_text,
+                emotion_text=final_emotion_text,
+                use_random=use_random,
+                interval_silence=interval_silence,
+                max_text_tokens_per_segment=max_text_tokens_per_segment,
+                temperature=temperature,
+                top_p=top_p,
+                top_k=top_k,
+                length_penalty=length_penalty,
+                num_beams=num_beams,
+                repetition_penalty=repetition_penalty,
+                max_mel_tokens=max_mel_tokens,
+                **engine_kwargs
+            )
+        except torch.OutOfMemoryError as e:
+            # Analyze audio after OOM to provide helpful feedback
+            audio_analysis = self._analyze_audio_after_oom(final_speaker_audio, final_emotion_audio, max_mel_tokens)
+            raise RuntimeError(
+                f"❌ IndexTTS-2 ran out of GPU memory during generation.\n{audio_analysis}"
+            ) from e
+
+        # Handle streaming vs non-streaming returns
+        if stream_return:
+            # In streaming mode, audio is a generator - return it as-is without caching
+            return audio
+        else:
+            # Cache the result for non-streaming mode
+            duration = audio.shape[-1] / 22050.0  # IndexTTS-2 uses 22050 Hz
+            self.audio_cache.cache_audio(cache_key, audio, duration)
+            # print(f"💾 IndexTTS-2 CACHED audio with key: {cache_key}")
+            return audio
+    
+    def _process_character_tags_with_emotions(self, text: str) -> List[Dict[str, Any]]:
+        """
+        Process character switching tags with emotion support.
+        """
+        from utils.text.character_parser.base_parser import CharacterParser
+        from utils.voice.discovery import get_available_characters, voice_discovery
+
+        # Create a temporary parser and set it up properly
+        temp_parser = CharacterParser()
+
+        # Get actual available characters and aliases
+        available_chars = get_available_characters()
+        character_aliases = voice_discovery.get_character_aliases()
+
+        # Build complete available set (like before but not hardcoded)
+        all_available = set()
+        if available_chars:
+            all_available.update(available_chars)
+        for alias, target in character_aliases.items():
+            all_available.add(alias.lower())
+            all_available.add(target.lower())
+
+        temp_parser.set_available_characters(list(all_available))
+
+        # Set language defaults
+        char_lang_defaults = voice_discovery.get_character_language_defaults()
+        for char, lang in char_lang_defaults.items():
+            temp_parser.set_character_language_default(char, lang)
+
+        # Use emotion-aware parsing
+        segments = temp_parser.split_by_character_with_emotions(text)
+
+        processed_segments = []
+        for character, segment_text, language, emotion in segments:
+            segment_info = {
+                'character': character,
+                'text': segment_text,
+                'language': language,
+                'emotion': emotion
+            }
+            processed_segments.append(segment_info)
+
+        return processed_segments
+    
+    def _generate_multi_character_segments(self, segments: List[Dict[str, Any]], 
+                                           default_speaker_audio: Optional[str], 
+                                           default_emotion_audio: Optional[str], 
+                                           **kwargs) -> torch.Tensor:
+        """
+        Generate audio for multiple character segments and combine them.
+        
+        Args:
+            segments: List of segment dictionaries with character, text, language, emotion info
+            default_speaker_audio: Default speaker reference audio
+            default_emotion_audio: Default emotion reference audio  
+            **kwargs: Additional generation parameters
+            
+        Returns:
+            Combined audio tensor [1, samples] at 22050 Hz
+        """
+        audio_segments = []
+        
+        # Get character mapping for all unique characters
+        unique_characters = set()
+        for segment in segments:
+            if segment.get('character'):
+                unique_characters.add(segment['character'])
+            if segment.get('emotion'):
+                unique_characters.add(segment['emotion'])
+        
+        # Get character mapping for IndexTTS
+        character_mapping = {}
+        if unique_characters:
+            character_mapping = get_character_mapping(list(unique_characters), engine_type="index_tts")
+        
+        print(f"🎭 IndexTTS-2: Processing {len(segments)} character segment(s) - {', '.join([s.get('character', 'narrator') for s in segments])}")
+        
+        for segment in segments:
+            character_name = segment.get('character', 'narrator')
+            segment_text = segment.get('text', '').strip()
+            emotion_ref = segment.get('emotion')
+            
+            if not segment_text:
+                continue
+                
+            # Determine speaker audio for this segment
+            speaker_audio = default_speaker_audio
+            if character_name and character_name in character_mapping:
+                character_audio_path = character_mapping[character_name][0]
+                if character_audio_path:
+                    speaker_audio = character_audio_path
+                    print(f"📖 Using character voice '{character_name}' | Ref: '{speaker_audio}'")
+                else:
+                    print(f"⚠️ Character '{character_name}' has no audio reference, using default")
+            
+            # Determine emotion audio for this segment
+            emotion_audio = default_emotion_audio  
+            if emotion_ref and emotion_ref in character_mapping:
+                emotion_audio_path = character_mapping[emotion_ref][0]
+                if emotion_audio_path:
+                    emotion_audio = emotion_audio_path
+                    print(f"😊 Using emotion reference '{emotion_ref}' | Ref: '{emotion_audio}'")
+            
+            # Generate cache key for this segment
+            segment_cache_key = self._generate_cache_key(
+                text=segment_text,
+                speaker_audio=speaker_audio,
+                emotion_audio=emotion_audio,
+                **kwargs
+            )
+
+            # Check cache first
+            cached_segment_audio = self.audio_cache.get_cached_audio(segment_cache_key)
+            if cached_segment_audio:
+                print(f"💾 Using cached IndexTTS-2 segment for '{character_name}': '{segment_text[:30]}...'")
+                segment_audio = cached_segment_audio[0]
+            else:
+                # Generate audio for this segment with OOM protection
+                try:
+                    segment_audio = self.engine.generate(
+                        text=segment_text,
+                        speaker_audio=speaker_audio,
+                        emotion_audio=emotion_audio,
+                        **kwargs
+                    )
+                except torch.OutOfMemoryError as e:
+                    # Analyze audio after OOM in multi-character segments
+                    audio_analysis = self._analyze_audio_after_oom(speaker_audio, emotion_audio, kwargs.get('max_mel_tokens', 1500))
+                    raise RuntimeError(
+                        f"❌ IndexTTS-2 OOM error in character segment '{character_name}'.\n{audio_analysis}"
+                    ) from e
+
+                # Cache the segment result
+                duration = segment_audio.shape[-1] / 22050.0  # IndexTTS-2 uses 22050 Hz
+                self.audio_cache.cache_audio(segment_cache_key, segment_audio, duration)
+                # print(f"💾 IndexTTS-2 CACHED segment for '{character_name}' with key: {segment_cache_key}")
+            
+            audio_segments.append(segment_audio)
+        
+        # Combine all audio segments
+        if audio_segments:
+            combined_audio = torch.cat(audio_segments, dim=-1)
+            return combined_audio
+        else:
+            # Return silence if no segments generated
+            return torch.zeros(1, 22050, dtype=torch.float32)
+    
+    def _generate_cache_key(self, **params) -> str:
+        """Generate cache key for IndexTTS-2."""
+        return self.audio_cache.generate_cache_key('index_tts', **params)
+
+    def _analyze_audio_after_oom(self, speaker_audio: str, emotion_audio: str, max_mel_tokens: int) -> str:
+        """
+        Analyze audio files after OOM to provide helpful feedback.
+        Uses fast header-based duration check to avoid loading large files.
+
+        Args:
+            speaker_audio: Path to speaker audio file
+            emotion_audio: Path to emotion audio file
+            max_mel_tokens: Current max_mel_tokens setting
+
+        Returns:
+            Formatted analysis message with recommendations
+        """
+        analysis_parts = []
+
+        # Analyze speaker audio
+        if speaker_audio and os.path.exists(speaker_audio):
+            try:
+                import soundfile as sf
+                info = sf.info(speaker_audio)
+                duration = info.duration
+
+                if duration > 60:
+                    analysis_parts.append(f"🔍 Speaker audio: {duration:.1f}s (TOO LONG - major OOM risk)")
+                    analysis_parts.append("   ✅ Solution: Use audio under 30 seconds")
+                elif duration > 30:
+                    analysis_parts.append(f"🔍 Speaker audio: {duration:.1f}s (long - may cause OOM)")
+                    analysis_parts.append("   ✅ Recommendation: Use shorter audio for stability")
+                else:
+                    analysis_parts.append(f"🔍 Speaker audio: {duration:.1f}s (length OK)")
+
+            except Exception:
+                analysis_parts.append("🔍 Speaker audio: Could not analyze duration")
+
+        # Analyze emotion audio
+        if emotion_audio and os.path.exists(emotion_audio):
+            try:
+                import soundfile as sf
+                info = sf.info(emotion_audio)
+                duration = info.duration
+
+                if duration > 60:
+                    analysis_parts.append(f"🔍 Emotion audio: {duration:.1f}s (TOO LONG - major OOM risk)")
+                elif duration > 30:
+                    analysis_parts.append(f"🔍 Emotion audio: {duration:.1f}s (long - may cause OOM)")
+                else:
+                    analysis_parts.append(f"🔍 Emotion audio: {duration:.1f}s (length OK)")
+
+            except Exception:
+                analysis_parts.append("🔍 Emotion audio: Could not analyze duration")
+
+        # Analyze max_mel_tokens setting
+        if max_mel_tokens > 1500:
+            analysis_parts.append(f"🔍 max_mel_tokens: {max_mel_tokens} (high - may cause OOM)")
+            analysis_parts.append("   ✅ Try reducing to 1000-1500")
+        else:
+            analysis_parts.append(f"🔍 max_mel_tokens: {max_mel_tokens} (setting OK)")
+
+        # Add general recommendations
+        analysis_parts.append("")
+        analysis_parts.append("💡 Quick fixes:")
+        analysis_parts.append("   • Use audio clips under 30 seconds")
+        analysis_parts.append("   • Reduce max_mel_tokens to 1000")
+        analysis_parts.append("   • Close other GPU applications")
+
+        return "\n".join(analysis_parts)
+
+    def _get_stable_audio_identifier(self, audio_path: str) -> str:
+        """
+        Get stable identifier for audio file using centralized audio hashing.
+        """
+        if not audio_path:
+            return audio_path
+
+        # Use our centralized audio hashing utility
+        from utils.audio.audio_hash import generate_stable_audio_component
+        return generate_stable_audio_component(audio_file_path=audio_path)
+    
+    def get_supported_formats(self) -> List[str]:
+        """Get supported audio formats."""
+        return ["wav", "mp3", "flac", "ogg"]
+    
+    def get_sample_rate(self) -> int:
+        """Get native sample rate."""
+        return 22050
+    
+    def get_emotion_labels(self) -> List[str]:
+        """Get supported emotion labels."""
+        return ["happy", "angry", "sad", "afraid", "disgusted", "melancholic", "surprised", "calm"]
+    
+    def create_emotion_vector(self, **emotions) -> List[float]:
+        """
+        Create emotion vector from keyword arguments.
+        
+        Args:
+            **emotions: Emotion intensities (e.g., happy=0.8, angry=0.2)
+            
+        Returns:
+            List of 8 emotion values
+        """
+        if self.engine:
+            return self.engine.create_emotion_vector(**emotions)
+        else:
+            # Fallback implementation
+            labels = self.get_emotion_labels()
+            vector = [0.0] * len(labels)
+            for i, label in enumerate(labels):
+                if label in emotions:
+                    vector[i] = max(0.0, min(1.2, float(emotions[label])))
+            return vector
+    
+    def unload(self):
+        """Unload the engine to free memory."""
+        if self.engine:
+            self.engine.unload()
+            self.engine = None
+    
+    def __del__(self):
+        """Cleanup on deletion."""
+        self.unload()

engines/adapters/rvc_adapter.py

@@ -160,7 +160,8 @@ class RVCEngineAdapter:
                     rms_mix_rate=rms_mix_rate,
                     resample_sr=resample_sr,
                     f0_autotune=f0_autotune,
-                    crepe_hop_length=crepe_hop_length
+                    crepe_hop_length=crepe_hop_length,
+                    **kwargs  # Pass through hubert_path and other engine parameters
                 )
                 
                 if result is None:
@@ -321,4 +322,8 @@ class RVCEngineAdapter:
         self._loaded_models.clear()
         if hasattr(self, 'rvc_engine'):
             self.rvc_engine.cleanup()
+
+        # CRITICAL: Clear minimal wrapper cache - this is where RVC models are actually stored
+        minimal_wrapper.clear_model_cache()
+
         print("RVC adapter cleanup completed")

engines/adapters/step_audio_editx_adapter.py

@@ -0,0 +1,439 @@
+"""
+Step Audio EditX Engine Adapter
+
+Provides standardized interface for Step Audio EditX integration with TTS Audio Suite.
+Handles parameter mapping, voice cloning, pause tag processing, and seed control.
+"""
+
+import os
+import sys
+import torch
+import tempfile
+import torchaudio
+from typing import Dict, Any, Optional, List, Union
+
+# Add project root to path
+current_dir = os.path.dirname(__file__)
+engines_dir = os.path.dirname(current_dir)
+project_root = os.path.dirname(engines_dir)
+if project_root not in sys.path:
+    sys.path.insert(0, project_root)
+
+from engines.step_audio_editx.step_audio_editx import StepAudioEditXEngine
+from utils.text.pause_processor import PauseTagProcessor
+from utils.audio.cache import get_audio_cache
+import folder_paths
+
+
+class StepAudioEditXEngineAdapter:
+    """
+    Adapter for Step Audio EditX engine providing unified interface compatibility.
+
+    Handles:
+    - Parameter mapping between unified interface and Step Audio EditX
+    - Voice cloning with prompt audio/text
+    - Pause tag processing and timing
+    - Seed control via global torch state
+    - Caching integration
+    - Model management
+    """
+
+    def __init__(self, node_instance):
+        """
+        Initialize the Step Audio EditX adapter.
+
+        Args:
+            node_instance: Parent node instance for context
+        """
+        self.node = node_instance
+        self.engine = None
+        self.audio_cache = get_audio_cache()
+
+        # Job-level timing tracker (persists across blocks)
+        self.job_tracker = None
+
+    def start_job(self, total_blocks: int, block_texts: list):
+        """
+        Initialize job tracker for time estimation across all blocks.
+
+        Args:
+            total_blocks: Number of blocks to process
+            block_texts: List of text lengths for each block (for weighted progress)
+        """
+        import time
+        total_text = sum(block_texts)
+        self.job_tracker = {
+            'start_time': time.time(),
+            'total_blocks': total_blocks,
+            'blocks_completed': 0,
+            'block_texts': block_texts,
+            'total_text': total_text,
+            'text_completed': 0,
+            'current_block_text': 0
+        }
+
+    def set_current_block(self, block_idx: int):
+        """Set the current block being processed."""
+        if self.job_tracker:
+            self.job_tracker['current_block_text'] = self.job_tracker['block_texts'][block_idx]
+
+    def complete_block(self):
+        """Mark current block as completed."""
+        if self.job_tracker:
+            self.job_tracker['blocks_completed'] += 1
+            self.job_tracker['text_completed'] += self.job_tracker['current_block_text']
+            # Track total tokens from completed blocks
+            self.job_tracker['total_tokens'] = self.job_tracker.get('total_tokens', 0) + self.job_tracker.get('current_block_tokens', 0)
+
+    def end_job(self):
+        """Clear job tracker after job completion."""
+        self.job_tracker = None
+
+    def load_base_model(self,
+                       model_path: str,
+                       device: str = "auto",
+                       torch_dtype: str = "bfloat16",
+                       quantization: Optional[str] = None):
+        """
+        Load Step Audio EditX engine via unified interface (with caching).
+
+        Args:
+            model_path: Model identifier (local:ModelName or ModelName for auto-download)
+            device: Target device (auto/cuda/cpu)
+            torch_dtype: Model precision (bfloat16/float16/float32/auto)
+            quantization: Quantization mode (int4/int8 or None)
+        """
+        from utils.models.unified_model_interface import unified_model_interface, ModelLoadConfig
+        from utils.device import resolve_torch_device
+
+        # Pass model_path directly - the downloader will handle "local:" resolution via factory
+        config = ModelLoadConfig(
+            engine_name="step_audio_editx",
+            model_type="tts",
+            model_name="Step-Audio-EditX",
+            model_path=model_path,  # Downloader will resolve this if it's "local:xxx"
+            device=resolve_torch_device(device),
+            additional_params={
+                "torch_dtype": torch_dtype,
+                "quantization": quantization
+            }
+        )
+
+        self.engine = unified_model_interface.load_model(config)
+
+    def generate_with_pause_tags(self,
+                                 text: str,
+                                 voice_ref: Optional[Dict[str, Any]],
+                                 params: Dict[str, Any],
+                                 process_pauses: bool = True,
+                                 character_name: Optional[str] = None) -> torch.Tensor:
+        """
+        Generate speech with pause tag processing (TTS Suite integration).
+
+        Args:
+            text: Input text (may contain <pause_X> tags)
+            voice_ref: Voice reference dict with 'prompt_audio_path' and 'prompt_text'
+            params: Generation parameters (including seed)
+            process_pauses: Whether to process pause tags
+            character_name: Character name for logging
+
+        Returns:
+            Generated audio tensor [1, samples] at 24000 Hz
+        """
+        if self.engine is None:
+            raise RuntimeError("Engine not loaded. Call load_base_model() first.")
+
+        # Extract seed from params
+        seed = params.get('seed', 0)
+
+        # Set global torch seed for reproducibility (Step Audio EditX uses global torch state)
+        if seed > 0:
+            torch.manual_seed(seed)
+            if torch.cuda.is_available():
+                torch.cuda.manual_seed_all(seed)
+
+        # Check if text has pause tags
+        if process_pauses and PauseTagProcessor.has_pause_tags(text):
+            # Process pause tags and generate segments
+            return self._generate_with_pauses(text, voice_ref, params, character_name)
+        else:
+            # Direct generation without pause processing
+            return self._generate_direct(text, voice_ref, params, character_name)
+
+    def _generate_direct(self,
+                        text: str,
+                        voice_ref: Optional[Dict[str, Any]],
+                        params: Dict[str, Any],
+                        character_name: Optional[str] = None) -> torch.Tensor:
+        """
+        Direct generation without pause processing.
+
+        Args:
+            text: Input text (clean, no pause tags)
+            voice_ref: Voice reference dict
+            params: Generation parameters
+            character_name: Character name for logging
+
+        Returns:
+            Audio tensor [1, samples]
+        """
+        # Get voice reference paths
+        prompt_audio_path, prompt_text = self._extract_voice_reference(voice_ref)
+
+        # Generate cache key and check cache
+        from utils.audio.audio_hash import generate_stable_audio_component
+        audio_component = generate_stable_audio_component(
+            audio_file_path=prompt_audio_path
+        ) if prompt_audio_path else "default_voice"
+
+        cache_key = self.audio_cache.generate_cache_key(
+            'step_audio_editx',
+            text=text,
+            audio_component=audio_component,
+            prompt_text=prompt_text,
+            temperature=params.get('temperature', 0.7),
+            do_sample=params.get('do_sample', True),
+            max_new_tokens=params.get('max_new_tokens', 8192),
+            seed=params.get('seed', 0),
+            model_path=params.get('model_path', 'Step-Audio-EditX'),
+            device=params.get('device', 'auto'),
+            torch_dtype=params.get('torch_dtype', 'bfloat16'),
+            quantization=params.get('quantization', None),
+            character=character_name or 'narrator'
+        )
+
+        # Check cache
+        cached_audio = self.audio_cache.get_cached_audio(cache_key)
+        if cached_audio:
+            char_desc = character_name or 'narrator'
+            print(f"💾 Using cached Step Audio EditX audio for '{char_desc}': '{text[:30]}...'")
+            return cached_audio[0]
+
+        # Create ComfyUI progress bar for generation tracking with time prediction
+        max_new_tokens = params.get('max_new_tokens', 8192)
+        progress_bar = None
+        try:
+            import comfy.utils
+            import time
+
+            # Get job tracker from adapter (for total job time estimation)
+            job_tracker = self.job_tracker
+
+            class TimedProgressBar:
+                """Progress bar wrapper with time tracking and prediction."""
+                def __init__(self, total, tracker):
+                    self.total = total
+                    self.current = 0
+                    self.start_time = time.time()
+                    self.wrapped = comfy.utils.ProgressBar(total)
+                    self.last_print = 0
+                    self.tracker = tracker
+
+                def update(self, delta=1):
+                    self.current += delta
+                    self.wrapped.update(delta)
+
+                def get_job_elapsed(self):
+                    """Get total job elapsed time in seconds."""
+                    if self.tracker:
+                        return time.time() - self.tracker['start_time']
+                    return None
+
+                def get_job_remaining_str(self):
+                    """Calculate and return job remaining time string based on actual data."""
+                    if not self.tracker or self.tracker['total_text'] <= 0:
+                        return None
+
+                    total_elapsed = time.time() - self.tracker['start_time']
+                    if total_elapsed <= 0:
+                        return None
+
+                    text_completed = self.tracker['text_completed']
+                    current_block_text = self.tracker['current_block_text']
+                    total_text = self.tracker['total_text']
+                    total_tokens_completed = self.tracker.get('total_tokens', 0)
+                    total_tokens = total_tokens_completed + self.current
+
+                    # Update current block tokens in tracker
+                    self.tracker['current_block_tokens'] = self.current
+
+                    if total_tokens <= 0:
+                        return None
+
+                    # Wait for more data before showing prediction
+                    # Need at least 1 completed chunk OR 150+ tokens in first chunk
+                    if total_tokens_completed == 0 and self.current < 150:
+                        return None
+
+                    # Estimate how much of current block is done based on tokens
+                    # Use completed blocks to estimate tokens per char
+                    if text_completed > 0 and total_tokens_completed > 0:
+                        tokens_per_char = total_tokens_completed / text_completed
+                    elif current_block_text > 0 and self.current > 0:
+                        # First block - use conservative estimate based on current data
+                        # Typical ratio is ~0.5-1.0 tokens per char
+                        tokens_per_char = 0.7
+                    else:
+                        tokens_per_char = 0.7
+
+                    # Estimate expected tokens for current block
+                    expected_current_tokens = current_block_text * tokens_per_char if tokens_per_char > 0 else self.current * 2
+
+                    # Current block progress (0 to 1)
+                    current_block_progress = min(self.current / expected_current_tokens, 0.99) if expected_current_tokens > 0 else 0.5
+
+                    # Text done = completed + portion of current block
+                    effective_text_done = text_completed + (current_block_text * current_block_progress)
+
+                    # Remaining text
+                    remaining_text = total_text - effective_text_done
+
+                    # Current speed (tokens per second)
+                    tokens_per_sec = total_tokens / total_elapsed
+
+                    # Remaining tokens estimate
+                    remaining_tokens = remaining_text * tokens_per_char if tokens_per_char > 0 else remaining_text
+
+                    # Remaining time
+                    if tokens_per_sec > 0 and remaining_tokens > 0:
+                        remaining_secs = remaining_tokens / tokens_per_sec
+
+                        if remaining_secs < 60:
+                            return f"~{remaining_secs:.0f}s left"
+                        else:
+                            return f"~{remaining_secs/60:.1f}m left"
+
+                    # If no remaining text but still processing, show nothing
+                    return None
+
+            progress_bar = TimedProgressBar(max_new_tokens, job_tracker)
+        except (ImportError, AttributeError):
+            pass  # ComfyUI progress not available
+
+        # Generate using clone mode
+        result = self.engine.clone(
+            prompt_wav_path=prompt_audio_path,
+            prompt_text=prompt_text,
+            target_text=text,
+            temperature=params.get('temperature', 0.7),
+            do_sample=params.get('do_sample', True),
+            max_new_tokens=max_new_tokens,
+            progress_bar=progress_bar
+        )
+
+        # Handle both wrapped (tensor) and unwrapped (tuple) returns
+        if isinstance(result, tuple):
+            audio_tensor, _ = result  # Unpack (audio, sample_rate) tuple
+        else:
+            audio_tensor = result
+
+        # Cache the result
+        duration = self.audio_cache._calculate_duration(audio_tensor, 'step_audio_editx')
+        self.audio_cache.cache_audio(cache_key, audio_tensor, duration)
+
+        return audio_tensor
+
+    def _generate_with_pauses(self,
+                             text: str,
+                             voice_ref: Optional[Dict[str, Any]],
+                             params: Dict[str, Any],
+                             character_name: Optional[str] = None) -> torch.Tensor:
+        """
+        Generate with pause tag processing.
+
+        NOTE: This returns a COMBINED audio tensor. Edit tags are NOT extracted here
+        because the processor needs to track individual segments for post-processing.
+        The processor should NOT call this for text with edit tags - it should
+        extract edit tags BEFORE pause processing.
+
+        Args:
+            text: Text with pause tags ([pause:2], [pause:1.5s], [pause:500ms])
+            voice_ref: Voice reference dict
+            params: Generation parameters
+            character_name: Character name for logging
+
+        Returns:
+            Combined audio tensor [1, samples]
+        """
+        # Parse pause tags - returns list of tuples: ('text', content) or ('pause', duration_seconds)
+        segments, clean_text = PauseTagProcessor.parse_pause_tags(text)
+
+        print(f"🎵 Processing {len(segments)} pause-delimited segments for '{character_name or 'unknown'}'")
+
+        audio_parts = []
+        sample_rate = 24000  # Step Audio EditX native sample rate
+
+        for segment_type, content in segments:
+            if segment_type == 'text':
+                # Generate audio for text segment
+                audio_tensor = self._generate_direct(content, voice_ref, params, character_name)
+
+                # Ensure correct shape [1, samples] (2D)
+                if audio_tensor.dim() == 1:
+                    audio_tensor = audio_tensor.unsqueeze(0)  # [samples] -> [1, samples]
+                elif audio_tensor.dim() == 3:
+                    audio_tensor = audio_tensor.squeeze(0)  # [1, 1, samples] -> [1, samples]
+
+                audio_parts.append(audio_tensor)
+
+            elif segment_type == 'pause':
+                # Create silence (content is duration in seconds)
+                silence = PauseTagProcessor.create_silence_segment(
+                    content, sample_rate
+                )
+                audio_parts.append(silence)
+
+        # Concatenate all parts
+        if not audio_parts:
+            return torch.zeros(1, 0)
+
+        combined_audio = torch.cat(audio_parts, dim=-1)
+        return combined_audio
+
+    def _extract_voice_reference(self, voice_ref: Optional[Dict[str, Any]]) -> tuple:
+        """
+        Extract voice reference audio path and text from voice_ref dict.
+
+        Args:
+            voice_ref: Voice reference dict (from voice discovery)
+
+        Returns:
+            Tuple of (prompt_audio_path, prompt_text)
+        """
+        if voice_ref is None or not isinstance(voice_ref, dict):
+            raise ValueError(
+                "Step Audio EditX requires voice reference. "
+                "Please provide voice_ref with 'prompt_audio_path' and 'prompt_text'"
+            )
+
+        # Extract paths
+        prompt_audio_path = voice_ref.get('prompt_audio_path') or voice_ref.get('audio_path')
+        prompt_text = voice_ref.get('prompt_text') or voice_ref.get('reference_text', '')
+
+        if not prompt_audio_path:
+            raise ValueError(
+                "Voice reference missing 'prompt_audio_path'. "
+                "Ensure your voice file is in voices/ directory and properly detected."
+            )
+
+        if not prompt_text or not prompt_text.strip():
+            raise ValueError(
+                f"Voice reference missing 'prompt_text' for {prompt_audio_path}. "
+                "Step Audio EditX requires transcription of reference audio. "
+                "Add prompt_text to your voice.json file."
+            )
+
+        return prompt_audio_path, prompt_text
+
+    def cleanup(self):
+        """
+        Clean up resources.
+
+        Note: For quantized models (int8/int4), we DON'T unload here because:
+        - Bitsandbytes models can't be moved to CPU and reloaded
+        - Auto-cleanup between generations would require full model reload
+        - Users should explicitly unload via the unload button when done
+        """
+        # Don't auto-unload - let the model stay in VRAM for reuse
+        # Only explicit unload (via button) should trigger model deletion
+        pass

engines/adapters/vibevoice_adapter.py

@@ -0,0 +1,833 @@
+"""
+VibeVoice Engine Adapter - Engine-specific adapter for VibeVoice
+Provides standardized interface for VibeVoice operations in unified engine
+"""
+
+import torch
+import re
+from typing import Dict, Any, Optional, List, Tuple
+import sys
+import os
+
+# Add parent directory for imports
+current_dir = os.path.dirname(__file__)
+parent_dir = os.path.dirname(current_dir)
+if parent_dir not in sys.path:
+    sys.path.insert(0, parent_dir)
+
+from utils.models.language_mapper import get_model_for_language
+from utils.text.pause_processor import PauseTagProcessor
+from utils.text.character_parser import CharacterParser
+from engines.vibevoice_engine.vibevoice_downloader import VIBEVOICE_MODELS
+from utils.models.manager import model_manager
+
+
+class VibeVoiceEngineAdapter:
+    """Engine-specific adapter for VibeVoice."""
+    
+    def __init__(self, node_instance, engine_config=None):
+        """
+        Initialize VibeVoice adapter.
+
+        Args:
+            node_instance: TTS node instance using this adapter
+            engine_config: Optional engine configuration dict
+        """
+        self.node = node_instance
+        self.engine_type = "vibevoice"
+        self.model_manager = model_manager
+        self.character_parser = CharacterParser()
+        self.pause_processor = PauseTagProcessor()
+
+        # Track character to speaker mapping for native multi-speaker mode
+        self._character_speaker_map = {}
+        self._speaker_voices = []
+
+        # Store model parameters from config
+        if engine_config:
+            self._model_name = engine_config.get('model', 'vibevoice-1.5B')
+            self._device = engine_config.get('device', 'auto')
+            self._attention_mode = engine_config.get('attention_mode', 'auto')
+            self._quantize_4bit = engine_config.get('quantize_llm_4bit', False)
+        else:
+            self._model_name = 'vibevoice-1.5B'
+            self._device = 'auto'
+            self._attention_mode = 'auto'
+            self._quantize_4bit = False
+
+    @property
+    def vibevoice_engine(self):
+        """Get VibeVoice engine, loading via unified interface if needed."""
+        from utils.models.unified_model_interface import load_tts_model
+
+        # Just call load - unified interface handles caching automatically
+        return load_tts_model(
+            engine_name="vibevoice",
+            model_name=self._model_name,
+            device=self._device,
+            attention_mode=self._attention_mode,
+            quantize_llm_4bit=self._quantize_4bit
+        )
+
+    @property
+    def current_model(self):
+        """Get current model from engine."""
+        return self.vibevoice_engine.model if self.vibevoice_engine else None
+
+    @property
+    def current_processor(self):
+        """Get current processor from engine."""
+        return self.vibevoice_engine.processor if self.vibevoice_engine else None
+
+    def get_model_for_language(self, lang_code: str, default_model: str) -> str:
+        """
+        Get VibeVoice model name for specified language.
+        
+        Args:
+            lang_code: Language code (e.g., 'en', 'zh')
+            default_model: Default model name
+            
+        Returns:
+            VibeVoice model name for the language
+            
+        Note: VibeVoice models support English and Chinese
+        """
+        # VibeVoice models support both English and Chinese
+        supported_languages = ['en', 'zh', 'zh-cn', 'chinese', 'english']
+        
+        if lang_code.lower() in supported_languages:
+            # Both models support EN/ZH, return the configured one
+            return default_model
+        else:
+            print(f"⚠️ VibeVoice: Language '{lang_code}' not officially supported (EN/ZH only)")
+            return default_model
+    
+    def load_base_model(self, model_name: str, device: str, attention_mode: str = "auto", quantize_llm_4bit: bool = False):
+        """
+        DEPRECATED: Model loading is now handled by unified TTS node.
+        This method is kept for backwards compatibility but does nothing.
+        The adapter now gets the engine dynamically from unified interface cache.
+
+        Args:
+            model_name: Model name (ignored, kept for compatibility)
+            device: Device (ignored, kept for compatibility)
+            attention_mode: Attention mode (ignored, kept for compatibility)
+            quantize_llm_4bit: Quantization (ignored, kept for compatibility)
+        """
+        # Store parameters for reference (some code might check these)
+        self._cached_model_name = model_name
+        self._cached_device = device
+        self._cached_attention_mode = attention_mode
+        self._cached_quantize_4bit = quantize_llm_4bit
+        # Engine is now accessed via @property vibevoice_engine which gets it from unified interface
+
+    def _parse_language_tags(self, text: str) -> Tuple[str, Optional[str]]:
+        """
+        Parse language tags like [de:Alice] from text.
+        
+        Args:
+            text: Text with potential language tags
+            
+        Returns:
+            Tuple of (processed_text, detected_language)
+        """
+        # Pattern to match [language:character] tags
+        lang_pattern = r'\[([a-zA-Z\-]+):([^\]]+)\]'
+        
+        detected_lang = None
+        processed_text = text
+        
+        # Find and process language tags
+        matches = re.findall(lang_pattern, text)
+        if matches:
+            # Take the first language found
+            lang_code, character = matches[0]
+            detected_lang = lang_code.lower()
+            
+            # Replace language tags with just character tags
+            for lang, char in matches:
+                processed_text = processed_text.replace(f'[{lang}:{char}]', f'[{char}]')
+            
+            # Warn about language since VibeVoice doesn't have language control
+            if detected_lang not in ['en', 'zh', 'chinese', 'english']:
+                print(f"⚠️ VibeVoice: Language tag '{detected_lang}' found but model only supports EN/ZH")
+        
+        return processed_text, detected_lang
+    
+    def _convert_character_to_speaker_format(self, text: str) -> Tuple[str, Dict[str, int]]:
+        """
+        Convert [Character] tags to Speaker N format for native multi-speaker.
+        Combines continuous character segments for better VibeVoice long-form generation.
+        
+        Args:
+            text: Text with [Character] tags
+            
+        Returns:
+            Tuple of (formatted_text, character_mapping)
+        """
+        # Parse character segments
+        segments = self.character_parser.parse_text(text)
+        
+        # Build character to speaker mapping and combine continuous segments
+        character_map = {}
+        character_blocks = {}  # Group continuous text by character
+        
+        for segment in segments:
+            char = segment.character
+            
+            # Assign speaker index if not already mapped
+            if char not in character_map:
+                speaker_idx = len(character_map)
+                if speaker_idx >= 4:
+                    print(f"⚠️ VibeVoice: More than 4 characters found, extra characters will use Speaker 3")
+                    speaker_idx = 3
+                character_map[char] = speaker_idx
+                character_blocks[char] = []
+            
+            # Accumulate text for this character (preserving paragraph structure)
+            character_blocks[char].append(segment.text.strip())
+        
+        # Create Speaker blocks by combining all text for each character
+        speaker_lines = []
+        for char in character_map.keys():
+            speaker_idx = character_map[char]
+            # Combine all text blocks for this character with double newlines for paragraph breaks
+            combined_text = '\n\n'.join(character_blocks[char])
+            speaker_lines.append(f"Speaker {speaker_idx}: {combined_text}")
+        
+        # Join with newlines for multi-speaker format
+        formatted_text = "\n".join(speaker_lines)
+        
+        return formatted_text, character_map
+    
+    def _combine_continuous_segments_for_vibevoice(self, segments: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
+        """
+        Combine continuous same-character segments for better VibeVoice long-form generation.
+        This is VibeVoice-specific and doesn't affect other engines.
+        
+        Args:
+            segments: Original segments from character parser
+            
+        Returns:
+            Combined segments where continuous same-character text is merged
+        """
+        if not segments:
+            return segments
+            
+        combined_segments = []
+        current_character = None
+        current_text_blocks = []
+        
+        for character, text in segments:
+            if character == current_character:
+                # Same character, accumulate text
+                current_text_blocks.append(text.strip())
+            else:
+                # Different character, finalize previous and start new
+                if current_character is not None:
+                    # Combine accumulated text blocks with double newlines for paragraph separation
+                    combined_text = '\n\n'.join(current_text_blocks)
+                    combined_segments.append((current_character, combined_text))
+                
+                # Start new character block
+                current_character = character
+                current_text_blocks = [text.strip()]
+        
+        # Don't forget the last character block
+        if current_character is not None:
+            combined_text = '\n\n'.join(current_text_blocks)
+            combined_segments.append((current_character, combined_text))
+        
+        return combined_segments
+    
+    def generate_segment_audio(self, text: str, char_audio: str, char_text: str, 
+                             character: str = "narrator", **params) -> torch.Tensor:
+        """
+        Generate VibeVoice audio for a text segment with caching support.
+        Follows the same pattern as other engines.
+        
+        Args:
+            text: Text to generate audio for
+            char_audio: Reference audio file path or audio dict
+            char_text: Reference text
+            character: Character name for caching
+            **params: Additional VibeVoice parameters
+            
+        Returns:
+            Generated audio tensor
+        """
+        # Extract parameters
+        seed = params.get("seed", 42)
+        enable_cache = params.get("enable_audio_cache", True)
+        model = params.get("model", "vibevoice-1.5B")
+        device = params.get("device", "auto")
+        attention_mode = params.get("attention_mode", "auto")
+        quantize_llm_4bit = params.get("quantize_llm_4bit", False)
+
+        # Initialize engine if not already done
+        self.load_base_model(model, device, attention_mode, quantize_llm_4bit)
+        
+        # Call engine with cache support
+        result = self.generate_segment(text, char_audio, {
+            **params,
+            'enable_cache': enable_cache,
+            'seed': seed
+        }, character=character)
+        
+        # Extract tensor from result
+        if isinstance(result, dict) and "waveform" in result:
+            audio_tensor = result["waveform"]
+            # Ensure proper dimensions for ComfyUI
+            if audio_tensor.dim() == 3:
+                audio_tensor = audio_tensor.squeeze(0)
+            if audio_tensor.dim() == 1:
+                audio_tensor = audio_tensor.unsqueeze(0)
+            return audio_tensor
+        
+        return result
+
+    def generate_segment(self, text: str, voice_ref: Optional[Dict], params: Dict, character: str = None) -> Dict:
+        """
+        Generate audio for a text segment.
+        
+        Args:
+            text: Text to generate
+            voice_ref: Voice reference audio
+            params: Generation parameters from engine config
+            character: Character name for cache isolation (required for proper caching)
+            
+        Returns:
+            Audio dict with waveform and sample_rate
+        """
+        # Extract parameters including new inference_steps
+        cfg_scale = params.get('cfg_scale', 1.3)
+        seed = params.get('seed', 42)
+        use_sampling = params.get('use_sampling', False)
+        temperature = params.get('temperature', 0.95)
+        top_p = params.get('top_p', 0.95)
+        inference_steps = params.get('inference_steps', 20)  # New parameter
+        max_new_tokens = params.get('max_new_tokens')
+        
+        # Check if model and processor are loaded
+        if self.current_model is None or self.current_processor is None:
+            raise RuntimeError("VibeVoice model not loaded. Call load_base_model() first.")
+        
+        # Prepare voice samples
+        voice_samples = self.vibevoice_engine._prepare_voice_samples([voice_ref])
+        
+        # For single segment, format as Speaker 1 (VibeVoice uses 1-based indexing)
+        formatted_text = f"Speaker 1: {text}"
+        
+        # Extract cache parameters
+        enable_cache = params.get('enable_cache', True)
+        # Use provided character or fall back to params, then default
+        if character is None:
+            character = params.get('character', 'narrator')
+        
+        # Generate stable audio component for cache consistency (like ChatterBox)
+        from utils.audio.audio_hash import generate_stable_audio_component
+        
+        # Use provided stable component, or generate from voice reference (like ChatterBox)
+        audio_component = params.get("stable_audio_component")
+        if not audio_component:
+            # Generate stable component like ChatterBox does
+            if voice_ref and isinstance(voice_ref, dict):
+                if 'waveform' in voice_ref:
+                    # Direct audio tensor format
+                    audio_component = generate_stable_audio_component(voice_ref, None)
+                elif 'audio_path' in voice_ref:
+                    # File path format  
+                    audio_component = generate_stable_audio_component(None, voice_ref['audio_path'])
+                else:
+                    audio_component = "main_reference"
+            else:
+                audio_component = "main_reference"
+        
+        # DEBUG: Print audio component to track cache invalidation
+        # print(f"🐛 VibeVoice DEBUG: character='{character}', audio_component='{audio_component[:50]}...'")
+        # Generate audio with inference_steps parameter
+        return self.vibevoice_engine.generate_speech(
+            text=formatted_text,
+            voice_samples=voice_samples,
+            cfg_scale=cfg_scale,
+            seed=seed,
+            use_sampling=use_sampling,
+            temperature=temperature,
+            top_p=top_p,
+            inference_steps=inference_steps,  # New parameter
+            max_new_tokens=max_new_tokens,
+            enable_cache=enable_cache,
+            character=character,
+            stable_audio_component=audio_component,
+            multi_speaker_mode=params.get('multi_speaker_mode', 'Custom Character Switching')
+        )
+    
+    def process_character_segments(self, segments: List[Tuple[str, str]], 
+                                  voice_mapping: Dict[str, Any],
+                                  params: Dict) -> List[Dict]:
+        """
+        Process multiple character segments.
+        
+        Args:
+            segments: List of (character, text) tuples
+            voice_mapping: Dict mapping character names to voice references
+            params: Generation parameters
+            
+        Returns:
+            List of audio dicts
+        """
+        audio_segments = []
+        
+        # VibeVoice-specific: Combine continuous same-character segments for better long-form generation
+        # This doesn't affect other engines since it's only applied here in VibeVoiceAdapter
+        combined_segments = self._combine_continuous_segments_for_vibevoice(segments)
+        
+        print(f"🔄 VibeVoice: Combined {len(segments)} segments into {len(combined_segments)} continuous blocks")
+        for i, (char, text) in enumerate(combined_segments):
+            text_preview = text.replace('\n\n', ' ¶ ')[:100] + "..." if len(text) > 100 else text.replace('\n\n', ' ¶ ')
+            print(f"   Block {i+1}: {char} -> {text_preview}")
+        
+        # Check if we should use native multi-speaker mode
+        multi_speaker_mode = params.get('multi_speaker_mode', 'Custom Character Switching')
+        
+        if multi_speaker_mode == "Native Multi-Speaker" and len(combined_segments) <= 4:
+            # Use native multi-speaker generation with combined segments
+            audio = self._generate_native_multispeaker(combined_segments, voice_mapping, params, None)
+            audio_segments.append(audio)
+        else:
+            # Custom Character Switching mode with VibeVoice-style grouped generation
+            # Group consecutive same-character segments and generate each group at once
+            audio_segments = self._generate_custom_character_switching_grouped(combined_segments, voice_mapping, params)
+        
+        return audio_segments
+    
+    def _generate_custom_character_switching_grouped(self, segments: List[Tuple[str, str]], 
+                                                   voice_mapping: Dict[str, Any],
+                                                   params: Dict) -> List[Dict]:
+        """
+        Generate audio for Custom Character Switching mode using VibeVoice-style grouped generation.
+        Groups consecutive same-character segments and generates each group at once like official VibeVoice.
+        
+        Args:
+            segments: List of (character, text) tuples  
+            voice_mapping: Dict mapping character names to voice references
+            params: Generation parameters
+            
+        Returns:
+            List of audio dicts in order
+        """
+        audio_segments = []
+        
+        if not segments:
+            return audio_segments
+        
+        # Group consecutive same-character segments for batch generation
+        character_groups = []
+        current_character = None
+        current_group = []
+        
+        for character, text in segments:
+            if character == current_character:
+                # Same character, add to current group
+                current_group.append(text)
+            else:
+                # Different character, finalize previous group and start new
+                if current_character is not None and current_group:
+                    character_groups.append((current_character, current_group))
+                
+                current_character = character
+                current_group = [text]
+        
+        # Don't forget the last group
+        if current_character is not None and current_group:
+            character_groups.append((current_character, current_group))
+        
+        print(f"🎭 Custom Character Switching: Processing {len(character_groups)} character groups")
+        
+        # Generate each character group using VibeVoice format
+        for group_idx, (character, text_list) in enumerate(character_groups):
+            print(f"🎤 Group {group_idx + 1}: Character '{character}' with {len(text_list)} segments")
+            
+            # Format as Speaker 1 entries (VibeVoice style) and combine
+            formatted_lines = []
+            for text in text_list:
+                formatted_lines.append(f"Speaker 1: {text.strip()}")
+            
+            # Combine all segments for this character into one script
+            combined_script = '\n'.join(formatted_lines)
+            
+            print(f"🎭 CUSTOM CHARACTER GROUP - Formatted text for VibeVoice:")
+            print("="*60)
+            print(combined_script)
+            print("="*60)
+            
+            # Handle pause tags across the entire combined script
+            if self.pause_processor.has_pause_tags(combined_script):
+                pause_segments, clean_text = self.pause_processor.parse_pause_tags(combined_script)
+                
+                for seg_type, content in pause_segments:
+                    if seg_type == 'text':
+                        voice_ref = voice_mapping.get(character)
+                        audio = self.generate_segment(content, voice_ref, params, character=character)
+                        audio_segments.append(audio)
+                    elif seg_type == 'pause':
+                        # Create silence segment
+                        silence = self.pause_processor.create_silence_segment(
+                            content, 24000, 
+                            device=torch.device('cpu'),
+                            dtype=torch.float32
+                        )
+                        audio_segments.append({
+                            "waveform": silence.unsqueeze(0),
+                            "sample_rate": 24000
+                        })
+            else:
+                # Generate the entire character group at once
+                voice_ref = voice_mapping.get(character)
+                audio = self.generate_segment(combined_script, voice_ref, params, character=character)
+                audio_segments.append(audio)
+        
+        return audio_segments
+    
+    def _generate_native_multispeaker(self, segments: List[Tuple[str, str]], 
+                                     voice_mapping: Dict[str, Any],
+                                     params: Dict,
+                                     global_char_to_speaker: Optional[Dict[str, int]] = None) -> Dict:
+        """
+        Generate using VibeVoice's native multi-speaker mode.
+        Supports both [Character] tags and manual "Speaker N:" format.
+        
+        Args:
+            segments: List of (character, text) tuples
+            voice_mapping: Dict mapping character names to voice references
+            params: Generation parameters
+            global_char_to_speaker: Global character-to-speaker mapping for consistent SRT processing
+            
+        Returns:
+            Combined audio dict
+        """
+        # Get speaker voice inputs from engine config for priority system
+        # For manual Speaker format, the main narrator voice comes from the TTS Text node
+        # We need to get it from a different source since voice_mapping may not have 'narrator' key
+        main_narrator_voice = voice_mapping.get('narrator')  # Try narrator first
+        if main_narrator_voice is None:
+            # If no 'narrator' key, get it from any available voice (fallback for manual Speaker format)
+            available_voices = [v for v in voice_mapping.values() if v is not None]
+            main_narrator_voice = available_voices[0] if available_voices else None
+            # print(f"🐛 Debug: No 'narrator' key, using fallback voice: {'✅ found' if main_narrator_voice else '❌ none available'}")
+        speaker_inputs = {
+            1: main_narrator_voice,  # Speaker 1 uses main narrator from TTS Text
+            2: params.get('speaker2_voice'),
+            3: params.get('speaker3_voice'), 
+            4: params.get('speaker4_voice')
+        }
+        
+        # print(f"🐛 Debug: speaker_inputs[1] (main narrator): {'✅ has voice' if speaker_inputs[1] else '❌ no voice'}")
+        
+        # Build speaker mapping and format text
+        character_map = {}
+        speaker_voices = []
+        formatted_lines = []
+        
+        print(f"🎭 Native multi-speaker: Processing {len(segments)} segments with characters: {[char for char, _ in segments]}")
+        print(f"🎤 Speaker inputs connected: {[f'Speaker {k}' for k, v in speaker_inputs.items() if v is not None]}")
+        
+        for character, text in segments:
+            # Check if this is already a manual "Speaker N:" format
+            manual_speaker = self._detect_manual_speaker_format(character)
+            
+            if manual_speaker is not None:
+                # Manual "Speaker N:" format - use speaker input directly
+                speaker_idx = manual_speaker - 1  # Convert to 0-based
+                if speaker_idx >= 4:
+                    speaker_idx = 3
+                    
+                # Get the appropriate voice based on speaker number
+                voice = speaker_inputs.get(manual_speaker)
+                if manual_speaker == 1:
+                    print(f"🎤 Manual format 'Speaker {manual_speaker}' -> using {'✅ main narrator (Tony)' if voice else '❌ no narrator, using default'}")
+                else:
+                    print(f"🎤 Manual format 'Speaker {manual_speaker}' -> using {'✅ connected input' if voice else '❌ no input, using default'}")
+                
+                # Ensure we have enough speaker_voices slots
+                while len(speaker_voices) <= speaker_idx:
+                    speaker_voices.append(None)
+                speaker_voices[speaker_idx] = voice
+                
+                formatted_lines.append(f"Speaker {manual_speaker}: {text.strip()}")
+                
+            else:
+                # Character tag format - use global mapping if provided (for SRT consistency)
+                if character not in character_map:
+                    # Special handling for numeric characters: [1] [2] [3] [4] -> map directly to Speaker N
+                    if character.isdigit() and 1 <= int(character) <= 4:
+                        speaker_idx = int(character) - 1  # Convert [1] to Speaker 1 (0-based index)
+                        character_map[character] = speaker_idx
+                        print(f"🔢 Numeric character '[{character}]' -> Speaker {int(character)} (direct mapping)")
+                    elif global_char_to_speaker and character in global_char_to_speaker:
+                        # Use global mapping for consistent SRT processing
+                        speaker_idx = global_char_to_speaker[character] - 1  # Convert to 0-based
+                        character_map[character] = speaker_idx
+                    else:
+                        # Fallback to sequential assignment
+                        speaker_idx = len(character_map)
+                        if speaker_idx >= 4:
+                            print(f"⚠️ VibeVoice: Limiting to 4 speakers, '{character}' will use Speaker 4")
+                            speaker_idx = 3  # Use 0-based internally, will convert to 1-based for format
+                        else:
+                            character_map[character] = speaker_idx
+                        
+                    # Priority system: speaker inputs override character aliases
+                    speaker_num = speaker_idx + 1
+                    connected_voice = speaker_inputs.get(speaker_num)
+                    character_voice = voice_mapping.get(character)
+                    
+                    if connected_voice is not None and character_voice is not None:
+                        print(f"⚠️ Priority: Speaker {speaker_num} input overrides ['{character}'] alias - using connected voice")
+                        voice = connected_voice
+                    elif connected_voice is not None:
+                        print(f"🎤 Speaker {speaker_num}: Using connected voice input")
+                        voice = connected_voice
+                    else:
+                        print(f"🎭 Character '{character}' -> Speaker {speaker_num}, using character voice")
+                        voice = character_voice
+                    
+                    # Ensure we have enough speaker_voices slots
+                    while len(speaker_voices) <= speaker_idx:
+                        speaker_voices.append(None)
+                    speaker_voices[speaker_idx] = voice
+                
+                speaker_idx = character_map.get(character, 3)
+                # Use 1-based Speaker format as per VibeVoice spec (Speaker 1:, Speaker 2:, etc.)
+                formatted_lines.append(f"Speaker {speaker_idx + 1}: {text.strip()}")
+        
+        # Join with newlines for multi-speaker format
+        formatted_text = "\n".join(formatted_lines)
+        print(f"🎭 NATIVE MULTI-SPEAKER - Complete formatted text for VibeVoice:")
+        print("="*60)
+        print(formatted_text)
+        print("="*60)
+        print(f"🎤 Using {len(speaker_voices)} voice samples for generation")
+        
+        # Validate and normalize voice references
+        normalized_voices = []
+        for i, voice in enumerate(speaker_voices):
+            if voice is None:
+                # print(f"🐛 Speaker {i+1} voice: None")
+                normalized_voices.append(None)
+            elif isinstance(voice, dict) and ('audio_path' in voice or 'waveform' in voice):
+                # Valid voice reference format
+                # if 'audio_path' in voice:
+                #     print(f"🐛 Speaker {i+1} voice: {voice['audio_path']}")
+                # else:
+                #     print(f"🐛 Speaker {i+1} voice: waveform tensor shape {voice['waveform'].shape}")
+                normalized_voices.append(voice)
+            elif isinstance(voice, str):
+                # Invalid format - just character name, try to convert to voice reference
+                print(f"❌ Speaker {i+1} received character name '{voice}' instead of voice reference")
+                print(f"💡 TIP: Connect Character Voices output (not character_name) to Speaker {i+1} input")
+                # Try to find matching voice file
+                try:
+                    from utils.voice.discovery import get_character_mapping
+                    char_mapping = get_character_mapping()
+                    if voice in char_mapping:
+                        voice_ref = char_mapping[voice]
+                        print(f"🔄 Auto-converted '{voice}' to voice reference")
+                        normalized_voices.append(voice_ref)
+                    else:
+                        print(f"⚠️ Unknown character '{voice}', using None")
+                        normalized_voices.append(None)
+                except Exception as e:
+                    print(f"⚠️ Could not convert character '{voice}': {e}")
+                    normalized_voices.append(None)
+            else:
+                print(f"❌ Speaker {i+1} invalid voice format: {type(voice)} - {str(voice)[:100]}...")
+                normalized_voices.append(None)
+        
+        # Use normalized voices
+        speaker_voices = normalized_voices
+        
+        # Check if model and processor are loaded
+        if self.current_model is None or self.current_processor is None:
+            raise RuntimeError("VibeVoice model not loaded. Call load_base_model() first.")
+        
+        # Prepare voice samples
+        voice_samples = self.vibevoice_engine._prepare_voice_samples(speaker_voices)
+        
+        # Generate stable audio component for multi-speaker (like ChatterBox)
+        from utils.audio.audio_hash import generate_stable_audio_component
+        
+        # For multi-speaker, use combined hash of all voices
+        combined_voice_hash = []
+        for voice in speaker_voices[:4]:  # Max 4 speakers
+            if voice is not None and isinstance(voice, dict):
+                if 'waveform' in voice:
+                    # Direct audio tensor format
+                    combined_voice_hash.append(generate_stable_audio_component(voice, None))
+                elif 'audio_path' in voice:
+                    # File path format
+                    combined_voice_hash.append(generate_stable_audio_component(None, voice['audio_path']))
+                else:
+                    combined_voice_hash.append("unknown_voice")
+            else:
+                combined_voice_hash.append("no_voice")
+        audio_component = f"multi_speaker_{'_'.join(combined_voice_hash)}"
+        
+        # Generate with multi-speaker text and inference_steps
+        return self.vibevoice_engine.generate_speech(
+            text=formatted_text,
+            voice_samples=voice_samples,
+            cfg_scale=params.get('cfg_scale', 1.3),
+            seed=params.get('seed', 42),
+            use_sampling=params.get('use_sampling', False),
+            temperature=params.get('temperature', 0.95),
+            top_p=params.get('top_p', 0.95),
+            inference_steps=params.get('inference_steps', 20),  # New parameter
+            max_new_tokens=params.get('max_new_tokens'),
+            enable_cache=params.get('enable_cache', True),
+            character="multi_speaker",
+            stable_audio_component=audio_component,
+            multi_speaker_mode=params.get('multi_speaker_mode', 'Native Multi-Speaker')
+        )
+    
+    def handle_pause_tags(self, text: str) -> Tuple[str, Optional[List]]:
+        """
+        Handle pause tags in text.
+        
+        Args:
+            text: Text potentially containing pause tags
+            
+        Returns:
+            Tuple of (processed_text, pause_segments)
+        """
+        if self.pause_processor.has_pause_tags(text):
+            segments, clean_text = self.pause_processor.parse_pause_tags(text)
+            return clean_text, segments
+        return text, None
+    
+    def generate_vibevoice_with_pause_tags(self, text: str, voice_ref: Optional[Dict], params: Dict,
+                                         enable_pause_tags: bool = True, character: str = "narrator"):
+        """
+        Generate VibeVoice audio with pause tag support.
+
+        Args:
+            text: Input text potentially with pause tags
+            voice_ref: Voice reference dict
+            params: Generation parameters
+            enable_pause_tags: Whether to process pause tags
+            character: Character name for logging
+
+        Returns:
+            If pause tags present: List of segment dicts with audio, text, edit_tags
+            If no pause tags: Single audio tensor (backwards compatible)
+        """
+        from utils.text.pause_processor import PauseTagProcessor
+        from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment
+
+        if not enable_pause_tags or not PauseTagProcessor.has_pause_tags(text):
+            # No pause tags, but still need to check for edit tags
+            clean_text, edit_tags = get_edit_tags_for_segment(text)
+
+            # If there are edit tags, return segment dict format for post-processing
+            if edit_tags:
+                result = self.generate_segment(clean_text, voice_ref, params, character)
+                waveform = result['waveform']
+                # Ensure proper tensor format
+                if waveform.dim() == 3:
+                    waveform = waveform.squeeze(0)
+                if waveform.dim() == 1:
+                    waveform = waveform.unsqueeze(0)
+
+                return [{
+                    'waveform': waveform,
+                    'sample_rate': 24000,
+                    'character': character,
+                    'text': clean_text,
+                    'original_text': text,
+                    'edit_tags': edit_tags
+                }]
+            else:
+                # No pause tags AND no edit tags - backwards compatible tensor return
+                result = self.generate_segment(text, voice_ref, params, character)
+                waveform = result['waveform']
+                # Ensure proper tensor format
+                if waveform.dim() == 3:
+                    waveform = waveform.squeeze(0)  # Remove batch dim
+                if waveform.dim() == 1:
+                    waveform = waveform.unsqueeze(0)  # Add channel dim
+                return waveform
+
+        print(f"🎵 VibeVoice: Processing pause tags in text")
+
+        # Process pause tags - returns list of ('text', content) or ('pause', duration)
+        pause_segments, _ = PauseTagProcessor.parse_pause_tags(text)
+
+        # Generate segment dicts for each pause-separated piece
+        segment_dicts = []
+
+        for seg_type, seg_content in pause_segments:
+            if seg_type == 'text':
+                # Extract edit tags from this text segment
+                clean_text, edit_tags = get_edit_tags_for_segment(seg_content)
+
+                # Generate TTS for this text piece
+                result = self.generate_segment(clean_text, voice_ref, params, character)
+                waveform = result['waveform']
+
+                # Ensure proper tensor format
+                if waveform.dim() == 3:
+                    waveform = waveform.squeeze(0)  # Remove batch dim
+                if waveform.dim() == 1:
+                    waveform = waveform.unsqueeze(0)  # Add channel dim
+
+                segment_dicts.append({
+                    'waveform': waveform,
+                    'sample_rate': 24000,
+                    'character': character,
+                    'text': clean_text,
+                    'original_text': seg_content,
+                    'edit_tags': edit_tags
+                })
+
+            else:  # seg_type == 'pause'
+                # Create silence segment
+                duration_seconds = seg_content
+                silence = PauseTagProcessor.create_silence_segment(
+                    duration_seconds, 24000, device=torch.device('cpu')
+                )
+
+                segment_dicts.append({
+                    'waveform': silence,
+                    'sample_rate': 24000,
+                    'character': character,
+                    'text': '',
+                    'original_text': f'[pause:{duration_seconds}]',
+                    'edit_tags': []
+                })
+
+        return segment_dicts
+    
+    def _detect_manual_speaker_format(self, character: str) -> Optional[int]:
+        """
+        Detect if character is already in manual "Speaker N:" format.
+        
+        Args:
+            character: Character name to check
+            
+        Returns:
+            Speaker number (1-4) if manual format, None otherwise
+        """
+        import re
+        # Match "Speaker N" (case insensitive, with optional trailing colon/whitespace)
+        match = re.match(r'^speaker\s*(\d+)\s*:?\s*$', character.strip(), re.IGNORECASE)
+        if match:
+            speaker_num = int(match.group(1))
+            if 1 <= speaker_num <= 4:
+                return speaker_num
+        return None
+    
+    def cleanup(self):
+        """Clean up resources - models are now managed by ModelManager"""
+        self._character_speaker_map.clear()
+        self._speaker_voices.clear()
+        # Don't clean up vibevoice_engine - it just holds references to cached models

engines/chatterbox/__init__.py

@@ -58,7 +58,9 @@ except ImportError:
 try:
     from .language_models import (
         get_chatterbox_models, get_model_config, get_model_files_for_language,
-        find_local_model_path, detect_model_format, get_available_languages
+        find_local_model_path, detect_model_format, get_available_languages,
+        is_model_incomplete, get_model_requirements, validate_model_completeness,
+        get_tokenizer_filename
     )
     LANGUAGE_MODELS_AVAILABLE = True
 except ImportError:
@@ -76,6 +78,14 @@ except ImportError:
         return ("pt", "ResembleAI/chatterbox")
     def detect_model_format(model_path):
         return "pt"
+    def is_model_incomplete(language):
+        return False
+    def get_model_requirements(language):
+        return []
+    def validate_model_completeness(model_path, language):
+        return True, []
+    def get_tokenizer_filename(language):
+        return "tokenizer.json"
 
 # SRT subtitle support modules - import independently
 try:
@@ -90,6 +100,8 @@ try:
         'ChatterboxTTS', 'ChatterboxVC', 'ChatterBoxF5TTS',
         'get_chatterbox_models', 'get_model_config', 'get_model_files_for_language',
         'find_local_model_path', 'detect_model_format', 'get_available_languages',
+        'is_model_incomplete', 'get_model_requirements', 'validate_model_completeness',
+        'get_tokenizer_filename',
         'SRTParser', 'SRTSubtitle', 'SRTParseError', 'validate_srt_timing_compatibility',
         'AudioTimingUtils', 'PhaseVocoderTimeStretcher', 'TimedAudioAssembler',
         'calculate_timing_adjustments', 'AudioTimingError'
@@ -100,5 +112,7 @@ except ImportError:
     __all__ = [
         'ChatterboxTTS', 'ChatterboxVC', 'ChatterBoxF5TTS',
         'get_chatterbox_models', 'get_model_config', 'get_model_files_for_language',
-        'find_local_model_path', 'detect_model_format', 'get_available_languages'
+        'find_local_model_path', 'detect_model_format', 'get_available_languages',
+        'is_model_incomplete', 'get_model_requirements', 'validate_model_completeness',
+        'get_tokenizer_filename'
     ]

engines/chatterbox/audio_timing.py

@@ -476,10 +476,10 @@ class TimedAudioAssembler:
             else:
                 raise AudioTimingError(f"Invalid stretcher_type: {stretcher_type}. Use 'ffmpeg' or 'phase_vocoder'")
     
-    def assemble_timed_audio(self, audio_segments: List[torch.Tensor], 
+    def assemble_timed_audio(self, audio_segments: List[torch.Tensor],
                            target_timings: List[Tuple[float, float]],
                            total_duration: Optional[float] = None,
-                           fade_duration: float = 0.01) -> torch.Tensor:
+                           fade_duration: float = 0.01) -> Tuple[torch.Tensor, str]:
         """
         Assemble audio segments with precise timing
         
@@ -581,8 +581,16 @@ class TimedAudioAssembler:
             # Place segment in output buffer with crossfading for overlaps
             self._place_segment_with_fade(output, audio_segment, start_sample, end_sample, fade_duration)
         
-        return output
-    
+        return output, self.stretch_method_used or "none"
+
+    def get_stretch_method_used(self) -> str:
+        """Get which stretching method was used in the last assembly"""
+        return self.stretch_method_used or "none"
+
+    def get_time_stretcher(self):
+        """Get the time stretcher object used (for backward compatibility)"""
+        return self.time_stretcher
+
     def _place_segment_with_fade(self, output: torch.Tensor, segment: torch.Tensor,
                                start_sample: int, end_sample: int, fade_duration: float):
         """
@@ -607,13 +615,28 @@ class TimedAudioAssembler:
             fade_in = torch.linspace(0, 1, fade_samples, device=segment.device)
             fade_out = torch.linspace(1, 0, fade_samples, device=segment.device)
             
-            # Fade in the beginning of new segment
-            if segment.dim() == 1:
-                segment[:fade_samples] *= fade_in
+            # Fade in the beginning of new segment and fade out existing content
+            if output.dim() == 1:
+                # 1D output (mono)
+                if segment.dim() == 1:
+                    segment[:fade_samples] *= fade_in
+                else:
+                    # Convert 2D segment to 1D for compatibility
+                    segment = torch.mean(segment, dim=0)
+                    segment[:fade_samples] *= fade_in
+                
                 # Fade out existing content at the beginning
                 output[start_sample:start_sample + fade_samples] *= fade_out
             else:
-                segment[:, :fade_samples] *= fade_in.unsqueeze(0)
+                # 2D output (stereo/multi-channel)
+                if segment.dim() == 1:
+                    # Convert 1D segment to 2D to match output
+                    segment = segment.unsqueeze(0).repeat(output.size(0), 1)
+                    segment[:, :fade_samples] *= fade_in.unsqueeze(0)
+                else:
+                    segment[:, :fade_samples] *= fade_in.unsqueeze(0)
+                
+                # Fade out existing content at the beginning
                 output[:, start_sample:start_sample + fade_samples] *= fade_out.unsqueeze(0)
             
             # Fade out the end of new segment if there's content after
@@ -622,16 +645,33 @@ class TimedAudioAssembler:
             
             if output.dim() == 1:
                 if end_check_start < end_check_end and torch.any(output[end_check_start:end_check_end] != 0):
-                    segment[-fade_samples:] *= fade_out
+                    if segment.dim() == 1:
+                        segment[-fade_samples:] *= fade_out
+                    else:
+                        segment[:, -fade_samples:] *= fade_out.unsqueeze(0)
             else:
                 if end_check_start < end_check_end and torch.any(output[:, end_check_start:end_check_end] != 0):
-                    segment[:, -fade_samples:] *= fade_out.unsqueeze(0)
+                    if segment.dim() == 1:
+                        # segment was already converted to 2D above, so this shouldn't happen
+                        pass
+                    else:
+                        segment[:, -fade_samples:] *= fade_out.unsqueeze(0)
         
-        # Add segment to output
+        # Add segment to output (segment dimensions should now match output)
         if output.dim() == 1:
-            output[start_sample:end_sample] += segment
+            if segment.dim() == 1:
+                output[start_sample:end_sample] += segment
+            else:
+                # This shouldn't happen after normalization above, but handle it
+                segment = torch.mean(segment, dim=0)
+                output[start_sample:end_sample] += segment
         else:
-            output[:, start_sample:end_sample] += segment
+            if segment.dim() == 2:
+                output[:, start_sample:end_sample] += segment
+            else:
+                # This shouldn't happen after normalization above, but handle it
+                segment = segment.unsqueeze(0).repeat(output.size(0), 1)
+                output[:, start_sample:end_sample] += segment
 
 
 def calculate_timing_adjustments(natural_durations: List[float], 

engines/chatterbox/gpu_aware_processor.py

@@ -38,9 +38,8 @@ class GPUAwareBatchProcessor:
     
     def _resolve_device(self, device: str) -> str:
         """Resolve device string to actual device."""
-        if device == "auto":
-            return "cuda" if torch.cuda.is_available() else "cpu"
-        return device
+        from utils.device import resolve_torch_device
+        return resolve_torch_device(device)
     
     def _determine_optimal_workers(self) -> int:
         """

engines/chatterbox/language_models.py

@@ -19,11 +19,67 @@ CHATTERBOX_MODELS = {
         "format": "safetensors",
         "description": "German ChatterBox model with high quality"
     },
+    "German (SebastianBodza)": {
+        "repo": "SebastianBodza/Kartoffelbox-v0.1",
+        "format": "mixed",
+        "description": "German model with emotion control features (<haha>, <wow> tags) - 600K samples"
+    },
+    "German (havok2)": {
+        "repo": "havok2/Kartoffelbox-v0.1_0.65h2",
+        "format": "safetensors",
+        "subdirectory": "finetuned_by_h2",
+        "description": "Hybrid German model (65% multi-speaker + 35% Kartoffelbox) - User-rated best quality"
+    },
     "Norwegian": {
         "repo": "akhbar/chatterbox-tts-norwegian", 
         "format": "safetensors",
         "description": "Norwegian ChatterBox model (Bokmål and Nynorsk dialects) - 532M parameters"
     },
+    "French": {
+        "repo": "Thomcles/Chatterbox-TTS-French",
+        "format": "safetensors",
+        "description": "French model trained on 1,400 hours of Emilia dataset - t3_cfg only, uses shared English components",
+        "incomplete": True
+    },
+    "Russian": {
+        "repo": "niobures/Chatterbox-TTS",
+        "format": "safetensors",
+        "subdirectory": "ru",
+        "description": "Russian model with training artifacts - Complete fine-tuned model"
+    },
+    "Armenian": {
+        "repo": "niobures/Chatterbox-TTS",
+        "format": "safetensors", 
+        "subdirectory": "hy",
+        "description": "Armenian model with training artifacts - Complete fine-tuned model"
+    },
+    "Georgian": {
+        "repo": "niobures/Chatterbox-TTS",
+        "format": "safetensors",
+        "subdirectory": "ka", 
+        "description": "Georgian model - Complete fine-tuned model with unique architecture"
+    },
+    "Japanese": {
+        "repo": "niobures/Chatterbox-TTS",
+        "format": "safetensors",
+        "subdirectory": "ja",
+        "description": "Japanese model - t3_cfg only, uses shared English components",
+        "incomplete": True
+    },
+    "Korean": {
+        "repo": "niobures/Chatterbox-TTS", 
+        "format": "safetensors",
+        "subdirectory": "ko",
+        "description": "Korean model - t3_cfg only, uses shared English components",
+        "incomplete": True
+    },
+    "Italian": {
+        "repo": "niobures/Chatterbox-TTS",
+        "format": "pt",
+        "subdirectory": "it,en",
+        "description": "Bilingual Italian/English model - complete model in single file, extended vocabulary (1500 tokens)",
+        "special": "unified_model"
+    },
 }
 
 def get_chatterbox_models() -> List[str]:
@@ -80,7 +136,14 @@ def get_model_config(language: str) -> Optional[Dict]:
             "description": f"Local ChatterBox model: {local_name}"
         }
     
-    return CHATTERBOX_MODELS.get(language)
+    config = CHATTERBOX_MODELS.get(language)
+    if config and config.get("subdirectory"):
+        # Create a copy with the subdirectory information for download handling
+        config_copy = config.copy()
+        config_copy["download_path"] = f"{config['repo']}/{config['subdirectory']}"
+        return config_copy
+    
+    return config
 
 def get_model_files_for_language(language: str) -> Tuple[str, str]:
     """
@@ -143,6 +206,112 @@ def detect_model_format(model_path: str) -> str:
     else:
         return "pt"
 
+def is_model_incomplete(language: str) -> bool:
+    """Check if a model is marked as incomplete (missing components)"""
+    config = CHATTERBOX_MODELS.get(language)
+    return config.get("incomplete", False) if config else False
+
+def is_unified_model(language: str) -> bool:
+    """Check if a model uses unified/special architecture (like Italian)"""
+    config = CHATTERBOX_MODELS.get(language)
+    return config.get("special") == "unified_model" if config else False
+
+def get_tokenizer_filename(language: str) -> str:
+    """Get the correct tokenizer filename for a language model"""
+    if language == "Japanese":
+        return "tokenizer_jp.json"
+    elif language == "Korean":
+        return "tokenizer_en_ko.json"
+    else:
+        return "tokenizer.json"
+
+def get_model_requirements(language: str) -> List[str]:
+    """Get list of required files for a ChatterBox model"""
+    
+    # Handle special Italian unified model
+    if language == "Italian":
+        return ["chatterbox_italian_final.pt", "config.json"]
+    
+    # Handle special cases for incomplete models
+    if language == "French":
+        # French model only has t3_cfg.safetensors, no tokenizer in repo
+        base_requirements = ["t3_cfg.safetensors"]
+    else:
+        # Other models have tokenizers
+        tokenizer_file = get_tokenizer_filename(language)
+        base_requirements = ["t3_cfg.safetensors", tokenizer_file]
+    
+    # Complete models need all components
+    if not is_model_incomplete(language):
+        base_requirements.extend(["s3gen.safetensors", "ve.safetensors", "conds.pt"])
+    
+    return base_requirements
+
+def validate_model_completeness(model_path: str, language: str) -> Tuple[bool, List[str]]:
+    """
+    Validate if a model has all required components.
+    Returns (is_complete, missing_files)
+    """
+    if not os.path.exists(model_path):
+        return False, ["Model directory not found"]
+    
+    required_files = get_model_requirements(language)
+    missing_files = []
+    
+    existing_files = os.listdir(model_path)
+    for required_file in required_files:
+        # Check for file with correct extension
+        file_found = False
+        for existing_file in existing_files:
+            if existing_file == required_file or existing_file.startswith(required_file.split('.')[0] + '.'):
+                file_found = True
+                break
+        
+        if not file_found:
+            missing_files.append(required_file)
+    
+    return len(missing_files) == 0, missing_files
+
+def supports_voice_conversion(language: str) -> bool:
+    """
+    Check if a language model supports voice conversion (has s3gen component).
+    
+    Args:
+        language: Language model name
+        
+    Returns:
+        True if VC is supported, False otherwise
+    """
+    # Check if model is marked as incomplete
+    if is_model_incomplete(language):
+        return False
+    
+    # Check if model requirements include s3gen (required for VC)
+    requirements = get_model_requirements(language)
+    has_s3gen = any(req.startswith("s3gen") for req in requirements)
+    
+    # For local models, check if s3gen file actually exists
+    if language.startswith("local:"):
+        local_path = find_local_model_path(language)
+        if local_path and os.path.exists(local_path):
+            # Check if s3gen file exists in the directory
+            for ext in [".safetensors", ".pt"]:
+                if os.path.exists(os.path.join(local_path, f"s3gen{ext}")):
+                    return True
+            return False
+    
+    return has_s3gen
+
+def get_vc_supported_languages() -> List[str]:
+    """Get list of languages that support voice conversion"""
+    all_languages = get_chatterbox_models()
+    return [lang for lang in all_languages if supports_voice_conversion(lang)]
+
+def get_vc_unsupported_languages() -> List[str]:
+    """Get list of languages that do NOT support voice conversion"""
+    all_languages = get_chatterbox_models()
+    return [lang for lang in all_languages if not supports_voice_conversion(lang)]
+
 def get_available_languages() -> List[str]:
     """Get list of available language names for display"""
     models = get_chatterbox_models()

engines/chatterbox/models/s3tokenizer/s3tokenizer.py

@@ -1,4 +1,16 @@
 from typing import List, Tuple
+import os
+import sys
+
+# 🔬 NUMBA WORKAROUND: Commented out - testing if still needed with numba 0.61.2+ and librosa 0.11.0+
+# Python 3.13 compatibility: Disable numba JIT for librosa compatibility
+# if sys.version_info >= (3, 13):
+#     os.environ['NUMBA_DISABLE_JIT'] = '1'
+#     try:
+#         import numba
+#         numba.config.DISABLE_JIT = True
+#     except ImportError:
+#         pass
 
 import numpy as np
 import librosa

engines/chatterbox/models/t3/inference/t3_hf_backend.py

@@ -119,10 +119,10 @@ class T3HuggingfaceBackend(LlamaPreTrainedModel, GenerationMixin):
             past_key_values=pkv,
             use_cache=use_cache,
             output_attentions=False,  # Always disable attention outputs
-            output_hidden_states=True,
             return_dict=True,
         )
-        hidden_states = tfmr_out.hidden_states[-1]  # (B, seq, dim)
+        # Use last_hidden_state directly instead of hidden_states (transformers compatibility)
+        hidden_states = tfmr_out.last_hidden_state  # (B, seq, dim)
 
         logits = self.speech_head(hidden_states)
         # assert inputs_embeds.size(0) == 1 # (disabled for CFG)

engines/chatterbox/models/t3/llama_configs.py

@@ -8,7 +8,7 @@ LLAMA_520M_CONFIG_DICT = dict(
     intermediate_size=4096,
     num_hidden_layers=30,
     num_attention_heads=16,
-    attn_implementation="sdpa",
+    attn_implementation="eager",  # Changed from sdpa to eager for compatibility with alignment analyzer
     head_dim=64,
     tie_word_embeddings=False,
     hidden_act="silu",

engines/chatterbox/models/t3/t3.py

@@ -93,8 +93,11 @@ class T3(nn.Module):
         Token cond data needs to be embedded, so that needs to be here instead of in `T3CondEnc`.
         """
         if t3_cond.cond_prompt_speech_tokens is not None and t3_cond.cond_prompt_speech_emb is None:
-            t3_cond.cond_prompt_speech_emb = self.speech_emb(t3_cond.cond_prompt_speech_tokens) + \
-                self.speech_pos_emb(t3_cond.cond_prompt_speech_tokens)
+            # CRITICAL: Ensure tokens are on the correct device before embedding
+            # (fixes device mismatch when models are reused after CPU offloading)
+            cond_tokens = t3_cond.cond_prompt_speech_tokens.to(device=self.device)
+            t3_cond.cond_prompt_speech_emb = self.speech_emb(cond_tokens) + \
+                self.speech_pos_emb(cond_tokens)
         return self.cond_enc(t3_cond)  # (B, len_cond, dim)
 
     def prepare_input_embeds(
@@ -391,7 +394,6 @@ class T3(nn.Module):
                 past_key_values=cache,
                 use_cache=True,
                 output_attentions=False,
-                output_hidden_states=True,
                 return_dict=True
             )
             

engines/chatterbox/models/voice_encoder/melspec.py

@@ -2,13 +2,14 @@ from functools import lru_cache
 
 from scipy import signal
 import numpy as np
-import librosa
+# Use librosa fallback for Python 3.13 compatibility
+from utils.audio.librosa_fallback import safe_mel_filters, safe_stft
 
 
 @lru_cache()
 def mel_basis(hp):
     assert hp.fmax <= hp.sample_rate // 2
-    return librosa.filters.mel(
+    return safe_mel_filters(
         sr=hp.sample_rate,
         n_fft=hp.n_fft,
         n_mels=hp.num_mels,
@@ -54,7 +55,7 @@ def melspectrogram(wav, hp, pad=True):
 def _stft(y, hp, pad=True):
     # NOTE: after 0.8, pad mode defaults to constant, setting this to reflect for
     #   historical consistency and streaming-version consistency
-    return librosa.stft(
+    return safe_stft(
         y,
         n_fft=hp.n_fft,
         hop_length=hp.hop_size,

engines/chatterbox/models/voice_encoder/voice_encoder.py

@@ -4,13 +4,14 @@ from typing import List, Union, Optional
 
 import numpy as np
 from numpy.lib.stride_tricks import as_strided
-import librosa
 import torch
 import torch.nn.functional as F
 from torch import nn, Tensor
 
 from .config import VoiceEncConfig
 from .melspec import melspectrogram
+# Use librosa fallback for Python 3.13 compatibility
+from utils.audio.librosa_fallback import safe_resample, safe_trim
 
 
 def pack(arrays, seq_len: int=None, pad_value=0):
@@ -258,13 +259,15 @@ class VoiceEncoder(nn.Module):
         :param trim_top_db: this argument was only added for the sake of compatibility with metavoice's implementation
         """
         if sample_rate != self.hp.sample_rate:
+            # Use fallback for resampling for Python 3.13 compatibility
             wavs = [
-                librosa.resample(wav, orig_sr=sample_rate, target_sr=self.hp.sample_rate, res_type="kaiser_fast")
+                safe_resample(wav, sample_rate, self.hp.sample_rate)
                 for wav in wavs
             ]
 
         if trim_top_db:
-            wavs = [librosa.effects.trim(wav, top_db=trim_top_db)[0] for wav in wavs]
+            # Use fallback trimming for Python 3.13 compatibility
+            wavs = [safe_trim(wav, top_db=trim_top_db) for wav in wavs]
 
         if "rate" not in kwargs:
             kwargs["rate"] = 1.3  # Resemble's default value.

engines/chatterbox/stateless_wrapper.py

@@ -13,23 +13,20 @@ Key benefits:
 
 import torch
 import torch.nn.functional as F
-import librosa
 import numpy as np
 from dataclasses import dataclass
 from pathlib import Path
 import warnings
 
+# Use librosa fallback for Python 3.13 compatibility
+from utils.audio.librosa_fallback import safe_load, safe_resample
+
 # Import ChatterBox components
 from engines.chatterbox.models.s3tokenizer import S3_SR, drop_invalid_tokens
 from engines.chatterbox.models.s3gen import S3GEN_SR
 from engines.chatterbox.models.t3.modules.cond_enc import T3Cond
 from engines.chatterbox.tts import Conditionals, punc_norm
 
-# Suppress perth warnings
-with warnings.catch_warnings():
-    warnings.simplefilter("ignore")
-    import perth
-
 
 class StatelessChatterBoxWrapper:
     """
@@ -164,11 +161,11 @@ class StatelessChatterBoxWrapper:
             if not wav_fpath:
                 return self._get_default_conditions(exaggeration)
             
-            # Load reference wav
-            s3gen_ref_wav, _sr = librosa.load(wav_fpath, sr=S3GEN_SR)
+            # Load reference wav using fallback for Python 3.13 compatibility
+            s3gen_ref_wav, sample_rate = safe_load(wav_fpath, sr=S3GEN_SR, mono=True)
             
-            # Resample to 16k for S3 tokenizer
-            ref_16k_wav = librosa.resample(s3gen_ref_wav, orig_sr=S3GEN_SR, target_sr=S3_SR)
+            # Resample to 16k for S3 tokenizer using fallback
+            ref_16k_wav = safe_resample(s3gen_ref_wav, S3GEN_SR, S3_SR)
             
             # Prepare S3Gen reference
             s3gen_ref_wav = s3gen_ref_wav[:self.DEC_COND_LEN]

engines/chatterbox/streaming_model_manager.py

@@ -30,42 +30,38 @@ class StreamingModelManager:
         return models
     
     def preload_models(self, language_codes: List[str], model_manager, device: str) -> None:
-        """Pre-load all required models for the given languages using universal smart loader."""
+        """Pre-load all required models for the given languages using unified interface."""
         required_models = self.get_required_models(language_codes)
         available_languages = get_available_languages()
-        
+
         print(f"🚀 STREAMING: Pre-loading {len(required_models)} models for {len(language_codes)} languages")
-        
-        # Use universal smart model loader for consistency
-        from utils.models.smart_loader import smart_model_loader
-        
+
+        # Use unified interface for consistency
+        from utils.models.unified_model_interface import load_tts_model
+
         for model_name in required_models:
             if model_name in self.preloaded_models:
                 print(f"♻️ {model_name} already loaded in streaming cache")
                 continue
-                
+
             if model_name not in available_languages:
                 print(f"⚠️ {model_name} model not available, using English fallback")
                 model_name = 'English'
-            
-            # Use smart loader to get or load the model
+
+            # Use unified interface to load the model
             try:
-                model_instance, was_cached = smart_model_loader.load_model_if_needed(
-                    engine_type="chatterbox",
+                model_instance = load_tts_model(
+                    engine_name="chatterbox",
                     model_name=model_name,
-                    current_model=self.preloaded_models.get(model_name),
                     device=device,
-                    load_callback=lambda d, m: model_manager.load_tts_model(d, m)
+                    language=model_name,
+                    force_reload=False
                 )
-                
+
                 # Store reference in our streaming cache
                 self.preloaded_models[model_name] = model_instance
-                
-                if was_cached:
-                    print(f"♻️ STREAMING: Reused {model_name} from smart loader (ID: {id(model_instance)})")
-                else:
-                    print(f"✅ STREAMING: Loaded {model_name} via smart loader (ID: {id(model_instance)})")
-                    
+                print(f"✅ STREAMING: Loaded {model_name} (ID: {id(model_instance)})")
+
             except Exception as e:
                 print(f"❌ Failed to load {model_name}: {e}")
                 # Try fallback to English if not already English
@@ -74,11 +70,11 @@ class StreamingModelManager:
                     self.preloaded_models[model_name] = self.preloaded_models['English']
                 else:
                     print(f"❌ No fallback available for {model_name}")
-            
+
             # Debug: Show all current model IDs
             if len(self.preloaded_models) > 1:
                 print(f"🔍 DEBUG: All stored model IDs: {[(k, id(v)) for k, v in self.preloaded_models.items()]}")
-        
+
         print(f"🚀 Model pre-loading complete! {len(self.preloaded_models)} models ready")
     
     def get_model_for_language(self, language_code: str, fallback_model=None):
@@ -116,6 +112,16 @@ class StreamingModelManager:
             return stateless_wrapper
         return None
     
+    def load_vc_model(self, device: str = "auto", force_reload: bool = False, language: str = "English"):
+        """
+        Voice Conversion not supported in StreamingModelManager.
+        This method exists to prevent signature errors when VC nodes try to call it.
+        """
+        raise NotImplementedError(
+            "Voice Conversion is not supported with StreamingModelManager. "
+            "StreamingModelManager is for TTS only. VC requires the main ModelManager from utils.models.manager."
+        )
+    
     def cleanup(self):
         """Clean up pre-loaded models to free memory."""
         print(f"🧹 Cleaning up {len(self.preloaded_models)} pre-loaded models")

engines/chatterbox/tts.py

@@ -2,11 +2,13 @@ from dataclasses import dataclass
 from pathlib import Path
 import os
 
-import librosa
 import torch
 import torch.nn.functional as F
 from huggingface_hub import hf_hub_download
 import warnings
+
+# Use librosa fallback for Python 3.13 compatibility
+from utils.audio.librosa_fallback import safe_load, safe_resample
 # Import safetensors for multilanguage model support
 from safetensors.torch import load_file
 
@@ -16,10 +18,15 @@ try:
 except ImportError:
     folder_paths = None
 
-# Import perth with warnings disabled
-with warnings.catch_warnings():
-    warnings.simplefilter("ignore")
-    import perth
+# Import perth with warnings disabled and graceful fallback
+try:
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore")
+        import perth
+    PERTH_AVAILABLE = True
+except ImportError:
+    perth = None
+    PERTH_AVAILABLE = False
 
 from .models.t3 import T3
 from .models.s3tokenizer import S3_SR, drop_invalid_tokens
@@ -30,12 +37,22 @@ from .models.t3.modules.cond_enc import T3Cond
 
 # Import language model registry
 try:
-    from .language_models import get_model_config, CHATTERBOX_MODELS
+    from .language_models import get_model_config, CHATTERBOX_MODELS, get_model_requirements, validate_model_completeness, get_tokenizer_filename, is_model_incomplete, is_unified_model
 except ImportError:
     # Fallback if language_models not available
     CHATTERBOX_MODELS = {"English": {"repo": "ResembleAI/chatterbox", "format": "pt"}}
     def get_model_config(language):
         return CHATTERBOX_MODELS.get(language, CHATTERBOX_MODELS["English"])
+    def get_model_requirements(language):
+        return ["t3_cfg.safetensors", "s3gen.safetensors", "ve.safetensors", "conds.pt", "tokenizer.json"]
+    def validate_model_completeness(model_path, language):
+        return True, []
+    def get_tokenizer_filename(language):
+        return "tokenizer.json"
+    def is_model_incomplete(language):
+        return False
+    def is_unified_model(language):
+        return False
 
 # Import modular processors
 from .overlapping_processor import OverlappingBatchProcessor, BatchingStrategy
@@ -107,6 +124,9 @@ class Conditionals:
 
     def to(self, device):
         self.t3 = self.t3.to(device=device)
+        # CRITICAL: Also move s3gen model to device (fixes speaker_encoder device mismatch)
+        if hasattr(self, 's3gen') and self.s3gen is not None:
+            self.s3gen = self.s3gen.to(device=device)
         for k, v in self.gen.items():
             if torch.is_tensor(v):
                 self.gen[k] = v.to(device=device)
@@ -145,49 +165,156 @@ class ChatterboxTTS:
         self.tokenizer = tokenizer
         self.device = device
         self.conds = conds
-        # Initialize watermarker silently (but disabled by default)
-        with warnings.catch_warnings():
-            warnings.simplefilter("ignore")
-            self.watermarker = perth.PerthImplicitWatermarker()
-        self.enable_watermarking = False  # Disabled by default for maximum compatibility
+        # Watermarking disabled by default for maximum compatibility
+        # Set to True to enable watermarking (requires perth library)
+        self.enable_watermarking = False
+        self.watermarker = None
+        self._watermarker_init_attempted = False
         
         # Initialize modular processors
         self.overlapping_processor = OverlappingBatchProcessor(self)
 
+    def to(self, device):
+        """
+        Move all model components to the specified device.
+
+        Critical for ComfyUI model management - ensures all components move together
+        when models are detached to CPU and later reloaded to CUDA.
+        """
+        self.device = device
+
+        # Move all PyTorch model components
+        if hasattr(self.t3, 'to'):
+            self.t3 = self.t3.to(device)
+        if hasattr(self.s3gen, 'to'):
+            self.s3gen = self.s3gen.to(device)
+        if hasattr(self.ve, 'to'):
+            self.ve = self.ve.to(device)
+        if hasattr(self.tokenizer, 'to'):
+            self.tokenizer = self.tokenizer.to(device)
+        if self.conds is not None and hasattr(self.conds, 'to'):
+            self.conds = self.conds.to(device)
+
+        return self
+
+    def _init_watermarker_if_needed(self):
+        """Initialize watermarker on first use if enabled"""
+        if self.enable_watermarking and not self._watermarker_init_attempted:
+            self._watermarker_init_attempted = True
+            try:
+                with warnings.catch_warnings():
+                    warnings.simplefilter("ignore")
+                    self.watermarker = perth.PerthImplicitWatermarker()
+                    if self.watermarker is None:
+                        raise ValueError("PerthImplicitWatermarker returned None")
+            except Exception as e:
+                print(f"❌ Failed to initialize watermarker: {e}")
+                self.watermarker = None
+                self.enable_watermarking = False
+
     @classmethod
-    def from_local(cls, ckpt_dir, device) -> 'ChatterboxTTS':
-        print(f"📦 Loading local ChatterBox models from: {ckpt_dir}")
+    def from_local(cls, ckpt_dir, device, language=None) -> 'ChatterboxTTS':
         ckpt_dir = Path(ckpt_dir)
+
+        # Resolve "auto" device to actual device
+        from utils.device import resolve_torch_device
+        device = resolve_torch_device(device)
+
+        # Handle Italian unified model (special architecture)
+        if language and is_unified_model(language):
+            return cls._load_unified_model(ckpt_dir, device, language)
         
-        # Auto-detect model format
-        def load_model_file(base_name: str):
+        # Determine if this is an incomplete model by checking language or directory structure
+        is_incomplete_model = False
+        if language:
+            is_incomplete_model = is_model_incomplete(language)
+        else:
+            # Check if critical files are missing (indicates incomplete model)
+            ve_exists = any((ckpt_dir / f"ve.{ext}").exists() for ext in ["safetensors", "pt"])
+            s3gen_exists = any((ckpt_dir / f"s3gen.{ext}").exists() for ext in ["safetensors", "pt"])
+            is_incomplete_model = not (ve_exists and s3gen_exists)
+        
+        # Auto-detect model format with fallback support for incomplete models
+        def load_model_file(base_name: str, required: bool = True):
             """Load model file with auto-detection of format (.safetensors preferred over .pt)"""
             safetensors_path = ckpt_dir / f"{base_name}.safetensors"
             pt_path = ckpt_dir / f"{base_name}.pt"
             
             if safetensors_path.exists():
-                print(f"📁 Loading {base_name} from safetensors format")
                 return load_file(safetensors_path, device=device)
             elif pt_path.exists():
-                print(f"📁 Loading {base_name} from pt format")
                 return torch.load(pt_path, map_location=device)
+            elif not required:
+                return None
             else:
                 raise FileNotFoundError(f"Neither {base_name}.safetensors nor {base_name}.pt found in {ckpt_dir}")
         
+        def load_from_english_fallback(base_name: str):
+            """Load component from English model when missing in incomplete model"""
+            from utils.downloads.unified_downloader import UnifiedDownloader
+            from .language_models import find_local_model_path
+            
+            # Try to find local English model first
+            english_path = find_local_model_path("English")
+            if english_path:
+                english_dir = Path(english_path)
+                safetensors_path = english_dir / f"{base_name}.safetensors"
+                pt_path = english_dir / f"{base_name}.pt"
+                
+                if safetensors_path.exists():
+                    print(f"📁 Loading {base_name} from local English model: {safetensors_path}")
+                    return load_file(safetensors_path, device=device)
+                elif pt_path.exists():
+                    print(f"📁 Loading {base_name} from local English model: {pt_path}")
+                    return torch.load(pt_path, map_location=device)
+            
+            # Download English model if not available locally
+            print(f"📦 Downloading English model components for incomplete language model...")
+            downloader = UnifiedDownloader()
+            english_dir = downloader.download_chatterbox_model("ResembleAI/chatterbox", "English")
+            if english_dir:
+                english_path = Path(english_dir)
+                safetensors_path = english_path / f"{base_name}.safetensors"
+                pt_path = english_path / f"{base_name}.pt"
+                
+                if safetensors_path.exists():
+                    print(f"📁 Loading {base_name} from downloaded English model: {safetensors_path}")
+                    return load_file(safetensors_path, device=device)
+                elif pt_path.exists():
+                    print(f"📁 Loading {base_name} from downloaded English model: {pt_path}")
+                    return torch.load(pt_path, map_location=device)
+            
+            raise FileNotFoundError(f"Could not load {base_name} from English fallback")
+        
         with warnings.catch_warnings():
             warnings.simplefilter("ignore")
             
-            # Load VoiceEncoder
+            # Load VoiceEncoder (use English fallback for incomplete models)
             ve = VoiceEncoder()
-            ve_state = load_model_file("ve")
+            ve_state = load_model_file("ve", required=not is_incomplete_model)
+            if ve_state is None and is_incomplete_model:
+                print(f"📎 Loading VoiceEncoder from English model (incomplete model fallback)")
+                ve_state = load_from_english_fallback("ve")
             ve.load_state_dict(ve_state)
             ve.to(device).eval()
 
-            # Load T3 config
+            # Load T3 config (always required)
             t3_state = load_model_file("t3_cfg")
             if "model" in t3_state.keys():
                 t3_state = t3_state["model"][0]
             
+            # Handle Japanese/Korean model state dict format (keys have "t3." prefix)
+            # Check if keys have "t3." prefix and need to be remapped
+            sample_keys = list(t3_state.keys())[:5]  # Check first few keys
+            if sample_keys and all(key.startswith("t3.") for key in sample_keys):
+                print(f"🔧 Remapping state dict keys (removing 't3.' prefix) for incomplete model compatibility")
+                # Remove "t3." prefix from all keys
+                new_state_dict = {}
+                for key, value in t3_state.items():
+                    new_key = key[3:] if key.startswith("t3.") else key  # Remove "t3." prefix
+                    new_state_dict[new_key] = value
+                t3_state = new_state_dict
+            
             # Create config with proper settings
             from .models.t3.t3 import T3Config
             config = T3Config()
@@ -201,23 +328,157 @@ class ChatterboxTTS:
             
             t3.to(device).eval()
 
-            # Load S3Gen
+            # Load S3Gen (use English fallback for incomplete models)
             s3gen = S3Gen()
-            s3gen_state = load_model_file("s3gen")
+            s3gen_state = load_model_file("s3gen", required=not is_incomplete_model)
+            if s3gen_state is None and is_incomplete_model:
+                print(f"📎 Loading S3Gen from English model (incomplete model fallback)")
+                s3gen_state = load_from_english_fallback("s3gen")
             # Apply JaneDoe84's critical fix: strict=False to handle missing keys
             s3gen.load_state_dict(s3gen_state, strict=False)
             s3gen.to(device).eval()
 
-            tokenizer = EnTokenizer(
-                str(ckpt_dir / "tokenizer.json")
-            )
+            # Find the correct tokenizer file (prioritize language-specific tokenizer)
+            tokenizer_file = None
+            
+            # First try the language-specific tokenizer based on language parameter
+            if language:
+                expected_tokenizer = get_tokenizer_filename(language)
+                expected_path = ckpt_dir / expected_tokenizer
+                if expected_path.exists():
+                    tokenizer_file = str(expected_path)
+                    print(f"🔤 Using language-specific tokenizer: {expected_tokenizer}")
+            
+            # If language-specific not found, try available tokenizers
+            if not tokenizer_file:
+                for possible_tokenizer in ["tokenizer_jp.json", "tokenizer_en_ko.json", "tokenizer.json"]:
+                    tokenizer_path = ckpt_dir / possible_tokenizer
+                    if tokenizer_path.exists():
+                        tokenizer_file = str(tokenizer_path)
+                        print(f"🔤 Using available tokenizer: {possible_tokenizer}")
+                        break
+            
+            # For incomplete models, fall back to English tokenizer
+            if not tokenizer_file and is_incomplete_model:
+                print(f"📎 Loading tokenizer from English model (incomplete model fallback)")
+                # Try to find English model locally first
+                from .language_models import find_local_model_path
+                english_path = find_local_model_path("English")
+                if english_path:
+                    english_tokenizer = Path(english_path) / "tokenizer.json"
+                    if english_tokenizer.exists():
+                        tokenizer_file = str(english_tokenizer)
+                        print(f"🔤 Using English tokenizer: {english_tokenizer}")
+                else:
+                    # Download English model if not available locally
+                    from utils.downloads.unified_downloader import UnifiedDownloader
+                    downloader = UnifiedDownloader()
+                    english_dir = downloader.download_chatterbox_model("ResembleAI/chatterbox", "English")
+                    if english_dir:
+                        english_path = Path(english_dir)
+                        english_tokenizer = english_path / "tokenizer.json"
+                        if english_tokenizer.exists():
+                            tokenizer_file = str(english_tokenizer)
+                            print(f"🔤 Using English tokenizer: {english_tokenizer}")
+            
+            if not tokenizer_file:
+                raise FileNotFoundError(f"No tokenizer file found in {ckpt_dir}")
+            
+            tokenizer = EnTokenizer(tokenizer_file)
 
             conds = None
             if (builtin_voice := ckpt_dir / "conds.pt").exists():
                 conds = Conditionals.load(builtin_voice).to(device)
 
             instance = cls(t3, s3gen, ve, tokenizer, device, conds=conds)
-            print("✅ Successfully loaded all local ChatterBox models")
+            print(f"📦 ChatterBox models loaded from: {ckpt_dir}")
+            return instance
+
+    @classmethod
+    def _load_unified_model(cls, ckpt_dir, device, language) -> 'ChatterboxTTS':
+        """
+        Load unified model (like Italian) that contains all components in a single checkpoint file.
+        Based on the deployment script from niobures/Chatterbox-TTS/it,en/deploy_italian_tts.py
+        """
+        ckpt_dir = Path(ckpt_dir)
+        print(f"🇮🇹 Loading unified {language} ChatterBox model from: {ckpt_dir}")
+        
+        # Load the unified model file
+        unified_model_path = ckpt_dir / "chatterbox_italian_final.pt"
+        if not unified_model_path.exists():
+            raise FileNotFoundError(f"Unified model file not found: {unified_model_path}")
+        
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore")
+            
+            print(f"📦 Loading checkpoint: {unified_model_path.name}")
+            checkpoint = torch.load(unified_model_path, map_location=device)
+            
+            # Extract model configuration
+            model_config = checkpoint.get('model_config', {})
+            vocab_size = model_config.get('vocab_size', 1500)
+            frozen_embeddings = model_config.get('frozen_embeddings', 704)
+            trainable_tokens = model_config.get('trainable_tokens', 796)
+            
+            print(f"🎯 Vocab size: {vocab_size}")
+            print(f"🔒 Frozen tokens: {frozen_embeddings}")
+            print(f"🔥 Trainable tokens: {trainable_tokens}")
+            
+            # Load base English ChatterBox components first
+            print(f"📦 Loading base English model components...")
+            
+            # Initialize components with standard ChatterBox architecture
+            ve = VoiceEncoder()
+            ve.load_state_dict(checkpoint['ve_state_dict'], strict=False)
+            ve.to(device).eval()
+            
+            # Load T3 config with Italian vocabulary size
+            from .models.t3.t3 import T3Config
+            config = T3Config()
+            # Override text token dictionary size for Italian extended vocab
+            config.text_tokens_dict_size = vocab_size  # Use the 1500 tokens from checkpoint
+            t3 = T3(config)
+            t3.load_state_dict(checkpoint['t3_state_dict'], strict=False)
+            t3.tfmr.output_attentions = False
+            t3.to(device).eval()
+            
+            # Load S3Gen
+            s3gen = S3Gen()
+            s3gen.load_state_dict(checkpoint['s3gen_state_dict'], strict=False)
+            s3gen.to(device).eval()
+            
+            # Load tokenizer for Italian model (fallback to English tokenizer)
+            # The Italian model uses language prefixes like [it] for Italian text
+            from .models.tokenizers import EnTokenizer
+            from .language_models import find_local_model_path
+            
+            # Try to use existing tokenizer if available in Italian model
+            tokenizer_path = ckpt_dir / "tokenizer.json"
+            if tokenizer_path.exists():
+                tokenizer = EnTokenizer(str(tokenizer_path))
+                print(f"🔤 Using Italian model tokenizer: tokenizer.json")
+            else:
+                # Fallback: use English model tokenizer (Italian extends English vocab)
+                english_model_path = find_local_model_path("English")
+                if english_model_path:
+                    english_tokenizer_path = Path(english_model_path) / "tokenizer.json"
+                    if english_tokenizer_path.exists():
+                        tokenizer = EnTokenizer(str(english_tokenizer_path))
+                        print(f"🔤 Using English tokenizer fallback: {english_tokenizer_path}")
+                    else:
+                        raise FileNotFoundError(f"No tokenizer found for Italian model and no English fallback available")
+                else:
+                    raise FileNotFoundError(f"No tokenizer available for Italian model and no English model found for fallback")
+            
+            # Check for conditional voices
+            conds = None
+            if (builtin_voice := ckpt_dir / "conds.pt").exists():
+                conds = Conditionals.load(builtin_voice).to(device)
+            
+            instance = cls(t3, s3gen, ve, tokenizer, device, conds=conds)
+            print(f"✅ Italian TTS model loaded successfully!")
+            print(f"💡 Use '[it]' prefix for Italian text, no prefix for English")
+            print(f"📦 Model loaded from: {ckpt_dir}")
             return instance
 
     @classmethod
@@ -252,7 +513,7 @@ class ChatterboxTTS:
                 try:
                     return try_local_first(
                         search_paths=search_paths,
-                        local_loader=lambda path: cls.from_local(path, device),
+                        local_loader=lambda path: cls.from_local(path, device, language),
                         fallback_loader=lambda: cls.from_pretrained(device, language="English"),
                         fallback_name="English",
                         original_request=language
@@ -265,119 +526,58 @@ class ChatterboxTTS:
         repo_id = model_config.get("repo", REPO_ID)
         model_format = model_config.get("format", "pt")
         
-        print(f"📦 Loading ChatterBox model for {language} from {repo_id}")
+        # Silent loading unless errors occur
         
-        # Define file extensions based on format
-        if model_format == "safetensors":
-            file_extensions = ["ve.safetensors", "t3_cfg.safetensors", "s3gen.safetensors", "tokenizer.json", "conds.pt"]
-        elif model_format == "pt":
-            file_extensions = ["ve.pt", "t3_cfg.pt", "s3gen.pt", "tokenizer.json", "conds.pt"]
-        else:
-            # Auto format - try both, safetensors preferred
-            print("🔍 Auto-detecting model format...")
-            file_extensions = ["ve.safetensors", "t3_cfg.safetensors", "s3gen.safetensors", "tokenizer.json", "conds.pt"]
-        
-        # Download files to local models directory
-        local_paths = []
-        for fpath in file_extensions:
-            # Define local path in TTS organization
-            local_model_path = os.path.join(folder_paths.models_dir, "TTS", "chatterbox", language, os.path.basename(fpath))
-            
-            if os.path.exists(local_model_path):
-                print(f"📁 Using local Chatterbox model: {local_model_path}")
-                local_paths.append(local_model_path)
-                continue
-            
-            # Check HuggingFace cache first
-            try:
-                hf_cached_file = hf_hub_download(repo_id=repo_id, filename=fpath, local_files_only=True)
-                print(f"📁 Using cached Chatterbox model: {hf_cached_file}")
-                local_paths.append(hf_cached_file)
-                continue
-            except Exception:
-                pass
-            
-            # Download to local directory
-            print(f"📥 Downloading Chatterbox model to local directory: {fpath}")
-            os.makedirs(os.path.dirname(local_model_path), exist_ok=True)
-            
-            try:
-                # Use unified downloader
-                from utils.downloads.unified_downloader import unified_downloader
-                
-                files_to_download = [{
-                    'remote': fpath,
-                    'local': os.path.basename(fpath)
-                }]
-                
-                downloaded_dir = unified_downloader.download_huggingface_model(
-                    repo_id=repo_id,
-                    model_name=language,
-                    files=files_to_download,
-                    engine_type="chatterbox"
-                )
-                
-                if downloaded_dir:
-                    local_model_path = os.path.join(downloaded_dir, os.path.basename(fpath))
-                    local_paths.append(local_model_path)
-                    print(f"✅ Downloaded Chatterbox model to: {local_model_path}")
-                else:
-                    raise Exception("Unified download failed")
-            except Exception as e:
-                # If safetensors fails, try .pt format
-                if fpath.endswith('.safetensors'):
-                    fallback_fpath = fpath.replace('.safetensors', '.pt')
-                    fallback_local_path = os.path.join(folder_paths.models_dir, "TTS", "chatterbox", language, os.path.basename(fallback_fpath))
-                    
-                    print(f"⚠️ {fpath} not found, trying {fallback_fpath}")
-                    try:
-                        # Use unified downloader for fallback
-                        files_to_download = [{
-                            'remote': fallback_fpath,
-                            'local': os.path.basename(fallback_fpath)
-                        }]
-                        
-                        downloaded_dir = unified_downloader.download_huggingface_model(
-                            repo_id=repo_id,
-                            model_name=language,
-                            files=files_to_download,
-                            engine_type="chatterbox"
-                        )
-                        
-                        if downloaded_dir:
-                            fallback_local_path = os.path.join(downloaded_dir, os.path.basename(fallback_fpath))
-                            local_paths.append(fallback_local_path)
-                            print(f"✅ Downloaded Chatterbox fallback model to: {fallback_local_path}")
-                        else:
-                            raise Exception("Unified fallback download failed")
-                    except Exception as e2:
-                        # Final fallback to HuggingFace cache
-                        print(f"⚠️ Failed to download to local directory, using HF cache: {e2}")
-                        try:
-                            local_path = hf_hub_download(repo_id=repo_id, filename=fallback_fpath)
-                            local_paths.append(local_path)
-                        except Exception as e3:
-                            print(f"❌ Failed to download both {fpath} and {fallback_fpath}: {e3}")
-                            raise e3
-                else:
-                    # Final fallback to HuggingFace cache
-                    print(f"⚠️ Failed to download to local directory, using HF cache: {e}")
-                    try:
-                        local_path = hf_hub_download(repo_id=repo_id, filename=fpath)
-                        local_paths.append(local_path)
-                    except Exception as e2:
-                        print(f"❌ Failed to download {fpath}: {e2}")
-                        raise e2
+        # Check if local model exists first
+        model_dir = os.path.join(folder_paths.models_dir, "TTS", "chatterbox", language)
+        if os.path.exists(model_dir):
+            # Validate model completeness
+            is_complete, missing_files = validate_model_completeness(model_dir, language)
+            if is_complete:
+                print(f"📁 Using existing local model: {model_dir}")
+                return cls.from_local(model_dir, device, language)
+            else:
+                print(f"⚠️ Local model incomplete, missing: {missing_files}")
 
-        # Use the directory of the first downloaded file
-        model_dir = Path(local_paths[0]).parent
-        return cls.from_local(model_dir, device)
+        # Use new unified ChatterBox downloader
+        from utils.downloads.unified_downloader import unified_downloader
+        
+        # Get files to download based on model completeness
+        files_to_download = get_model_requirements(language)
+        
+        # Handle mixed format models (try safetensors first, fallback to pt)
+        if model_format == "mixed":
+            # Try safetensors first
+            files_to_download = [f.replace('.pt', '.safetensors') if f.endswith('.pt') else f for f in files_to_download]
+        elif model_format == "pt":
+            files_to_download = [f.replace('.safetensors', '.pt') if f.endswith('.safetensors') else f for f in files_to_download]
+        
+        # Download model using new method with subdirectory support
+        subdirectory = model_config.get("subdirectory")
+        downloaded_dir = unified_downloader.download_chatterbox_model(
+            repo_id=repo_id,
+            model_name=language,
+            subdirectory=subdirectory,
+            files=files_to_download
+        )
+        
+        if downloaded_dir:
+            print(f"✅ Downloaded ChatterBox model: {language}")
+            return cls.from_local(downloaded_dir, device, language)
+        else:
+            # Fallback to English if download fails
+            print(f"❌ Failed to download {language} model, falling back to English")
+            if language != "English":
+                return cls.from_pretrained(device, language="English")
+            else:
+                raise Exception("Failed to download English ChatterBox model")
 
     def prepare_conditionals(self, wav_fpath, exaggeration=0.5):
-        ## Load reference wav
-        s3gen_ref_wav, _sr = librosa.load(wav_fpath, sr=S3GEN_SR)
-
-        ref_16k_wav = librosa.resample(s3gen_ref_wav, orig_sr=S3GEN_SR, target_sr=S3_SR)
+        # Load reference wav using fallback for Python 3.13 compatibility
+        s3gen_ref_wav, sample_rate = safe_load(wav_fpath, sr=S3GEN_SR, mono=True)
+        
+        # Resample to 16k for S3 tokenizer using fallback
+        ref_16k_wav = safe_resample(s3gen_ref_wav, S3GEN_SR, S3_SR)
 
         s3gen_ref_wav = s3gen_ref_wav[:self.DEC_COND_LEN]
         s3gen_ref_dict = self.s3gen.embed_ref(s3gen_ref_wav, S3GEN_SR, device=self.device)
@@ -453,10 +653,12 @@ class ChatterboxTTS:
             )
             wav = wav.squeeze(0).detach().cpu().numpy()
             if self.enable_watermarking:
-                watermarked_wav = self.watermarker.apply_watermark(wav, sample_rate=self.sr)
-                return torch.from_numpy(watermarked_wav).unsqueeze(0)
-            else:
-                return torch.from_numpy(wav).unsqueeze(0)
+                self._init_watermarker_if_needed()
+                if self.watermarker is not None:
+                    watermarked_wav = self.watermarker.apply_watermark(wav, sample_rate=self.sr)
+                    return torch.from_numpy(watermarked_wav).unsqueeze(0)
+            
+            return torch.from_numpy(wav).unsqueeze(0)
     
     def generate_batch(
         self,

engines/chatterbox/vc.py

@@ -1,21 +1,31 @@
 from pathlib import Path
 import os
 
-import librosa
 import torch
 import warnings
 from huggingface_hub import hf_hub_download
 
+# Use librosa fallback for Python 3.13 compatibility
+from utils.audio.librosa_fallback import safe_load
+
 # Import folder_paths for model directory detection
 try:
     import folder_paths
 except ImportError:
     folder_paths = None
 
-# Import perth with warnings disabled
-with warnings.catch_warnings():
-    warnings.simplefilter("ignore")
-    import perth
+# Import perth with warnings disabled and graceful fallback
+try:
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore")
+        import perth
+    PERTH_AVAILABLE = True
+except ImportError:
+    perth = None
+    PERTH_AVAILABLE = False
+
+# Import safetensors for multilanguage model support
+from safetensors.torch import load_file
 
 from .models.s3tokenizer import S3_SR
 from .models.s3gen import S3GEN_SR, S3Gen
@@ -37,11 +47,11 @@ class ChatterboxVC:
         self.sr = S3GEN_SR
         self.s3gen = s3gen
         self.device = device
-        # Initialize watermarker silently (but disabled by default)
-        with warnings.catch_warnings():
-            warnings.simplefilter("ignore")
-            self.watermarker = perth.PerthImplicitWatermarker()
-        self.enable_watermarking = False  # Disabled by default for maximum compatibility
+        # Watermarking disabled by default for maximum compatibility
+        # Set to True to enable watermarking (requires perth library)
+        self.enable_watermarking = False
+        self.watermarker = None
+        self._watermarker_init_attempted = False
         if ref_dict is None:
             self.ref_dict = None
         else:
@@ -50,11 +60,64 @@ class ChatterboxVC:
                 for k, v in ref_dict.items()
             }
 
+    def to(self, device):
+        """
+        Move all model components to the specified device.
+
+        Critical for ComfyUI model management - ensures all components move together
+        when models are detached to CPU and later reloaded to CUDA.
+        """
+        self.device = device
+
+        # Move s3gen model
+        if hasattr(self.s3gen, 'to'):
+            self.s3gen = self.s3gen.to(device)
+
+        # Move reference dict tensors
+        if self.ref_dict is not None:
+            self.ref_dict = {
+                k: v.to(device) if torch.is_tensor(v) else v
+                for k, v in self.ref_dict.items()
+            }
+
+        return self
+
+    def _init_watermarker_if_needed(self):
+        """Initialize watermarker on first use if enabled"""
+        if self.enable_watermarking and not self._watermarker_init_attempted:
+            self._watermarker_init_attempted = True
+            try:
+                if PERTH_AVAILABLE and hasattr(perth, 'PerthImplicitWatermarker'):
+                    with warnings.catch_warnings():
+                        warnings.simplefilter("ignore")
+                        self.watermarker = perth.PerthImplicitWatermarker()
+                        if self.watermarker is None:
+                            raise ValueError("PerthImplicitWatermarker returned None")
+                else:
+                    raise AttributeError("PerthImplicitWatermarker not available in perth module")
+            except Exception as e:
+                print(f"❌ Failed to initialize watermarker: {e}")
+                self.watermarker = None
+                self.enable_watermarking = False
+
     @classmethod
     def from_local(cls, ckpt_dir, device) -> 'ChatterboxVC':
         print(f"📦 Loading local ChatterBox VC models from: {ckpt_dir}")
         ckpt_dir = Path(ckpt_dir)
         
+        # Auto-detect model format (same as TTS implementation)
+        def load_model_file(base_name: str):
+            """Load model file with auto-detection of format (.safetensors preferred over .pt)"""
+            safetensors_path = ckpt_dir / f"{base_name}.safetensors"
+            pt_path = ckpt_dir / f"{base_name}.pt"
+            
+            if safetensors_path.exists():
+                return load_file(safetensors_path, device=device)
+            elif pt_path.exists():
+                return torch.load(pt_path, map_location=device)
+            else:
+                raise FileNotFoundError(f"Neither {base_name}.safetensors nor {base_name}.pt found in {ckpt_dir}")
+        
         with warnings.catch_warnings():
             warnings.simplefilter("ignore")
             
@@ -64,9 +127,10 @@ class ChatterboxVC:
                 ref_dict = states['gen']
 
             s3gen = S3Gen()
-            s3gen.load_state_dict(
-                torch.load(ckpt_dir / "s3gen.pt")
-            )
+            # Load s3gen with auto-format detection
+            s3gen_state = load_model_file("s3gen")
+            # Apply JaneDoe84's critical fix: strict=False to handle missing keys (like Norwegian model)
+            s3gen.load_state_dict(s3gen_state, strict=False)
             s3gen.to(device).eval()
 
             instance = cls(s3gen, device, ref_dict=ref_dict)
@@ -85,6 +149,17 @@ class ChatterboxVC:
         Returns:
             ChatterboxVC model instance
         """
+        # Handle local models first - redirect to from_local
+        if language.startswith("local:"):
+            from .language_models import find_local_model_path
+            local_path = find_local_model_path(language)
+            if local_path and os.path.exists(local_path):
+                print(f"📁 Redirecting to local model at: {local_path}")
+                return cls.from_local(local_path, device)
+            else:
+                print(f"⚠️ Local model path not found for {language}, falling back to English")
+                language = "English"
+        
         # Import language model support
         try:
             from .language_models import get_model_config
@@ -109,11 +184,36 @@ class ChatterboxVC:
         
         print(f"📦 Loading ChatterBox VC model for {language} from {repo_id}")
         
-        # Download VC models to local directory
+        # Sanitize language name for path construction (strip "local:" prefix)
+        clean_language = language[6:] if language.startswith("local:") else language
+        
+        # Download VC models to local directory (use dynamic requirements like TTS)
+        try:
+            from .language_models import get_model_requirements, get_model_config
+            # Get VC-specific files (filter out TTS-only files) 
+            all_files = get_model_requirements(language)
+            vc_files = [f for f in all_files if any(f.startswith(component) for component in ["s3gen", "conds"])]
+            
+            # Separate required vs optional files (like TTS does)
+            required_files = [f for f in vc_files if f.startswith("s3gen")]  # s3gen is required
+            optional_files = [f for f in vc_files if f.startswith("conds")]   # conds is optional
+            
+            # Get subdirectory info like TTS does
+            model_config = get_model_config(language)
+            subdirectory = model_config.get("subdirectory") if model_config else None
+            
+        except ImportError:
+            # Fallback for older systems
+            required_files = ["s3gen.pt"]
+            optional_files = ["conds.pt"]
+            subdirectory = None
+        
         local_paths = []
-        for fpath in ["s3gen.pt", "conds.pt"]:
-            # Define local path in TTS organization
-            local_model_path = os.path.join(folder_paths.models_dir, "TTS", "chatterbox", language, fpath)
+        all_vc_files = required_files + optional_files
+        
+        for fpath in all_vc_files:
+            # Define local path in TTS organization (use clean language name)
+            local_model_path = os.path.join(folder_paths.models_dir, "TTS", "chatterbox", clean_language, fpath)
             
             if os.path.exists(local_model_path):
                 print(f"📁 Using local Chatterbox VC model: {local_model_path}")
@@ -126,7 +226,8 @@ class ChatterboxVC:
                 print(f"📁 Using cached Chatterbox VC model: {hf_cached_file}")
                 local_paths.append(hf_cached_file)
                 continue
-            except Exception:
+            except Exception as cache_error:
+                print(f"📋 Cache check for {fpath}: {str(cache_error)[:100]}... - will download")
                 pass
             
             # Download to local directory
@@ -134,17 +235,22 @@ class ChatterboxVC:
             os.makedirs(os.path.dirname(local_model_path), exist_ok=True)
             
             try:
-                # Use unified downloader
+                # Use unified downloader with subdirectory support
                 from utils.downloads.unified_downloader import unified_downloader
                 
+                # Build correct remote path with subdirectory
+                remote_path = fpath
+                if subdirectory:
+                    remote_path = f"{subdirectory}/{fpath}"
+                
                 files_to_download = [{
-                    'remote': fpath,
-                    'local': os.path.basename(fpath)
+                    'remote': remote_path,
+                    'local': os.path.basename(fpath)  # Save directly to language folder
                 }]
                 
                 downloaded_dir = unified_downloader.download_huggingface_model(
                     repo_id=repo_id,
-                    model_name=language,
+                    model_name=clean_language,
                     files=files_to_download,
                     engine_type="chatterbox"
                 )
@@ -154,20 +260,56 @@ class ChatterboxVC:
                     local_paths.append(local_model_path)
                     print(f"✅ Downloaded Chatterbox VC model to: {local_model_path}")
                 else:
-                    raise Exception("Unified download failed")
+                    # Check if this is an optional file before raising error
+                    is_optional = fpath in optional_files
+                    if is_optional:
+                        print(f"ℹ️ Optional file {fpath} not available in {language} model - continuing without it")
+                        continue
+                    else:
+                        raise Exception("Unified download failed")
             except Exception as e:
-                # Fallback to HuggingFace cache
+                # Check for 401 authorization errors - these should be propagated
+                error_str = str(e)
+                if "401" in error_str or "Unauthorized" in error_str:
+                    raise RuntimeError(f"401 Unauthorized: {language} model requires authentication - {e}")
+                
+                # Fallback to HuggingFace cache for other errors
                 print(f"⚠️ Failed to download to local directory, using HF cache: {e}")
-                local_path = hf_hub_download(repo_id=repo_id, filename=fpath)
-                local_paths.append(local_path)
+                try:
+                    local_path = hf_hub_download(repo_id=repo_id, filename=fpath)
+                    local_paths.append(local_path)
+                except Exception as hf_error:
+                    # Check HF cache error for 401 as well
+                    hf_error_str = str(hf_error)
+                    if "401" in hf_error_str or "Unauthorized" in hf_error_str:
+                        raise RuntimeError(f"401 Unauthorized: {language} model requires authentication - {hf_error}")
+                    else:
+                        # Handle missing files based on whether they're required or optional
+                        is_optional = fpath in optional_files
+                        if is_optional and ("404" in hf_error_str or "Not Found" in hf_error_str):
+                            print(f"ℹ️ Optional file {fpath} not available in {language} model - continuing without it")
+                            continue  # Skip optional files that are missing
+                        else:
+                            raise hf_error  # Re-raise errors for required files
 
+        # Log final source summary
+        sources = []
+        for path in local_paths:
+            if "models/TTS/chatterbox" in str(path):
+                sources.append("local")
+            elif "cache" in str(path).lower():
+                sources.append("cache") 
+            else:
+                sources.append("downloaded")
+        print(f"📦 Loading ChatterBox VC using: {', '.join(set(sources))} sources")
+        
         # Use the directory of the first downloaded file
         model_dir = Path(local_paths[0]).parent
         return cls.from_local(model_dir, device)
 
     def set_target_voice(self, wav_fpath):
-        ## Load reference wav
-        s3gen_ref_wav, _sr = librosa.load(wav_fpath, sr=S3GEN_SR)
+        # Load reference wav using fallback for Python 3.13 compatibility
+        s3gen_ref_wav, sample_rate = safe_load(wav_fpath, sr=S3GEN_SR, mono=True)
 
         s3gen_ref_wav = s3gen_ref_wav[:self.DEC_COND_LEN]
         self.ref_dict = self.s3gen.embed_ref(s3gen_ref_wav, S3GEN_SR, device=self.device)
@@ -183,8 +325,10 @@ class ChatterboxVC:
             assert self.ref_dict is not None, "Please `prepare_conditionals` first or specify `target_voice_path`"
 
         with torch.inference_mode():
-            audio_16, _ = librosa.load(audio, sr=S3_SR)
-            audio_16 = torch.from_numpy(audio_16).float().to(self.device)[None, ]
+            # Load audio using fallback for Python 3.13 compatibility
+            audio_16_wav, _ = safe_load(audio, sr=S3_SR, mono=True)
+            # Convert to the expected format
+            audio_16 = torch.from_numpy(audio_16_wav).float().to(self.device)[None, ]
 
             s3_tokens, _ = self.s3gen.tokenizer(audio_16)
             wav, _ = self.s3gen.inference(
@@ -192,7 +336,9 @@ class ChatterboxVC:
                 ref_dict=self.ref_dict,
             )
             wav = wav.squeeze(0).detach().cpu().numpy()
-            if self.enable_watermarking:
+            # Initialize watermarker lazily if needed
+            self._init_watermarker_if_needed()
+            if self.enable_watermarking and self.watermarker is not None:
                 watermarked_wav = self.watermarker.apply_watermark(wav, sample_rate=self.sr)
                 return torch.from_numpy(watermarked_wav).unsqueeze(0)
             else:

engines/chatterbox_official_23lang/__init__.py

@@ -0,0 +1,118 @@
+# Set up global warning filters
+import warnings
+
+# Filter out specific warning messages
+warnings.filterwarnings("ignore", message=".*LoRACompatibleLinear.*")
+warnings.filterwarnings("ignore", message=".*PerthNet.*")
+warnings.filterwarnings("ignore", message=".*requires authentication.*")
+
+# Import main TTS/VC modules with error handling
+try:
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore")
+        from .tts import ChatterboxOfficial23LangTTS
+    TTS_AVAILABLE = True
+except ImportError:
+    TTS_AVAILABLE = False
+    # Create dummy class
+    class ChatterboxOfficial23LangTTS:
+        @classmethod
+        def from_pretrained(cls, device):
+            raise ImportError("ChatterboxOfficial23LangTTS not available - missing dependencies")
+        @classmethod
+        def from_local(cls, path, device):
+            raise ImportError("ChatterboxOfficial23LangTTS not available - missing dependencies")
+
+try:
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore")
+        from .vc import ChatterboxOfficial23LangVC
+    VC_AVAILABLE = True
+except ImportError:
+    VC_AVAILABLE = False
+    # Create dummy class
+    class ChatterboxOfficial23LangVC:
+        @classmethod
+        def from_pretrained(cls, device):
+            raise ImportError("ChatterboxVC not available - missing dependencies")
+        @classmethod
+        def from_local(cls, path, device):
+            raise ImportError("ChatterboxVC not available - missing dependencies")
+
+# F5-TTS support modules - import independently
+try:
+    from .f5tts import ChatterBoxF5TTS, F5TTS_AVAILABLE
+    F5TTS_SUPPORT_AVAILABLE = F5TTS_AVAILABLE
+except ImportError:
+    F5TTS_SUPPORT_AVAILABLE = False
+    # Create dummy class
+    class ChatterBoxF5TTS:
+        @classmethod
+        def from_pretrained(cls, device, model_name):
+            raise ImportError("F5-TTS not available - missing dependencies")
+        @classmethod
+        def from_local(cls, path, device, model_name):
+            raise ImportError("F5-TTS not available - missing dependencies")
+
+# Language models support
+try:
+    from .language_models import (
+        get_chatterbox_models, get_model_config, get_model_files_for_language,
+        find_local_model_path, detect_model_format, get_available_languages,
+        is_model_incomplete, get_model_requirements, validate_model_completeness,
+        get_tokenizer_filename
+    )
+    LANGUAGE_MODELS_AVAILABLE = True
+except ImportError:
+    LANGUAGE_MODELS_AVAILABLE = False
+    # Create dummy functions for compatibility
+    def get_available_languages():
+        return ["English"]
+    def find_local_model_path(language):
+        return None
+    def get_chatterbox_models():
+        return ["English"]
+    def get_model_config(language):
+        return None
+    def get_model_files_for_language(language):
+        return ("pt", "ResembleAI/chatterbox")
+    def detect_model_format(model_path):
+        return "pt"
+    def is_model_incomplete(language):
+        return False
+    def get_model_requirements(language):
+        return []
+    def validate_model_completeness(model_path, language):
+        return True, []
+    def get_tokenizer_filename(language):
+        return "tokenizer.json"
+
+# SRT subtitle support modules - import independently
+try:
+    from .srt_parser import SRTParser, SRTSubtitle, SRTParseError, validate_srt_timing_compatibility
+    from .audio_timing import (
+        AudioTimingUtils, PhaseVocoderTimeStretcher, TimedAudioAssembler,
+        calculate_timing_adjustments, AudioTimingError
+    )
+    SRT_AVAILABLE = True
+    
+    __all__ = [
+        'ChatterboxOfficial23LangTTS', 'ChatterboxOfficial23LangVC',
+        'get_official_23lang_models', 'get_model_config', 'get_model_files_for_language',
+        'find_local_model_path', 'detect_model_format', 'get_available_languages',
+        'is_model_incomplete', 'get_model_requirements', 'validate_model_completeness',
+        'get_tokenizer_filename',
+        'SRTParser', 'SRTSubtitle', 'SRTParseError', 'validate_srt_timing_compatibility',
+        'AudioTimingUtils', 'PhaseVocoderTimeStretcher', 'TimedAudioAssembler',
+        'calculate_timing_adjustments', 'AudioTimingError'
+    ]
+except ImportError:
+    SRT_AVAILABLE = False
+    # SRT support not available - only export main modules and language functions
+    __all__ = [
+        'ChatterboxOfficial23LangTTS', 'ChatterboxOfficial23LangVC',
+        'get_official_23lang_models', 'get_model_config', 'get_model_files_for_language',
+        'find_local_model_path', 'detect_model_format', 'get_available_languages',
+        'is_model_incomplete', 'get_model_requirements', 'validate_model_completeness',
+        'get_tokenizer_filename'
+    ]

engines/chatterbox_official_23lang/adaptive_processor.py

@@ -0,0 +1,326 @@
+"""
+Adaptive Batch Processor for ChatterBox TTS
+
+Dynamically adjusts worker count based on real-time performance monitoring.
+Starts with user's batch_size (respecting their insight) and adapts from there.
+
+Optional feature - can be disabled to respect user's exact worker count preference.
+Modular design following CLAUDE.md guidelines - under 500 lines, focused responsibility.
+"""
+
+import torch
+import time
+from concurrent.futures import ThreadPoolExecutor, as_completed, Future
+from typing import List, Tuple, Dict, Optional, Deque
+from collections import deque
+from dataclasses import dataclass
+import threading
+
+
+@dataclass
+class PerformanceMetric:
+    """Single performance measurement."""
+    timestamp: float
+    worker_count: int
+    completion_time: float
+    tokens_per_second: float
+    total_throughput: float  # texts completed per second
+
+
+class AdaptivePerformanceMonitor:
+    """
+    Monitors real-time performance and suggests worker count adjustments.
+    """
+    
+    def __init__(self, initial_workers: int, adaptation_enabled: bool = True):
+        """
+        Initialize performance monitor.
+        
+        Args:
+            initial_workers: User's preferred starting worker count
+            adaptation_enabled: Whether to suggest adaptations
+        """
+        self.initial_workers = initial_workers
+        self.current_optimal_workers = initial_workers
+        self.adaptation_enabled = adaptation_enabled
+        
+        # Performance tracking
+        self.metrics: Deque[PerformanceMetric] = deque(maxlen=20)  # Last 20 measurements
+        self.worker_performance: Dict[int, List[float]] = {}  # worker_count -> [throughputs]
+        
+        # Adaptation logic
+        self.evaluation_interval = 3  # Evaluate every N completions
+        self.completions_since_eval = 0
+        self.last_adaptation_time = time.time()
+        self.min_adaptation_interval = 10.0  # Wait 10s between adaptations
+        
+        # Stability tracking
+        self.stable_performance_threshold = 0.1  # 10% variance
+        self.performance_declining_threshold = 0.15  # 15% decline triggers reduction
+        
+        print(f"🔧 Adaptive Monitor: Starting with {initial_workers} workers (user preference)")
+        print(f"🎛️ Dynamic adaptation: {'ENABLED' if adaptation_enabled else 'DISABLED'}")
+    
+    def record_completion(self, completion_time: float, tokens_per_second: float) -> Optional[int]:
+        """
+        Record a text completion and return suggested worker count change.
+        
+        Args:
+            completion_time: Time taken for this text (seconds)
+            tokens_per_second: Generation speed for this text
+            
+        Returns:
+            Suggested new worker count, or None if no change recommended
+        """
+        if not self.adaptation_enabled:
+            return None  # Respect user's exact worker count
+        
+        # Record this completion
+        current_time = time.time()
+        
+        # Calculate current throughput (approximate) - FIXED: prevent division by zero
+        recent_completions = len([m for m in self.metrics if current_time - m.timestamp < 10])
+        if self.metrics:
+            time_window = max(0.1, current_time - self.metrics[0].timestamp)  # Minimum 0.1s window
+            total_throughput = recent_completions / min(10, time_window)
+        else:
+            total_throughput = 0.0  # No data yet
+        
+        metric = PerformanceMetric(
+            timestamp=current_time,
+            worker_count=self.current_optimal_workers,
+            completion_time=completion_time,
+            tokens_per_second=tokens_per_second,
+            total_throughput=total_throughput
+        )
+        self.metrics.append(metric)
+        
+        # Track performance for this worker count
+        if self.current_optimal_workers not in self.worker_performance:
+            self.worker_performance[self.current_optimal_workers] = []
+        self.worker_performance[self.current_optimal_workers].append(total_throughput)
+        
+        # Check if it's time to evaluate
+        self.completions_since_eval += 1
+        if self.completions_since_eval >= self.evaluation_interval:
+            return self._evaluate_adaptation()
+        
+        return None
+    
+    def _evaluate_adaptation(self) -> Optional[int]:
+        """Evaluate whether to adapt worker count."""
+        self.completions_since_eval = 0
+        current_time = time.time()
+        
+        # Don't adapt too frequently
+        if current_time - self.last_adaptation_time < self.min_adaptation_interval:
+            return None
+        
+        if len(self.metrics) < 6:  # Need some data
+            return None
+        
+        # Get recent performance
+        recent_metrics = list(self.metrics)[-6:]  # Last 6 completions
+        current_avg_throughput = sum(m.total_throughput for m in recent_metrics) / len(recent_metrics)
+        current_avg_speed = sum(m.tokens_per_second for m in recent_metrics) / len(recent_metrics)
+        
+        # Compare with earlier performance for same worker count
+        worker_history = self.worker_performance.get(self.current_optimal_workers, [])
+        if len(worker_history) > 6:
+            earlier_avg = sum(worker_history[-12:-6]) / 6  # Earlier batch
+            current_avg = sum(worker_history[-6:]) / 6     # Current batch
+            
+            performance_change = (current_avg - earlier_avg) / earlier_avg
+            
+            if performance_change < -self.performance_declining_threshold:
+                # Performance declining significantly - reduce workers
+                new_workers = max(1, self.current_optimal_workers - 1)
+                print(f"🔽 ADAPTIVE: Performance declining ({performance_change:.1%}), reducing workers {self.current_optimal_workers} → {new_workers}")
+                return self._suggest_adaptation(new_workers)
+            
+            elif performance_change > -0.05 and current_avg_speed > 3.0:  # Stable + fast
+                # Performance stable and individual workers are fast - try more workers
+                # But don't go crazy - cap at 2x user's initial preference
+                max_workers = min(self.initial_workers * 2, 8)
+                if self.current_optimal_workers < max_workers:
+                    new_workers = self.current_optimal_workers + 1
+                    print(f"🔼 ADAPTIVE: Performance stable & fast ({current_avg_speed:.1f} it/s), trying more workers {self.current_optimal_workers} → {new_workers}")
+                    return self._suggest_adaptation(new_workers)
+        
+        return None
+    
+    def _suggest_adaptation(self, new_workers: int) -> int:
+        """Record adaptation suggestion."""
+        self.last_adaptation_time = time.time()
+        self.current_optimal_workers = new_workers
+        return new_workers
+    
+    def get_performance_summary(self) -> Dict:
+        """Get current performance summary."""
+        if not self.metrics:
+            return {"status": "No data yet"}
+        
+        recent = list(self.metrics)[-5:] if len(self.metrics) >= 5 else list(self.metrics)
+        avg_throughput = sum(m.total_throughput for m in recent) / len(recent)
+        avg_speed = sum(m.tokens_per_second for m in recent) / len(recent)
+        
+        return {
+            "current_workers": self.current_optimal_workers,
+            "initial_workers": self.initial_workers,
+            "avg_throughput": avg_throughput,
+            "avg_speed_per_worker": avg_speed,
+            "adaptation_enabled": self.adaptation_enabled,
+            "total_completions": len(self.metrics)
+        }
+
+
+class AdaptiveBatchProcessor:
+    """
+    Batch processor with optional dynamic worker adaptation.
+    
+    Starts with user's batch_size (respecting their insight) and optionally
+    adapts based on real-time performance monitoring.
+    """
+    
+    def __init__(self, tts_model, enable_adaptation: bool = True):
+        """
+        Initialize adaptive processor.
+        
+        Args:
+            tts_model: ChatterboxTTS instance
+            enable_adaptation: Whether to enable dynamic adaptation
+        """
+        self.tts_model = tts_model
+        self.enable_adaptation = enable_adaptation
+        self.current_executor: Optional[ThreadPoolExecutor] = None
+        self.monitor: Optional[AdaptivePerformanceMonitor] = None
+        self.adaptation_lock = threading.Lock()
+    
+    def process_texts_adaptively(
+        self, 
+        texts: List[str], 
+        temperature: float, 
+        cfg_weight: float, 
+        exaggeration: float,
+        user_batch_size: int
+    ) -> List[torch.Tensor]:
+        """
+        Process texts with optional adaptive worker management.
+        
+        Args:
+            texts: List of text strings
+            temperature: Sampling temperature
+            cfg_weight: CFG weight
+            exaggeration: Emotion exaggeration
+            user_batch_size: User's preferred batch size (starting point)
+            
+        Returns:
+            List of audio tensors in original order
+        """
+        # Initialize monitor with user's preference
+        self.monitor = AdaptivePerformanceMonitor(
+            initial_workers=user_batch_size,
+            adaptation_enabled=self.enable_adaptation
+        )
+        
+        effective_workers = min(user_batch_size, len(texts))
+        
+        if self.enable_adaptation:
+            print(f"🤖 ADAPTIVE PROCESSING: Starting with {effective_workers} workers (user preference), will adapt based on performance")
+        else:
+            print(f"🔒 FIXED PROCESSING: Using exactly {effective_workers} workers as requested")
+        
+        return self._process_with_adaptive_workers(
+            texts, temperature, cfg_weight, exaggeration, effective_workers
+        )
+    
+    def _process_with_adaptive_workers(
+        self, 
+        texts: List[str], 
+        temperature: float, 
+        cfg_weight: float, 
+        exaggeration: float,
+        initial_workers: int
+    ) -> List[torch.Tensor]:
+        """Process texts with adaptive worker management."""
+        
+        results = [None] * len(texts)
+        completed_indices = set()
+        current_workers = initial_workers
+        
+        def _process_single_text(text_index: int, text: str) -> Tuple[int, torch.Tensor, float, float]:
+            """Process single text and return timing metrics."""
+            start_time = time.time()
+            
+            try:
+                print(f"  🧵 Worker {text_index+1}: Processing: {text[:30]}...")
+                
+                # Track generation performance
+                gen_start = time.time()
+                audio = self.tts_model.generate(
+                    text=text,
+                    audio_prompt_path=None,
+                    exaggeration=exaggeration,
+                    cfg_weight=cfg_weight,
+                    temperature=temperature,
+                )
+                gen_time = time.time() - gen_start
+                
+                # Estimate tokens/second (rough approximation)
+                estimated_tokens = len(text) * 2  # Rough estimate
+                tokens_per_second = estimated_tokens / gen_time if gen_time > 0 else 0
+                
+                total_time = time.time() - start_time
+                print(f"  ✅ Worker {text_index+1}: Completed in {total_time:.1f}s ({tokens_per_second:.1f} tok/s)")
+                
+                return text_index, audio, total_time, tokens_per_second
+                
+            except Exception as e:
+                total_time = time.time() - start_time
+                print(f"  ❌ Worker {text_index+1}: Failed in {total_time:.1f}s - {e}")
+                return text_index, torch.zeros(1, 1000), total_time, 0.0
+        
+        # Process all texts
+        with ThreadPoolExecutor(max_workers=current_workers) as executor:
+            # Submit all tasks
+            futures = {
+                executor.submit(_process_single_text, i, text): i
+                for i, text in enumerate(texts)
+            }
+            
+            completed_count = 0
+            total_start_time = time.time()
+            
+            for future in as_completed(futures):
+                text_index, audio, completion_time, tokens_per_second = future.result()
+                results[text_index] = audio
+                completed_indices.add(text_index)
+                completed_count += 1
+                
+                # Record performance and get adaptation suggestion
+                if self.monitor:
+                    suggested_workers = self.monitor.record_completion(completion_time, tokens_per_second)
+                    
+                    # Note: For simplicity, we don't actually restart the executor mid-process
+                    # Real adaptation would need more complex future management
+                    # This logs what WOULD happen for user feedback
+                    if suggested_workers and suggested_workers != current_workers:
+                        print(f"📊 ADAPTATION SUGGESTION: Would adjust to {suggested_workers} workers (currently {current_workers})")
+                        # In a full implementation, you'd restart executor with new worker count
+                        # current_workers = suggested_workers
+                
+                progress_percent = int(100 * completed_count / len(texts))
+                print(f"  📊 Progress: {completed_count}/{len(texts)} completed ({progress_percent}%)")
+        
+        total_time = time.time() - total_start_time
+        avg_time = total_time / len(texts)
+        
+        # Performance summary
+        if self.monitor:
+            summary = self.monitor.get_performance_summary()
+            print(f"✅ ADAPTIVE PROCESSING COMPLETED: {len(results)} texts in {total_time:.1f}s")
+            print(f"📈 Performance: {summary['avg_throughput']:.2f} texts/sec, {summary['avg_speed_per_worker']:.1f} tok/s per worker")
+            if self.enable_adaptation:
+                print(f"🎛️ Final recommendation: Use {summary['current_workers']} workers for this workload")
+        
+        return results

engines/chatterbox_official_23lang/audio_timing.py

@@ -0,0 +1,674 @@
+"""
+Audio Timing Utilities for ChatterBox TTS SRT Support
+Provides time-stretching, silence padding, and sample-accurate timing conversion
+"""
+
+# Smart numba compatibility for audio timing
+from utils.compatibility import setup_numba_compatibility
+import sys
+import os
+setup_numba_compatibility(quick_startup=True, verbose=False)
+
+import torch
+import torchaudio
+import numpy as np
+from typing import Tuple, Optional, List, Union
+import librosa
+from scipy.signal import stft, istft
+import warnings
+import subprocess
+import tempfile
+import os
+import shutil
+import soundfile as sf
+
+
+class AudioTimingError(Exception):
+    """Exception raised when audio timing operations fail"""
+    pass
+
+
+class AudioTimingUtils:
+    """
+    Utilities for audio timing manipulation and synchronization
+    """
+    
+    @staticmethod
+    def seconds_to_samples(seconds: float, sample_rate: int) -> int:
+        """
+        Convert time in seconds to sample count
+        
+        Args:
+            seconds: Time in seconds
+            sample_rate: Audio sample rate
+            
+        Returns:
+            Number of samples (integer)
+        """
+        return int(seconds * sample_rate)
+    
+    @staticmethod
+    def samples_to_seconds(samples: int, sample_rate: int) -> float:
+        """
+        Convert sample count to time in seconds
+        
+        Args:
+            samples: Number of samples
+            sample_rate: Audio sample rate
+            
+        Returns:
+            Time in seconds
+        """
+        return samples / sample_rate
+    
+    @staticmethod
+    def get_audio_duration(audio: torch.Tensor, sample_rate: int) -> float:
+        """
+        Get duration of audio tensor in seconds
+        
+        Args:
+            audio: Audio tensor (1D or 2D)
+            sample_rate: Audio sample rate
+            
+        Returns:
+            Duration in seconds
+        """
+        if audio.dim() == 1:
+            return audio.size(0) / sample_rate
+        elif audio.dim() == 2:
+            return audio.size(-1) / sample_rate
+        else:
+            raise AudioTimingError(f"Unsupported audio tensor dimensions: {audio.dim()}")
+    
+    @staticmethod
+    def create_silence(duration_seconds: float, sample_rate: int, 
+                      channels: int = 1, device: Optional[torch.device] = None) -> torch.Tensor:
+        """
+        Create silence tensor of specified duration
+        
+        Args:
+            duration_seconds: Duration of silence in seconds
+            sample_rate: Audio sample rate
+            channels: Number of audio channels
+            device: Target device for tensor
+            
+        Returns:
+            Silence tensor of shape [channels, samples] or [samples] if channels=1
+        """
+        if duration_seconds < 0:
+            raise AudioTimingError(f"Duration cannot be negative: {duration_seconds}")
+        
+        num_samples = AudioTimingUtils.seconds_to_samples(duration_seconds, sample_rate)
+        
+        if channels == 1:
+            silence = torch.zeros(num_samples, device=device)
+        else:
+            silence = torch.zeros(channels, num_samples, device=device)
+        
+        return silence
+    
+    @staticmethod
+    def pad_audio_to_duration(audio: torch.Tensor, target_duration: float, 
+                            sample_rate: int, pad_mode: str = "end") -> torch.Tensor:
+        """
+        Pad audio to reach target duration
+        
+        Args:
+            audio: Input audio tensor
+            target_duration: Target duration in seconds
+            sample_rate: Audio sample rate
+            pad_mode: Where to add padding ("start", "end", "both")
+            
+        Returns:
+            Padded audio tensor
+        """
+        current_duration = AudioTimingUtils.get_audio_duration(audio, sample_rate)
+        
+        if current_duration >= target_duration:
+            return audio  # No padding needed
+        
+        padding_duration = target_duration - current_duration
+        padding_samples = AudioTimingUtils.seconds_to_samples(padding_duration, sample_rate)
+        
+        if audio.dim() == 1:
+            padding = torch.zeros(padding_samples, device=audio.device, dtype=audio.dtype)
+        else:
+            padding = torch.zeros(audio.size(0), padding_samples, device=audio.device, dtype=audio.dtype)
+        
+        if pad_mode == "start":
+            return torch.cat([padding, audio], dim=-1)
+        elif pad_mode == "end":
+            return torch.cat([audio, padding], dim=-1)
+        elif pad_mode == "both":
+            half_padding = padding_samples // 2
+            if audio.dim() == 1:
+                start_pad = torch.zeros(half_padding, device=audio.device, dtype=audio.dtype)
+                end_pad = torch.zeros(padding_samples - half_padding, device=audio.device, dtype=audio.dtype)
+            else:
+                start_pad = torch.zeros(audio.size(0), half_padding, device=audio.device, dtype=audio.dtype)
+                end_pad = torch.zeros(audio.size(0), padding_samples - half_padding, device=audio.device, dtype=audio.dtype)
+            return torch.cat([start_pad, audio, end_pad], dim=-1)
+        else:
+            raise AudioTimingError(f"Invalid pad_mode: {pad_mode}. Use 'start', 'end', or 'both'")
+
+
+class PhaseVocoderTimeStretcher:
+    """
+    Phase vocoder-based time stretching implementation
+    """
+    
+    def __init__(self, hop_length: int = 512, win_length: int = 2048):
+        """
+        Initialize phase vocoder
+        
+        Args:
+            hop_length: STFT hop length
+            win_length: STFT window length
+        """
+        self.hop_length = hop_length
+        self.win_length = win_length
+    
+    def time_stretch(self, audio: torch.Tensor, stretch_factor: float, 
+                    sample_rate: int) -> torch.Tensor:
+        """
+        Time-stretch audio using phase vocoder
+        
+        Args:
+            audio: Input audio tensor (1D or 2D)
+            stretch_factor: Time stretching factor (>1 = slower, <1 = faster)
+            sample_rate: Audio sample rate
+            
+        Returns:
+            Time-stretched audio tensor
+        """
+        if stretch_factor <= 0:
+            raise AudioTimingError(f"Stretch factor must be positive: {stretch_factor}")
+        
+        if abs(stretch_factor - 1.0) < 1e-6:
+            return audio  # No stretching needed
+        
+        # Handle different tensor dimensions
+        original_shape = audio.shape
+        if audio.dim() == 1:
+            audio = audio.unsqueeze(0)  # Add channel dimension
+        elif audio.dim() > 2:
+            raise AudioTimingError(f"Unsupported audio tensor dimensions: {audio.dim()}")
+        
+        stretched_channels = []
+        
+        for channel in range(audio.size(0)):
+            channel_audio = audio[channel].cpu().numpy()
+            
+            try:
+                # Use librosa for phase vocoder time stretching
+                stretched = librosa.effects.time_stretch(
+                    channel_audio, 
+                    rate=1.0/stretch_factor,  # librosa uses rate = 1/stretch_factor
+                    hop_length=self.hop_length
+                )
+                stretched_channels.append(torch.from_numpy(stretched))
+                
+            except Exception as e:
+                # Fallback to simple resampling if phase vocoder fails
+                warnings.warn(f"Phase vocoder failed, using simple resampling: {e}")
+                stretched = self._simple_time_stretch(channel_audio, stretch_factor)
+                stretched_channels.append(torch.from_numpy(stretched))
+        
+        # Combine channels
+        result = torch.stack(stretched_channels, dim=0).to(audio.device)
+        
+        # Restore original shape if input was 1D
+        if len(original_shape) == 1:
+            result = result.squeeze(0)
+        
+        return result
+    
+    def _simple_time_stretch(self, audio: np.ndarray, stretch_factor: float) -> np.ndarray:
+        """
+        Simple time stretching using interpolation (fallback method)
+        
+        Args:
+            audio: Input audio array
+            stretch_factor: Time stretching factor
+            
+        Returns:
+            Time-stretched audio array
+        """
+        original_length = len(audio)
+        new_length = int(original_length * stretch_factor)
+        
+        # Create new time indices
+        old_indices = np.linspace(0, original_length - 1, original_length)
+        new_indices = np.linspace(0, original_length - 1, new_length)
+        
+        # Interpolate
+        stretched = np.interp(new_indices, old_indices, audio)
+        
+        return stretched
+
+
+class FFmpegTimeStretcher:
+    """FFmpeg time stretching using atempo filter"""
+    
+    def __init__(self):
+        """Verify FFmpeg is available"""
+        try:
+            result = subprocess.run(['ffmpeg', '-version'], capture_output=True)
+            if result.returncode != 0:
+                raise AudioTimingError("FFmpeg check failed")
+        except Exception as e:
+            raise AudioTimingError(f"FFmpeg not found: {str(e)}")
+    def _build_filter_chain(self, stretch_factor: float) -> str:
+        """Build safe FFmpeg filter chain for any stretch factor"""
+        if stretch_factor <= 0:
+            raise AudioTimingError("Invalid stretch factor")
+            
+        speed = 1/stretch_factor
+        
+        # Direct speed if within safe range
+        if 0.5 <= speed <= 2.0:
+            return f"aresample=async=0,atempo={speed:0.6f}"
+            
+        # Chain filters for extreme speeds
+        filters = []
+        remaining = speed
+        
+        while remaining < 0.5:
+            filters.append('atempo=0.5')  # Max slowdown
+            remaining *= 2
+            if len(filters) > 4:  # Safety limit
+                raise AudioTimingError("Too extreme slowdown")
+                
+        while remaining > 2.0:
+            filters.append('atempo=2.0')  # Max speedup
+            remaining /= 2
+            if len(filters) > 4:
+                raise AudioTimingError("Too extreme speedup")
+                
+        # Final adjustment
+        if abs(remaining - 1.0) > 1e-6:
+            filters.append(f'atempo={remaining:0.6f}')
+            
+        filter_str = f"aresample=async=0,{','.join(filters)}"
+        return filter_str
+        
+    def time_stretch(self, audio: torch.Tensor, stretch_factor: float, sample_rate: int) -> torch.Tensor:
+        """Time stretch audio using FFmpeg with safe filter chaining"""
+        # Full validation
+        if not isinstance(audio, torch.Tensor):
+            raise AudioTimingError("Input must be a tensor")
+            
+        if not isinstance(stretch_factor, (int, float)) or stretch_factor <= 0:
+            raise AudioTimingError(f"Invalid stretch factor: {stretch_factor}")
+            
+        if abs(stretch_factor - 1.0) < 1e-6:
+            return audio
+            
+        try:
+            # Get filter chain
+            filter_str = self._build_filter_chain(stretch_factor)
+            
+            # Shape handling
+            original_shape = audio.shape
+            if audio.dim() == 1:
+                audio = audio.unsqueeze(0)
+            elif audio.dim() > 2:
+                raise AudioTimingError("Only mono/stereo supported")
+                
+            # Process channels
+            stretched = []
+            with tempfile.TemporaryDirectory() as temp_dir:
+                
+                for i in range(audio.shape[0]):
+                    # Setup paths
+                    in_path = os.path.join(temp_dir, f'in_{i}.wav')
+                    out_path = os.path.join(temp_dir, f'out_{i}.wav')
+                    
+                    try:
+                        # Save input
+                        data = audio[i].cpu().numpy()
+                        sf.write(in_path, data, sample_rate,
+                                format='WAV', subtype='FLOAT')
+                        
+                        # Process with FFmpeg
+                        cmd = [
+                            'ffmpeg',
+                            '-nostdin',      # No interaction
+                            '-y',            # Overwrite output
+                            '-hide_banner',  # Less noise
+                            '-i', in_path,   # Input
+                            '-filter:a', filter_str,  # Audio filter
+                            '-c:a', 'pcm_f32le',  # Output format
+                            '-ar', str(sample_rate),
+                            '-ac', '1',      # Force mono
+                            '-v', 'error',   # Only errors (less verbose)
+                            out_path
+                        ]
+                        
+                        result = subprocess.run(cmd, capture_output=True, text=True)
+                        
+                        if result.returncode != 0:
+                            raise AudioTimingError(f"FFmpeg processing failed: {result.stderr}")
+                            
+                        if not os.path.exists(out_path) or os.path.getsize(out_path) == 0:
+                            raise AudioTimingError("No output produced")
+                            
+                        # Load and verify
+                        audio_data, sr = sf.read(out_path)
+                        if sr != sample_rate:
+                            raise AudioTimingError(f"Sample rate mismatch: {sr}")
+                            
+                        if len(audio_data) == 0:
+                            raise AudioTimingError("Empty output")
+                            
+                        stretched.append(torch.from_numpy(audio_data))
+                        
+                    except Exception as e:
+                        raise AudioTimingError(f"Channel {i} failed: {str(e)}")
+            
+            # Combine results
+            if not stretched:
+                raise AudioTimingError("No audio was processed successfully")
+                
+            result = torch.stack(stretched, dim=0).to(audio.device)
+            return result.squeeze(0) if len(original_shape) == 1 else result
+            
+        except Exception as e:
+            raise AudioTimingError(f"Time stretching failed: {str(e)}")
+                    
+                    # Run FFmpeg with checks
+    def _process_audio_file(self, in_path: str, out_path: str, filter_str: str,
+                          sample_rate: int) -> np.ndarray:
+        """Process a single audio file through FFmpeg"""
+        cmd = [
+            'ffmpeg',
+            '-hide_banner',   # Reduce noise
+            '-nostdin',       # No interactive mode
+            '-y',            # Overwrite output
+            '-i', in_path,
+            '-filter:a', filter_str,
+            '-acodec', 'pcm_f32le',
+            '-ar', str(sample_rate),
+            '-ac', '1',
+            '-v', 'info',     # Informative output level
+            out_path
+        ]
+        
+        try:
+            result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
+            
+            if result.returncode != 0:
+                raise AudioTimingError(f"FFmpeg failed: {result.stderr}")
+                
+            # Verify output
+            if not os.path.exists(out_path):
+                raise AudioTimingError("No output file produced")
+                
+            out_size = os.path.getsize(out_path)
+            if out_size == 0:
+                raise AudioTimingError("Empty output file")
+                
+            # Load and validate
+            audio_data, sr = sf.read(out_path)
+            
+            if sr != sample_rate:
+                raise AudioTimingError(f"Sample rate mismatch: got {sr}, expected {sample_rate}")
+                
+            if len(audio_data) == 0:
+                raise AudioTimingError("No audio data in output")
+                
+            return audio_data
+            
+        except subprocess.TimeoutExpired:
+            raise AudioTimingError("FFmpeg process timed out")
+        except Exception as e:
+            raise AudioTimingError(f"FFmpeg processing failed: {str(e)}")
+
+        if not stretched:
+            raise AudioTimingError("No channels were processed successfully")
+            
+        try:
+            # Stack channels and restore shape
+            result = torch.stack(stretched, dim=0).to(audio.device)
+            print(f"Successfully processed all channels")
+            return result.squeeze(0) if len(original_shape) == 1 else result
+            
+        except Exception as e:
+            raise AudioTimingError(f"Failed to combine channels: {str(e)}")
+    
+    def _build_atempo_chain(self, stretch_factor: float) -> str:
+        """Simple atempo command that stays within safe ranges"""
+        if stretch_factor <= 0:
+            raise AudioTimingError("Invalid stretch factor")
+            
+        speed = 1/stretch_factor
+        if not (0.5 <= speed <= 2.0):
+            raise AudioTimingError("Speed factor out of supported range")
+            
+        return f'atempo={speed:0.6f}'
+
+
+class TimedAudioAssembler:
+    """
+    Assembles audio segments with precise timing control
+    """
+    
+    def __init__(self, sample_rate: int, stretcher_type: str = "ffmpeg",
+                 time_stretcher: Optional[Union[PhaseVocoderTimeStretcher, FFmpegTimeStretcher]] = None):
+        """
+        Initialize audio assembler
+        
+        Args:
+            sample_rate: Target sample rate for output
+            stretcher_type: Type of time stretcher to use ("ffmpeg" or "phase_vocoder")
+            time_stretcher: Custom time stretching utility (creates default if None)
+        """
+        self.sample_rate = sample_rate
+        self.stretch_method_used = None  # Track which stretching method was used
+        
+        if time_stretcher is not None:
+            if not isinstance(time_stretcher, (PhaseVocoderTimeStretcher, FFmpegTimeStretcher)):
+                raise AudioTimingError("time_stretcher must be PhaseVocoderTimeStretcher or FFmpegTimeStretcher")
+            self.time_stretcher = time_stretcher
+        else:
+            if stretcher_type == "ffmpeg":
+                try:
+                    self.time_stretcher = FFmpegTimeStretcher()
+                except AudioTimingError as e:
+                    # Fall back to phase vocoder silently
+                    self.time_stretcher = PhaseVocoderTimeStretcher()
+            elif stretcher_type == "phase_vocoder":
+                self.time_stretcher = PhaseVocoderTimeStretcher()
+            else:
+                raise AudioTimingError(f"Invalid stretcher_type: {stretcher_type}. Use 'ffmpeg' or 'phase_vocoder'")
+    
+    def assemble_timed_audio(self, audio_segments: List[torch.Tensor], 
+                           target_timings: List[Tuple[float, float]],
+                           total_duration: Optional[float] = None,
+                           fade_duration: float = 0.01) -> torch.Tensor:
+        """
+        Assemble audio segments with precise timing
+        
+        Args:
+            audio_segments: List of audio tensors to assemble
+            target_timings: List of (start_time, end_time) tuples in seconds
+            total_duration: Total duration of output (auto-calculated if None)
+            fade_duration: Crossfade duration in seconds for overlaps
+            
+        Returns:
+            Assembled audio tensor
+        """
+        if len(audio_segments) != len(target_timings):
+            raise AudioTimingError(
+                f"Number of audio segments ({len(audio_segments)}) must match "
+                f"number of timings ({len(target_timings)})"
+            )
+        
+        if not audio_segments:
+            raise AudioTimingError("No audio segments provided")
+        
+        # Calculate total duration if not provided
+        if total_duration is None:
+            total_duration = max(end_time for _, end_time in target_timings)
+        
+        # Create output buffer
+        total_samples = AudioTimingUtils.seconds_to_samples(total_duration, self.sample_rate)
+        
+        # Determine output shape based on first segment
+        first_segment = audio_segments[0]
+        if first_segment.dim() == 1:
+            output = torch.zeros(total_samples, device=first_segment.device, dtype=first_segment.dtype)
+        else:
+            output = torch.zeros(first_segment.size(0), total_samples, 
+                               device=first_segment.device, dtype=first_segment.dtype)
+        
+        # Process each segment
+        for i, (audio_segment, (start_time, end_time)) in enumerate(zip(audio_segments, target_timings)):
+            if start_time < 0 or end_time <= start_time:
+                raise AudioTimingError(f"Invalid timing for segment {i}: {start_time} -> {end_time}")
+            
+            target_duration = end_time - start_time
+            current_duration = AudioTimingUtils.get_audio_duration(audio_segment, self.sample_rate)
+            
+            # If input audio is empty but target duration is positive, create silence.
+            if current_duration == 0.0 and target_duration > 0.0:
+                num_output_channels = first_segment.size(0) if first_segment.dim() > 1 else 1
+                audio_segment = AudioTimingUtils.create_silence(
+                    target_duration,
+                    self.sample_rate,
+                    channels=num_output_channels,
+                    device=first_segment.device, # Use device/dtype consistent with output buffer
+                    dtype=first_segment.dtype
+                )
+                current_duration = target_duration # Update duration, no stretching needed for this segment
+                self.stretch_method_used = "silence_created"
+            
+            # Time-stretch if needed
+            if abs(current_duration - target_duration) > 0.01:  # 10ms tolerance
+                if current_duration == 0.0: # Should only be true if target_duration is also ~0.0
+                    stretch_factor = 1.0 # Avoid division by zero
+                else:
+                    stretch_factor = target_duration / current_duration
+                
+                # Determine and track stretching method
+                if isinstance(self.time_stretcher, FFmpegTimeStretcher):
+                    self.stretch_method_used = "ffmpeg"
+                elif isinstance(self.time_stretcher, PhaseVocoderTimeStretcher):
+                    self.stretch_method_used = "phase_vocoder"
+                
+                audio_segment = self.time_stretcher.time_stretch(
+                    audio_segment, stretch_factor, self.sample_rate
+                )
+            else:
+                # No stretching needed
+                self.stretch_method_used = "none"
+            
+            # Calculate sample positions
+            start_sample = AudioTimingUtils.seconds_to_samples(start_time, self.sample_rate)
+            end_sample = AudioTimingUtils.seconds_to_samples(end_time, self.sample_rate)
+            
+            # Ensure segment fits exactly
+            target_samples = end_sample - start_sample
+            current_samples = audio_segment.size(-1)
+            
+            if current_samples != target_samples:
+                # Fine-tune length to match exactly
+                if current_samples > target_samples:
+                    audio_segment = audio_segment[..., :target_samples]
+                else:
+                    padding_needed = target_samples - current_samples
+                    if audio_segment.dim() == 1:
+                        padding = torch.zeros(padding_needed, device=audio_segment.device, dtype=audio_segment.dtype)
+                    else:
+                        padding = torch.zeros(audio_segment.size(0), padding_needed, 
+                                            device=audio_segment.device, dtype=audio_segment.dtype)
+                    audio_segment = torch.cat([audio_segment, padding], dim=-1)
+            
+            # Place segment in output buffer with crossfading for overlaps
+            self._place_segment_with_fade(output, audio_segment, start_sample, end_sample, fade_duration)
+        
+        return output
+    
+    def _place_segment_with_fade(self, output: torch.Tensor, segment: torch.Tensor,
+                               start_sample: int, end_sample: int, fade_duration: float):
+        """
+        Place audio segment in output buffer with crossfading for overlaps
+        """
+        segment_length = segment.size(-1)
+        fade_samples = min(
+            AudioTimingUtils.seconds_to_samples(fade_duration, self.sample_rate),
+            segment_length // 4  # Don't fade more than 25% of segment
+        )
+        
+        # Check for overlap with existing content
+        if output.dim() == 1:
+            existing_content = output[start_sample:end_sample]
+            has_overlap = torch.any(existing_content != 0)
+        else:
+            existing_content = output[:, start_sample:end_sample]
+            has_overlap = torch.any(existing_content != 0)
+        
+        if has_overlap and fade_samples > 0:
+            # Apply crossfade
+            fade_in = torch.linspace(0, 1, fade_samples, device=segment.device)
+            fade_out = torch.linspace(1, 0, fade_samples, device=segment.device)
+            
+            # Fade in the beginning of new segment
+            if segment.dim() == 1:
+                segment[:fade_samples] *= fade_in
+                # Fade out existing content at the beginning
+                output[start_sample:start_sample + fade_samples] *= fade_out
+            else:
+                segment[:, :fade_samples] *= fade_in.unsqueeze(0)
+                output[:, start_sample:start_sample + fade_samples] *= fade_out.unsqueeze(0)
+            
+            # Fade out the end of new segment if there's content after
+            end_check_start = min(end_sample, output.size(-1) - fade_samples)
+            end_check_end = min(end_sample + fade_samples, output.size(-1))
+            
+            if output.dim() == 1:
+                if end_check_start < end_check_end and torch.any(output[end_check_start:end_check_end] != 0):
+                    segment[-fade_samples:] *= fade_out
+            else:
+                if end_check_start < end_check_end and torch.any(output[:, end_check_start:end_check_end] != 0):
+                    segment[:, -fade_samples:] *= fade_out.unsqueeze(0)
+        
+        # Add segment to output
+        if output.dim() == 1:
+            output[start_sample:end_sample] += segment
+        else:
+            output[:, start_sample:end_sample] += segment
+
+
+def calculate_timing_adjustments(natural_durations: List[float], 
+                               target_timings: List[Tuple[float, float]]) -> List[dict]:
+    """
+    Calculate timing adjustments needed for each audio segment
+    
+    Args:
+        natural_durations: Natural durations of TTS-generated segments
+        target_timings: Target (start_time, end_time) tuples from SRT
+        
+    Returns:
+        List of adjustment dictionaries with timing information
+    """
+    adjustments = []
+    
+    for i, (natural_duration, (start_time, end_time)) in enumerate(zip(natural_durations, target_timings)):
+        target_duration = end_time - start_time
+        stretch_factor = target_duration / natural_duration if natural_duration > 0 else 1.0
+        
+        adjustment = {
+            'segment_index': i,
+            'natural_duration': natural_duration,
+            'target_duration': target_duration,
+            'start_time': start_time,
+            'end_time': end_time,
+            'stretch_factor': stretch_factor,
+            'needs_stretching': abs(stretch_factor - 1.0) > 0.05,  # 5% tolerance
+            'stretch_type': 'compress' if stretch_factor < 1.0 else 'expand' if stretch_factor > 1.0 else 'none'
+        }
+        
+        adjustments.append(adjustment)
+    
+    return adjustments

engines/chatterbox_official_23lang/batch_processor.py

@@ -0,0 +1,187 @@
+"""
+Batch Processor Module
+
+Handles actual batch processing of character groups.
+Works with character_grouper.py to process multiple segments simultaneously.
+"""
+
+from typing import Dict, List, Any
+import torch
+from .character_grouper import CharacterGroup
+
+
+class BatchProcessor:
+    """
+    Processes character groups using batch inference.
+    
+    This is where the actual batching magic happens - instead of:
+    1. Process segment 3 (crestfallen_original)
+    2. Process segment 5 (crestfallen_original)
+    
+    We do:
+    1. Batch process segments [3, 5] for crestfallen_original simultaneously
+    """
+    
+    def __init__(self, tts_model):
+        self.tts_model = tts_model
+    
+    def process_character_group_batch(
+        self,
+        character_group: CharacterGroup,
+        inputs: Dict[str, Any],
+        voice_refs: Dict[str, str],
+        chunker,
+        language: str
+    ) -> Dict[int, torch.Tensor]:
+        """
+        Process all segments for a character using TRUE batch processing.
+        
+        Args:
+            character_group: Group of segments for same character
+            inputs: Node inputs (exaggeration, temperature, etc.)
+            voice_refs: Character voice reference mapping
+            chunker: Text chunking utility
+            language: Language code for logging
+        
+        Returns:
+            Dict mapping original_idx -> generated_audio_tensor
+        """
+        character = character_group.character
+        segments = character_group.segments
+        
+        print(f"🚀 BATCH PROCESSING: {character} - {len(segments)} segments in {language}")
+        
+        # Collect all texts for batching
+        batch_texts = []
+        segment_chunk_counts = []  # Track how many chunks each segment produces
+        
+        for segment in segments:
+            # Apply chunking if needed
+            if inputs["enable_chunking"] and len(segment.segment_text) > inputs["max_chars_per_chunk"]:
+                chunks = chunker.split_into_chunks(segment.segment_text, inputs["max_chars_per_chunk"])
+            else:
+                chunks = [segment.segment_text]
+            
+            # Process each chunk (apply crash protection, etc.)
+            processed_chunks = []
+            for chunk_text in chunks:
+                # processed_chunk = self._apply_crash_protection(chunk_text, inputs.get("crash_protection_template", "hmm ,, {seg} hmm ,,"))  # DISABLED FOR TESTING
+                processed_chunk = chunk_text  # Direct text without crash protection
+                processed_chunks.append(processed_chunk)
+            
+            batch_texts.extend(processed_chunks)
+            segment_chunk_counts.append(len(processed_chunks))
+        
+        # Get voice reference for this character
+        char_audio_prompt = voice_refs[character]
+        
+        # THE ACTUAL BATCH PROCESSING - this is the key improvement
+        print(f"⚡ Batch generating {len(batch_texts)} chunks for {character}")
+        print(f"🔧 DEBUG: batch_size from inputs = {inputs.get('batch_size', 4)}")
+        try:
+            batch_audio = self.tts_model.generate_batch(
+                texts=batch_texts,
+                audio_prompt_path=char_audio_prompt,
+                exaggeration=inputs["exaggeration"],
+                cfg_weight=inputs["cfg_weight"],
+                temperature=inputs["temperature"],
+                batch_size=inputs.get("batch_size", 4),
+                max_workers=inputs.get("batch_size", 4),  # Use batch_size as max_workers  
+                enable_adaptive_batching=(inputs.get("batch_size", 1) > 1)  # Auto-enable based on batch_size
+            )
+            
+            print(f"✅ BATCH SUCCESS: Generated {len(batch_audio)} audio chunks")
+            
+        except Exception as e:
+            print(f"❌ BATCH FAILED: {e}")
+            raise  # Let calling code handle fallback
+        
+        # Map batch results back to original segments
+        results = {}
+        batch_idx = 0
+        
+        for i, segment in enumerate(segments):
+            chunk_count = segment_chunk_counts[i]
+            
+            # Collect all chunks for this segment
+            segment_chunks = []
+            for _ in range(chunk_count):
+                segment_chunks.append(batch_audio[batch_idx])
+                batch_idx += 1
+            
+            # Combine chunks if multiple
+            if len(segment_chunks) == 1:
+                segment_audio = segment_chunks[0]
+            else:
+                segment_audio = torch.cat(segment_chunks, dim=-1)
+            
+            results[segment.original_idx] = segment_audio
+        
+        return results
+    
+    def process_character_group_sequential(
+        self,
+        character_group: CharacterGroup,
+        inputs: Dict[str, Any],
+        voice_refs: Dict[str, str],
+        required_language: str,
+        total_segments: int,
+        language: str,
+        stable_audio_component: str,
+        chunker,
+        generate_tts_method
+    ) -> Dict[int, torch.Tensor]:
+        """
+        Fallback: Process character group sequentially.
+        Used when batch processing fails or for single segments.
+        """
+        character = character_group.character
+        segments = character_group.segments
+        
+        print(f"→ SEQUENTIAL: {character} - {len(segments)} segments in {language}")
+        
+        results = {}
+        char_audio_prompt = voice_refs[character]
+        
+        for segment in segments:
+            segment_display_idx = segment.original_idx + 1  # 1-based for display
+            
+            print(f"🎤 Generating segment {segment_display_idx}/{total_segments} for '{character}' (lang: {language})")
+            
+            # Apply chunking
+            if inputs["enable_chunking"] and len(segment.segment_text) > inputs["max_chars_per_chunk"]:
+                chunks = chunker.split_into_chunks(segment.segment_text, inputs["max_chars_per_chunk"])
+            else:
+                chunks = [segment.segment_text]
+            
+            # Process each chunk sequentially
+            segment_audio_chunks = []
+            for chunk_i, chunk_text in enumerate(chunks):
+                if len(chunks) > 1:
+                    print(f"  → Chunk {chunk_i+1}/{len(chunks)}")
+                
+                chunk_audio = generate_tts_method(
+                    chunk_text, char_audio_prompt, inputs["exaggeration"],
+                    inputs["temperature"], inputs["cfg_weight"], required_language,
+                    True, character=character, seed=inputs["seed"],
+                    enable_cache=inputs.get("enable_audio_cache", True),
+                    crash_protection_template=inputs.get("crash_protection_template", "hmm ,, {seg} hmm ,,"),
+                    stable_audio_component=stable_audio_component
+                )
+                segment_audio_chunks.append(chunk_audio)
+            
+            # Combine chunks for this segment
+            if len(segment_audio_chunks) == 1:
+                segment_audio = segment_audio_chunks[0]
+            else:
+                segment_audio = torch.cat(segment_audio_chunks, dim=-1)
+            
+            results[segment.original_idx] = segment_audio
+        
+        return results
+    
+    def _apply_crash_protection(self, text: str, padding_template: str, min_length: int = 15) -> str:
+        """Apply crash protection padding to short texts."""
+        if len(text.strip()) < min_length:
+            return padding_template.format(seg=text)
+        return text

engines/chatterbox_official_23lang/character_grouper.py

@@ -0,0 +1,139 @@
+"""
+Character Grouper Module
+
+Groups segments by character WITHIN each language group.
+Integrates with existing language grouping system - does not replace it.
+
+Fixes the issue where segments like:
+- Segment 3: crestfallen_original  
+- Segment 5: crestfallen_original
+Were processed separately instead of being batched together.
+"""
+
+from typing import List, Dict, Tuple
+from dataclasses import dataclass
+
+
+@dataclass
+class CharacterSegment:
+    """Represents a single segment for a specific character within a language."""
+    original_idx: int
+    character: str
+    segment_text: str
+    segment_lang: str
+    
+    def __str__(self):
+        return f"Segment({self.original_idx+1}: {self.character})"
+
+
+class CharacterGroup:
+    """A group of segments for the same character within a language."""
+    
+    def __init__(self, character: str):
+        self.character = character
+        self.segments: List[CharacterSegment] = []
+    
+    def add_segment(self, segment: CharacterSegment):
+        """Add a segment to this character group."""
+        self.segments.append(segment)
+    
+    def can_batch(self) -> bool:
+        """Check if this group has enough segments to warrant batch processing."""
+        return len(self.segments) > 1
+    
+    def get_segment_data(self) -> List[Tuple[int, str, str, str]]:
+        """Get segment data in the format expected by existing code: (original_idx, character, segment_text, segment_lang)"""
+        return [(seg.original_idx, seg.character, seg.segment_text, seg.segment_lang) for seg in self.segments]
+    
+    def __len__(self):
+        return len(self.segments)
+    
+    def __str__(self):
+        indices = [str(seg.original_idx + 1) for seg in self.segments]  # 1-based for display
+        batch_indicator = "🚀" if self.can_batch() else "→"
+        return f"{batch_indicator} {self.character}({len(self.segments)}): #{','.join(indices)}"
+
+
+class CharacterGrouper:
+    """
+    Groups segments by character WITHIN a language group.
+    
+    This works with the existing language grouping system:
+    
+    EXISTING: language_groups[lang] = [(idx, char, text, lang), ...]
+    NEW:      character_groups = group_by_character(lang_segments)
+    
+    So we go from:
+    lang_segments = [(1,'narrator',...), (2,'david',...), (3,'narrator',...)]
+    
+    To:
+    character_groups = {
+        'narrator': CharacterGroup([segment1, segment3]),
+        'david': CharacterGroup([segment2])
+    }
+    """
+    
+    @staticmethod
+    def group_by_character(lang_segments: List[Tuple[int, str, str, str]]) -> Dict[str, CharacterGroup]:
+        """
+        Group segments from a language by character.
+        
+        Args:
+            lang_segments: List of (original_idx, character, segment_text, segment_lang)
+                          This comes from the existing language grouping system
+        
+        Returns:
+            Dict mapping character_name -> CharacterGroup
+        """
+        character_groups = {}
+        
+        for original_idx, character, segment_text, segment_lang in lang_segments:
+            if character not in character_groups:
+                character_groups[character] = CharacterGroup(character)
+            
+            segment = CharacterSegment(
+                original_idx=original_idx,
+                character=character,
+                segment_text=segment_text,
+                segment_lang=segment_lang
+            )
+            
+            character_groups[character].add_segment(segment)
+        
+        return character_groups
+    
+    @staticmethod
+    def print_character_grouping_summary(language: str, character_groups: Dict[str, CharacterGroup]):
+        """Print summary of character grouping for a language."""
+        total_segments = sum(len(group) for group in character_groups.values())
+        batchable_segments = sum(len(group) for group in character_groups.values() if group.can_batch())
+        
+        group_summaries = [str(group) for group in character_groups.values()]
+        
+        print(f"🎭 {language} CHARACTER GROUPS ({batchable_segments}/{total_segments} batchable):")
+        print(f"   {' | '.join(group_summaries)}")
+    
+    @staticmethod
+    def validate_character_grouping(original_lang_segments: List[Tuple[int, str, str, str]], 
+                                   character_groups: Dict[str, CharacterGroup]):
+        """Validate that character grouping preserves all segments."""
+        original_count = len(original_lang_segments)
+        grouped_count = sum(len(group) for group in character_groups.values())
+        
+        if original_count != grouped_count:
+            raise RuntimeError(f"Character grouping lost segments: {original_count} -> {grouped_count}")
+        
+        # Validate all original indices are preserved
+        original_indices = {idx for idx, _, _, _ in original_lang_segments}
+        grouped_indices = set()
+        
+        for group in character_groups.values():
+            for seg in group.segments:
+                grouped_indices.add(seg.original_idx)
+        
+        if original_indices != grouped_indices:
+            missing = original_indices - grouped_indices
+            extra = grouped_indices - original_indices
+            raise RuntimeError(f"Character grouping index mismatch. Missing: {missing}, Extra: {extra}")
+        
+        print(f"   ✅ Character grouping validated: {grouped_count} segments preserved")

engines/chatterbox_official_23lang/continuous_worker_processor.py

@@ -0,0 +1,277 @@
+"""
+Continuous Worker Processor - True Non-Stop Parallelization for ChatterBox
+
+This implements what the user actually wanted:
+- Workers continuously process work without waiting for batches
+- No idle workers when user sets high worker counts
+- Maintains the beautiful logging structure from the original
+- Falls back gracefully to sequential processing for 0-1 workers
+"""
+
+import torch
+import time
+import threading
+from queue import Queue, Empty
+from typing import List, Dict, Any, Optional
+from dataclasses import dataclass
+from concurrent.futures import ThreadPoolExecutor
+import gc
+
+@dataclass
+class ContinuousWorkItem:
+    """Single work item for continuous processing."""
+    text_index: int
+    text: str
+    temperature: float
+    cfg_weight: float
+    exaggeration: float
+
+@dataclass
+class ContinuousResult:
+    """Result from continuous processing."""
+    text_index: int
+    audio: torch.Tensor
+    processing_time: float
+    worker_id: int
+    success: bool = True
+    error_msg: str = ""
+
+class ContinuousWorkerProcessor:
+    """
+    True continuous worker processor that maintains workers always busy.
+    Falls back to sequential processing gracefully for 0-1 workers.
+    Maintains beautiful logging structure.
+    """
+    
+    def __init__(self, tts_model):
+        self.tts_model = tts_model
+        
+    def process_texts_continuously(
+        self,
+        texts: List[str],
+        temperature: float,
+        cfg_weight: float, 
+        exaggeration: float,
+        max_workers: int
+    ) -> List[torch.Tensor]:
+        """
+        Process texts with continuous workers or fallback to sequential.
+        
+        Args:
+            texts: List of texts to process
+            temperature: Sampling temperature
+            cfg_weight: CFG weight
+            exaggeration: Exaggeration factor
+            max_workers: Number of workers (0-1 = sequential, >1 = continuous)
+        
+        Returns:
+            List of audio tensors
+        """
+        if not texts:
+            return []
+            
+        # SMART FALLBACK: Use sequential processing for 0-1 workers
+        if max_workers <= 1:
+            print(f"→ SEQUENTIAL FALLBACK: {len(texts)} texts (workers={max_workers})")
+            return self._process_sequential(texts, temperature, cfg_weight, exaggeration)
+        
+        # Use continuous worker processing for >1 workers
+        print(f"🌊 CONTINUOUS PROCESSING: {len(texts)} texts with {max_workers} workers")
+        return self._process_with_continuous_workers(texts, temperature, cfg_weight, exaggeration, max_workers)
+    
+    def _process_sequential(
+        self,
+        texts: List[str], 
+        temperature: float,
+        cfg_weight: float,
+        exaggeration: float
+    ) -> List[torch.Tensor]:
+        """Sequential processing - clean and organized like the original."""
+        results = []
+        
+        for i, text in enumerate(texts):
+            print(f"  🎤 Sequential {i+1}/{len(texts)}: {text[:40]}...")
+            
+            try:
+                audio = self.tts_model.generate(
+                    text=text,
+                    audio_prompt_path=None,  # Already prepared
+                    temperature=temperature,
+                    cfg_weight=cfg_weight,
+                    exaggeration=exaggeration
+                )
+                results.append(audio)
+                print(f"  ✅ Sequential {i+1}: Completed")
+                
+            except Exception as e:
+                print(f"  ❌ Sequential {i+1}: Failed - {e}")
+                results.append(torch.zeros(1, 1000))
+        
+        print(f"✅ SEQUENTIAL COMPLETE: {len(texts)} texts processed")
+        return results
+    
+    def _process_with_continuous_workers(
+        self,
+        texts: List[str],
+        temperature: float,
+        cfg_weight: float, 
+        exaggeration: float,
+        max_workers: int
+    ) -> List[torch.Tensor]:
+        """
+        Continuous worker processing - workers never idle.
+        Maintains beautiful logging structure.
+        """
+        # Build work queue
+        work_queue = Queue()
+        result_queue = Queue()
+        shutdown_event = threading.Event()
+        
+        # Fill queue with all work
+        for i, text in enumerate(texts):
+            work_item = ContinuousWorkItem(
+                text_index=i,
+                text=text,
+                temperature=temperature,
+                cfg_weight=cfg_weight,
+                exaggeration=exaggeration
+            )
+            work_queue.put(work_item)
+        
+        print(f"📥 Work queue filled: {len(texts)} items")
+        
+        # Start workers
+        workers = []
+        worker_threads = []
+        
+        for worker_id in range(max_workers):
+            worker = ContinuousWorker(
+                worker_id=worker_id + 1,
+                work_queue=work_queue,
+                result_queue=result_queue,
+                tts_model=self.tts_model,
+                shutdown_event=shutdown_event
+            )
+            
+            thread = threading.Thread(target=worker.run)
+            thread.start()
+            
+            workers.append(worker)
+            worker_threads.append(thread)
+        
+        print(f"🚀 Started {max_workers} continuous workers")
+        
+        # Collect results
+        results = [None] * len(texts)
+        completed_count = 0
+        start_time = time.time()
+        
+        while completed_count < len(texts):
+            try:
+                result = result_queue.get(timeout=5.0)
+                results[result.text_index] = result.audio
+                completed_count += 1
+                
+                # Beautiful progress logging (like original)
+                if completed_count % max(1, len(texts) // 10) == 0:
+                    progress = int(100 * completed_count / len(texts))
+                    elapsed = time.time() - start_time
+                    rate = completed_count / elapsed if elapsed > 0 else 0
+                    print(f"📊 Continuous progress: {completed_count}/{len(texts)} ({progress}%) - {rate:.1f}/sec")
+                
+            except Empty:
+                print("⏳ Waiting for continuous workers...")
+                continue
+        
+        # Shutdown workers
+        shutdown_event.set()
+        
+        # Send shutdown signals
+        for _ in range(max_workers):
+            work_queue.put(None)
+        
+        # Wait for threads
+        for thread in worker_threads:
+            thread.join(timeout=3.0)
+        
+        total_time = time.time() - start_time
+        throughput = len(texts) / total_time if total_time > 0 else 0
+        
+        print(f"✅ CONTINUOUS COMPLETE: {len(texts)} texts in {total_time:.1f}s ({throughput:.2f}/sec)")
+        
+        return results
+
+class ContinuousWorker:
+    """Individual continuous worker that processes from shared queue."""
+    
+    def __init__(self, worker_id: int, work_queue: Queue, result_queue: Queue, 
+                 tts_model, shutdown_event: threading.Event):
+        self.worker_id = worker_id
+        self.work_queue = work_queue
+        self.result_queue = result_queue
+        self.tts_model = tts_model
+        self.shutdown_event = shutdown_event
+        self.processed_count = 0
+        
+    def run(self):
+        """Main worker loop - continuously processes work."""
+        print(f"  🧵 Worker {self.worker_id}: Ready for continuous work")
+        
+        while not self.shutdown_event.is_set():
+            try:
+                # Get next work item
+                work_item = self.work_queue.get(timeout=1.0)
+                
+                if work_item is None:  # Shutdown signal
+                    break
+                
+                print(f"  🧵 Worker {self.worker_id}: Processing: {work_item.text[:30]}...")
+                
+                # Process work item
+                start_time = time.time()
+                try:
+                    audio = self.tts_model.generate(
+                        text=work_item.text,
+                        audio_prompt_path=None,  # Already prepared
+                        temperature=work_item.temperature,
+                        cfg_weight=work_item.cfg_weight,
+                        exaggeration=work_item.exaggeration
+                    )
+                    
+                    processing_time = time.time() - start_time
+                    self.processed_count += 1
+                    
+                    result = ContinuousResult(
+                        text_index=work_item.text_index,
+                        audio=audio,
+                        processing_time=processing_time,
+                        worker_id=self.worker_id,
+                        success=True
+                    )
+                    
+                    print(f"  ✅ Worker {self.worker_id}: Completed in {processing_time:.1f}s")
+                    
+                except Exception as e:
+                    processing_time = time.time() - start_time
+                    result = ContinuousResult(
+                        text_index=work_item.text_index,
+                        audio=torch.zeros(1, 1000),
+                        processing_time=processing_time,
+                        worker_id=self.worker_id,
+                        success=False,
+                        error_msg=str(e)
+                    )
+                    
+                    print(f"  ❌ Worker {self.worker_id}: Failed - {e}")
+                
+                # Send result back
+                self.result_queue.put(result)
+                self.work_queue.task_done()
+                
+            except Empty:
+                # Timeout - check shutdown and continue
+                continue
+            except Exception as e:
+                print(f"  ❌ Worker {self.worker_id}: Unexpected error - {e}")
+        
+        print(f"  🛑 Worker {self.worker_id}: Shutdown ({self.processed_count} processed)")

engines/chatterbox_official_23lang/gpu_aware_processor.py

@@ -0,0 +1,240 @@
+"""
+GPU-Aware Batch Processor for ChatterBox TTS
+
+Intelligently limits parallel processing based on GPU capabilities to avoid resource contention.
+Uses empirical testing to find optimal worker count for maximum throughput.
+
+Modular design following CLAUDE.md guidelines - under 500 lines, focused responsibility.
+"""
+
+import torch
+import psutil
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from typing import List, Tuple, Optional
+import time
+
+
+class GPUAwareBatchProcessor:
+    """
+    Intelligent batch processor that optimizes worker count based on GPU capabilities.
+    
+    Instead of using all available workers (which causes resource contention),
+    this processor finds the optimal balance between parallelism and GPU efficiency.
+    """
+    
+    def __init__(self, tts_model, device: str = "auto"):
+        """
+        Initialize with GPU-aware configuration.
+        
+        Args:
+            tts_model: ChatterboxTTS instance
+            device: Target device ("cuda", "cpu", "auto")
+        """
+        self.tts_model = tts_model
+        self.device = self._resolve_device(device)
+        self.optimal_workers = self._determine_optimal_workers()
+        
+        print(f"🔧 GPU-Aware Processor initialized: {self.optimal_workers} optimal workers for {self.device}")
+    
+    def _resolve_device(self, device: str) -> str:
+        """Resolve device string to actual device."""
+        from utils.device import resolve_torch_device
+        return resolve_torch_device(device)
+    
+    def _determine_optimal_workers(self) -> int:
+        """
+        Determine optimal worker count based on hardware capabilities.
+        
+        Returns:
+            Optimal number of parallel workers
+        """
+        if self.device == "cpu":
+            # CPU-only: use more workers since no GPU contention
+            return min(psutil.cpu_count() or 4, 8)
+        
+        # GPU-based optimization
+        if torch.cuda.is_available():
+            gpu_memory_gb = torch.cuda.get_device_properties(0).total_memory / (1024**3)
+            
+            # Empirical optimization based on ChatterBox GPU requirements
+            if gpu_memory_gb >= 24:      # RTX 4090, A6000, etc.
+                return 4  # Sweet spot for high-end cards
+            elif gpu_memory_gb >= 16:    # RTX 4080, etc.
+                return 3
+            elif gpu_memory_gb >= 12:    # RTX 4070 Ti, etc.
+                return 2
+            elif gpu_memory_gb >= 8:     # RTX 4060 Ti, etc.  
+                return 2
+            else:                        # < 8GB VRAM
+                return 1  # Sequential processing safer
+        
+        return 2  # Conservative fallback
+    
+    def process_texts_optimized(
+        self, 
+        texts: List[str], 
+        temperature: float, 
+        cfg_weight: float, 
+        exaggeration: float,
+        requested_workers: int
+    ) -> List[torch.Tensor]:
+        """
+        Process texts with GPU-optimized worker count.
+        
+        Args:
+            texts: List of text strings
+            temperature: Sampling temperature
+            cfg_weight: CFG weight
+            exaggeration: Emotion exaggeration
+            requested_workers: User-requested worker count (will be clamped to optimal)
+            
+        Returns:
+            List of audio tensors in original order
+        """
+        # Use optimal workers regardless of user request for best performance
+        effective_workers = min(self.optimal_workers, len(texts))
+        
+        if effective_workers != requested_workers:
+            print(f"🔧 GPU Optimization: Using {effective_workers} workers instead of {requested_workers} for optimal GPU utilization")
+        
+        return self._process_with_workers(texts, temperature, cfg_weight, exaggeration, effective_workers)
+    
+    def _process_with_workers(
+        self, 
+        texts: List[str], 
+        temperature: float, 
+        cfg_weight: float, 
+        exaggeration: float,
+        max_workers: int
+    ) -> List[torch.Tensor]:
+        """Process texts with specified worker count."""
+        
+        def _process_single_text(text_index: int, text: str) -> Tuple[int, torch.Tensor]:
+            """Process single text with GPU-aware error handling."""
+            try:
+                print(f"  🧵 GPU-Worker {text_index+1}: Processing: {text[:30]}...")
+                
+                # Clear GPU cache before processing to avoid OOM
+                if self.device.startswith("cuda"):
+                    torch.cuda.empty_cache()
+                
+                start_time = time.time()
+                audio = self.tts_model.generate(
+                    text=text,
+                    audio_prompt_path=None,
+                    exaggeration=exaggeration,
+                    cfg_weight=cfg_weight,
+                    temperature=temperature,
+                )
+                
+                duration = time.time() - start_time
+                print(f"  ✅ GPU-Worker {text_index+1}: Completed in {duration:.1f}s")
+                return text_index, audio
+                
+            except Exception as e:
+                print(f"  ❌ GPU-Worker {text_index+1}: Failed - {e}")
+                return text_index, torch.zeros(1, 1000)
+        
+        print(f"🚀 GPU-OPTIMIZED PROCESSING: {len(texts)} texts with {max_workers} optimal GPU workers")
+        
+        # Process with optimal worker count
+        results = [None] * len(texts)
+        
+        with ThreadPoolExecutor(max_workers=max_workers) as executor:
+            futures = [
+                executor.submit(_process_single_text, i, text)
+                for i, text in enumerate(texts)
+            ]
+            
+            completed_count = 0
+            total_duration = time.time()
+            
+            for future in as_completed(futures):
+                text_index, audio = future.result()
+                results[text_index] = audio
+                completed_count += 1
+                
+                progress_percent = int(100 * completed_count / len(texts))
+                print(f"  📊 Progress: {completed_count}/{len(texts)} completed ({progress_percent}%)")
+        
+        total_time = time.time() - total_duration
+        avg_time = total_time / len(texts)
+        print(f"✅ GPU-OPTIMIZED PROCESSING COMPLETED: {len(results)} texts in {total_time:.1f}s (avg: {avg_time:.1f}s/text)")
+        
+        return results
+    
+    def benchmark_worker_counts(self, sample_texts: List[str], max_test_workers: int = 8) -> dict:
+        """
+        Benchmark different worker counts to find empirically optimal settings.
+        
+        Args:
+            sample_texts: Representative text samples for benchmarking
+            max_test_workers: Maximum workers to test
+            
+        Returns:
+            Dict with benchmark results
+        """
+        if len(sample_texts) < 2:
+            print("⚠️ Need at least 2 sample texts for benchmarking")
+            return {}
+        
+        print(f"🔬 Benchmarking worker counts 1-{max_test_workers} with {len(sample_texts)} sample texts...")
+        
+        results = {}
+        
+        for workers in range(1, min(max_test_workers + 1, len(sample_texts) + 1)):
+            print(f"🧪 Testing {workers} workers...")
+            
+            start_time = time.time()
+            _ = self._process_with_workers(
+                sample_texts[:workers], 0.8, 0.5, 0.5, workers
+            )
+            duration = time.time() - start_time
+            
+            throughput = workers / duration  # texts per second
+            results[workers] = {
+                'duration': duration,
+                'throughput': throughput,
+                'texts_processed': workers
+            }
+            
+            print(f"  📈 {workers} workers: {duration:.1f}s total, {throughput:.2f} texts/sec")
+        
+        # Find optimal
+        best_workers = max(results.keys(), key=lambda w: results[w]['throughput'])
+        best_throughput = results[best_workers]['throughput']
+        
+        print(f"🏆 OPTIMAL: {best_workers} workers with {best_throughput:.2f} texts/sec throughput")
+        
+        return results
+
+
+class ProcessingStrategy:
+    """Strategy selector for different processing approaches."""
+    
+    @staticmethod
+    def choose_strategy(num_texts: int, requested_workers: int, device: str) -> str:
+        """Choose optimal processing strategy."""
+        if device == "cpu":
+            return "cpu_parallel"
+        elif num_texts == 1:
+            return "single"
+        elif torch.cuda.is_available():
+            gpu_memory_gb = torch.cuda.get_device_properties(0).total_memory / (1024**3)
+            if gpu_memory_gb < 8:
+                return "sequential"  # Low VRAM - avoid parallelism
+            else:
+                return "gpu_optimized"
+        else:
+            return "sequential"
+    
+    @staticmethod
+    def get_strategy_explanation(strategy: str) -> str:
+        """Get explanation of chosen strategy."""
+        explanations = {
+            "single": "Single text - direct processing",
+            "sequential": "Sequential processing - low GPU memory",
+            "cpu_parallel": "CPU parallel processing - no GPU constraints",
+            "gpu_optimized": "GPU-optimized parallel processing - balanced for throughput"
+        }
+        return explanations.get(strategy, "Unknown strategy")

engines/chatterbox_official_23lang/language_models.py

@@ -0,0 +1,282 @@
+"""
+ChatterBox Official 23-Lang Language Model Registry
+Manages the official ResembleAI multilingual ChatterBox model supporting 23 languages
+"""
+
+import os
+import folder_paths
+from typing import Dict, List, Tuple, Optional
+
+# Official 23-language ChatterBox model configuration
+# Based on ResembleAI's official multilingual implementation
+OFFICIAL_23LANG_MODELS = {
+    # The single unified multilingual model supporting all 23 languages
+    "ChatterBox Official 23-Lang": {
+        "repo": "ResembleAI/chatterbox",
+        "format": "mixed",  # Uses both .pt and .safetensors
+        "description": "Official ResembleAI multilingual ChatterBox model supporting 23 languages",
+        "languages": [
+            "ar",  # Arabic
+            "da",  # Danish
+            "de",  # German
+            "el",  # Greek
+            "en",  # English
+            "es",  # Spanish
+            "fi",  # Finnish
+            "fr",  # French
+            "he",  # Hebrew
+            "hi",  # Hindi
+            "it",  # Italian
+            "ja",  # Japanese
+            "ko",  # Korean
+            "ms",  # Malay
+            "nl",  # Dutch
+            "no",  # Norwegian
+            "pl",  # Polish
+            "pt",  # Portuguese
+            "ru",  # Russian
+            "sv",  # Swedish
+            "sw",  # Swahili
+            "tr",  # Turkish
+            "zh",  # Chinese
+        ],
+        "required_files": {
+            "v1": [
+                "t3_23lang.safetensors",  # Multilingual T3 model v1
+                "s3gen.pt",               # S3Gen model (same as English)
+                "ve.pt",                  # Voice encoder (same as English)
+                "mtl_tokenizer.json",     # Multilingual tokenizer
+                "conds.pt"                # Conditioning (optional)
+            ],
+            "v2": [
+                "t3_mtl23ls_v2.safetensors",      # Multilingual T3 model v2 with enhanced tokenization
+                "s3gen.pt",                        # S3Gen model (same as English)
+                "ve.pt",                           # Voice encoder (same as English)
+                "grapheme_mtl_merged_expanded_v1.json",  # Enhanced grapheme/phoneme mappings with special tokens
+                "mtl_tokenizer.json",              # Multilingual tokenizer (may be updated for v2)
+                "conds.pt"                         # Conditioning (optional)
+            ]
+        },
+        "multilingual": True
+    }
+}
+
+# Language code to display name mapping  
+SUPPORTED_LANGUAGES = {
+    "ar": "Arabic",
+    "da": "Danish",
+    "de": "German", 
+    "el": "Greek",
+    "en": "English",
+    "es": "Spanish",
+    "fi": "Finnish",
+    "fr": "French",
+    "he": "Hebrew",
+    "hi": "Hindi",
+    "it": "Italian",
+    "ja": "Japanese",
+    "ko": "Korean",
+    "ms": "Malay",
+    "nl": "Dutch",
+    "no": "Norwegian",
+    "pl": "Polish",
+    "pt": "Portuguese",
+    "ru": "Russian",
+    "sv": "Swedish",
+    "sw": "Swahili",
+    "tr": "Turkish",
+    "zh": "Chinese",
+}
+
+def get_official_23lang_models() -> List[str]:
+    """
+    Get list of available official 23-lang models.
+    This implementation only supports the unified multilingual model.
+    """
+    models = list(OFFICIAL_23LANG_MODELS.keys())
+    
+    # Check for local models in ComfyUI models directory
+    try:
+        # Check both new TTS organization and legacy path
+        tts_models_dir = os.path.join(folder_paths.models_dir, "TTS", "chatterbox_official_23lang")
+        legacy_models_dir = os.path.join(folder_paths.models_dir, "chatterbox_official_23lang")
+        
+        # Try TTS path first, then legacy
+        for models_dir in [tts_models_dir, legacy_models_dir]:
+            if not os.path.exists(models_dir):
+                continue
+                
+            for item in os.listdir(models_dir):
+                item_path = os.path.join(models_dir, item)
+                if os.path.isdir(item_path):
+                    # Check if it contains official 23-lang model files
+                    required_files = ["t3_23lang.safetensors", "s3gen.pt", "ve.pt", "mtl_tokenizer.json"]
+                    has_model = True
+                    
+                    for required_file in required_files:
+                        file_path = os.path.join(item_path, required_file)
+                        if not os.path.exists(file_path):
+                            has_model = False
+                            break
+                    
+                    if has_model:
+                        local_model = f"local:{item}"
+                        if local_model not in models:
+                            models.append(local_model)
+    except Exception:
+        pass  # Ignore errors in model discovery
+    
+    return models
+
+def get_supported_languages() -> List[str]:
+    """Get list of supported language codes."""
+    return list(SUPPORTED_LANGUAGES.keys())
+
+def get_supported_language_names() -> List[str]:
+    """Get list of supported language display names."""
+    return list(SUPPORTED_LANGUAGES.values())
+
+def get_language_name(language_code: str) -> str:
+    """Get display name for a language code."""
+    return SUPPORTED_LANGUAGES.get(language_code, language_code)
+
+def get_model_config(model_name: str) -> Optional[Dict]:
+    """Get configuration for a specific model"""
+    if model_name.startswith("local:"):
+        # Local model
+        local_name = model_name[6:]  # Remove "local:" prefix
+        return {
+            "repo": None,
+            "format": "mixed",  # Auto-detect format
+            "local_path": os.path.join(folder_paths.models_dir, "TTS", "chatterbox_official_23lang", local_name),
+            "description": f"Local ChatterBox Official 23-Lang model: {local_name}",
+            "multilingual": True,
+            "languages": list(SUPPORTED_LANGUAGES.keys())
+        }
+    
+    return OFFICIAL_23LANG_MODELS.get(model_name)
+
+def get_model_files_for_model(model_name: str) -> Tuple[str, str]:
+    """
+    Get the expected file format and repo for a model.
+    Returns (format, repo_id) tuple.
+    """
+    config = get_model_config(model_name)
+    if not config:
+        # Default to the official model
+        config = OFFICIAL_23LANG_MODELS["ChatterBox Official 23-Lang"]
+    
+    return config.get("format", "mixed"), config.get("repo")
+
+def find_local_model_path(model_name: str) -> Optional[str]:
+    """Find local model path for a given model"""
+    if model_name.startswith("local:"):
+        local_name = model_name[6:]
+        # Try TTS path first, then legacy
+        for base_dir in ["TTS", ""]:
+            if base_dir:
+                model_path = os.path.join(folder_paths.models_dir, base_dir, "chatterbox_official_23lang", local_name)
+            else:
+                model_path = os.path.join(folder_paths.models_dir, "chatterbox_official_23lang", local_name)
+            if os.path.exists(model_path):
+                return model_path
+    else:
+        # Check if we have a local version of a predefined model
+        # Try TTS path first, then legacy
+        for base_dir in ["TTS", ""]:
+            if base_dir:
+                model_path = os.path.join(folder_paths.models_dir, base_dir, "chatterbox_official_23lang", model_name)
+            else:
+                model_path = os.path.join(folder_paths.models_dir, "chatterbox_official_23lang", model_name)
+            if os.path.exists(model_path):
+                return model_path
+    
+    return None
+
+def detect_model_format(model_path: str) -> str:
+    """
+    Auto-detect the format of models in a directory.
+    Returns 'safetensors', 'pt', or 'mixed'
+    """
+    if not os.path.exists(model_path):
+        return "mixed"  # Default format for official model
+    
+    has_safetensors = False
+    has_pt = False
+    
+    for file in os.listdir(model_path):
+        if file.endswith(".safetensors"):
+            has_safetensors = True
+        elif file.endswith(".pt"):
+            has_pt = True
+    
+    if has_safetensors and has_pt:
+        return "mixed"
+    elif has_safetensors:
+        return "safetensors"
+    else:
+        return "pt"
+
+def get_model_requirements(model_name: str, model_version: str = "v1") -> List[str]:
+    """Get list of required files for the official 23-lang model based on version"""
+    config = get_model_config(model_name)
+    if config and "required_files" in config:
+        required_files = config["required_files"]
+
+        # Handle version-specific requirements
+        if isinstance(required_files, dict):
+            return required_files.get(model_version, required_files.get("v1", []))
+        else:
+            # Legacy format - return as-is
+            return required_files
+
+    # Default requirements for official model v1
+    return [
+        "t3_23lang.safetensors",
+        "s3gen.pt",
+        "ve.pt",
+        "mtl_tokenizer.json",
+        "conds.pt"  # Optional but expected
+    ]
+
+def validate_model_completeness(model_path: str, model_name: str, model_version: str = "v1") -> Tuple[bool, List[str]]:
+    """
+    Validate if a model has all required components.
+    Returns (is_complete, missing_files)
+    """
+    if not os.path.exists(model_path):
+        return False, ["Model directory not found"]
+
+    required_files = get_model_requirements(model_name, model_version)
+    missing_files = []
+    optional_files = ["conds.pt", "grapheme_mtl_merged_expanded_v1.json"]  # These are optional
+
+    existing_files = os.listdir(model_path)
+    for required_file in required_files:
+        # Check for exact file name
+        if required_file not in existing_files:
+            # Only consider it missing if it's not optional
+            if required_file not in optional_files:
+                missing_files.append(required_file)
+
+    return len(missing_files) == 0, missing_files
+
+def supports_voice_conversion(model_name: str) -> bool:
+    """
+    Check if a model supports voice conversion.
+    Official 23-lang model supports VC via S3Gen.
+    """
+    return True  # Official model supports VC
+
+def get_available_models() -> List[str]:
+    """Get list of available model names for display"""
+    return get_official_23lang_models()
+
+# Legacy compatibility functions (renamed from original chatterbox functions)
+def get_chatterbox_models():
+    """Legacy compatibility - redirect to official models"""
+    return get_official_23lang_models()
+
+def get_available_languages():
+    """Legacy compatibility - redirect to supported languages"""  
+    return get_supported_language_names()

engines/chatterbox_official_23lang/models/s3gen/__init__.py

@@ -0,0 +1,2 @@
+from .s3gen import S3Token2Wav as S3Gen
+from .const import S3GEN_SR

engines/chatterbox_official_23lang/models/s3gen/configs.py

@@ -0,0 +1,10 @@
+from ..utils import AttrDict
+
+CFM_PARAMS = AttrDict({
+    "sigma_min": 1e-06,
+    "solver": "euler",
+    "t_scheduler": "cosine",
+    "training_cfg_rate": 0.2,
+    "inference_cfg_rate": 0.7,
+    "reg_loss_type": "l1"
+})

engines/chatterbox_official_23lang/models/s3gen/const.py

@@ -0,0 +1 @@
+S3GEN_SR = 24000

engines/chatterbox_official_23lang/models/s3gen/decoder.py

@@ -0,0 +1,317 @@
+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Zhihao Du)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from einops import pack, rearrange, repeat
+
+from .utils.mask import add_optional_chunk_mask
+from .matcha.decoder import SinusoidalPosEmb, Block1D, ResnetBlock1D, Downsample1D, \
+    TimestepEmbedding, Upsample1D
+from .matcha.transformer import BasicTransformerBlock
+
+
+def mask_to_bias(mask: torch.Tensor, dtype: torch.dtype) -> torch.Tensor:
+    assert mask.dtype == torch.bool
+    assert dtype in [torch.float32, torch.bfloat16, torch.float16]
+    mask = mask.to(dtype)
+    # attention mask bias
+    # NOTE(Mddct): torch.finfo jit issues
+    #     chunk_masks = (1.0 - chunk_masks) * torch.finfo(dtype).min
+    mask = (1.0 - mask) * -1.0e+10
+    return mask
+
+
+
+class Transpose(torch.nn.Module):
+    def __init__(self, dim0: int, dim1: int):
+        super().__init__()
+        self.dim0 = dim0
+        self.dim1 = dim1
+
+    def forward(self, x: torch.Tensor):
+        x = torch.transpose(x, self.dim0, self.dim1)
+        return x
+
+
+class CausalBlock1D(Block1D):
+    def __init__(self, dim: int, dim_out: int):
+        super(CausalBlock1D, self).__init__(dim, dim_out)
+        self.block = torch.nn.Sequential(
+            CausalConv1d(dim, dim_out, 3),
+            Transpose(1, 2),
+            nn.LayerNorm(dim_out),
+            Transpose(1, 2),
+            nn.Mish(),
+        )
+
+    def forward(self, x: torch.Tensor, mask: torch.Tensor):
+        output = self.block(x * mask)
+        return output * mask
+
+
+class CausalResnetBlock1D(ResnetBlock1D):
+    def __init__(self, dim: int, dim_out: int, time_emb_dim: int, groups: int = 8):
+        super(CausalResnetBlock1D, self).__init__(dim, dim_out, time_emb_dim, groups)
+        self.block1 = CausalBlock1D(dim, dim_out)
+        self.block2 = CausalBlock1D(dim_out, dim_out)
+
+
+class CausalConv1d(torch.nn.Conv1d):
+    def __init__(
+        self,
+        in_channels: int,
+        out_channels: int,
+        kernel_size: int,
+        stride: int = 1,
+        dilation: int = 1,
+        groups: int = 1,
+        bias: bool = True,
+        padding_mode: str = 'zeros',
+        device=None,
+        dtype=None
+    ) -> None:
+        super(CausalConv1d, self).__init__(in_channels, out_channels,
+                                           kernel_size, stride,
+                                           padding=0, dilation=dilation,
+                                           groups=groups, bias=bias,
+                                           padding_mode=padding_mode,
+                                           device=device, dtype=dtype)
+        assert stride == 1
+        self.causal_padding = (kernel_size - 1, 0)
+
+    def forward(self, x: torch.Tensor):
+        x = F.pad(x, self.causal_padding)
+        x = super(CausalConv1d, self).forward(x)
+        return x
+
+
+class ConditionalDecoder(nn.Module):
+    def __init__(
+        self,
+        in_channels=320,
+        out_channels=80,
+        causal=True,
+        channels=[256],
+        dropout=0.0,
+        attention_head_dim=64,
+        n_blocks=4,
+        num_mid_blocks=12,
+        num_heads=8,
+        act_fn="gelu",
+    ):
+        """
+        This decoder requires an input with the same shape of the target. So, if your text content
+        is shorter or longer than the outputs, please re-sampling it before feeding to the decoder.
+        """
+        super().__init__()
+        channels = tuple(channels)
+        self.in_channels = in_channels
+        self.out_channels = out_channels
+        self.causal = causal
+        self.time_embeddings = SinusoidalPosEmb(in_channels)
+        time_embed_dim = channels[0] * 4
+        self.time_mlp = TimestepEmbedding(
+            in_channels=in_channels,
+            time_embed_dim=time_embed_dim,
+            act_fn="silu",
+        )
+        self.down_blocks = nn.ModuleList([])
+        self.mid_blocks = nn.ModuleList([])
+        self.up_blocks = nn.ModuleList([])
+
+        # NOTE jrm: `static_chunk_size` is missing?
+        self.static_chunk_size = 0
+
+        output_channel = in_channels
+        for i in range(len(channels)):  # pylint: disable=consider-using-enumerate
+            input_channel = output_channel
+            output_channel = channels[i]
+            is_last = i == len(channels) - 1
+            resnet = CausalResnetBlock1D(dim=input_channel, dim_out=output_channel, time_emb_dim=time_embed_dim) if self.causal else \
+                ResnetBlock1D(dim=input_channel, dim_out=output_channel, time_emb_dim=time_embed_dim)
+            transformer_blocks = nn.ModuleList(
+                [
+                    BasicTransformerBlock(
+                        dim=output_channel,
+                        num_attention_heads=num_heads,
+                        attention_head_dim=attention_head_dim,
+                        dropout=dropout,
+                        activation_fn=act_fn,
+                    )
+                    for _ in range(n_blocks)
+                ]
+            )
+            downsample = (
+                Downsample1D(output_channel) if not is_last else
+                CausalConv1d(output_channel, output_channel, 3) if self.causal else nn.Conv1d(output_channel, output_channel, 3, padding=1)
+            )
+            self.down_blocks.append(nn.ModuleList([resnet, transformer_blocks, downsample]))
+
+        for _ in range(num_mid_blocks):
+            input_channel = channels[-1]
+            out_channels = channels[-1]
+            resnet = CausalResnetBlock1D(dim=input_channel, dim_out=output_channel, time_emb_dim=time_embed_dim) if self.causal else \
+                ResnetBlock1D(dim=input_channel, dim_out=output_channel, time_emb_dim=time_embed_dim)
+
+            transformer_blocks = nn.ModuleList(
+                [
+                    BasicTransformerBlock(
+                        dim=output_channel,
+                        num_attention_heads=num_heads,
+                        attention_head_dim=attention_head_dim,
+                        dropout=dropout,
+                        activation_fn=act_fn,
+                    )
+                    for _ in range(n_blocks)
+                ]
+            )
+
+            self.mid_blocks.append(nn.ModuleList([resnet, transformer_blocks]))
+
+        channels = channels[::-1] + (channels[0],)
+        for i in range(len(channels) - 1):
+            input_channel = channels[i] * 2
+            output_channel = channels[i + 1]
+            is_last = i == len(channels) - 2
+            resnet = CausalResnetBlock1D(
+                dim=input_channel,
+                dim_out=output_channel,
+                time_emb_dim=time_embed_dim,
+            ) if self.causal else ResnetBlock1D(
+                dim=input_channel,
+                dim_out=output_channel,
+                time_emb_dim=time_embed_dim,
+            )
+            transformer_blocks = nn.ModuleList(
+                [
+                    BasicTransformerBlock(
+                        dim=output_channel,
+                        num_attention_heads=num_heads,
+                        attention_head_dim=attention_head_dim,
+                        dropout=dropout,
+                        activation_fn=act_fn,
+                    )
+                    for _ in range(n_blocks)
+                ]
+            )
+            upsample = (
+                Upsample1D(output_channel, use_conv_transpose=True)
+                if not is_last
+                else CausalConv1d(output_channel, output_channel, 3) if self.causal else nn.Conv1d(output_channel, output_channel, 3, padding=1)
+            )
+            self.up_blocks.append(nn.ModuleList([resnet, transformer_blocks, upsample]))
+        self.final_block = CausalBlock1D(channels[-1], channels[-1]) if self.causal else Block1D(channels[-1], channels[-1])
+        self.final_proj = nn.Conv1d(channels[-1], self.out_channels, 1)
+        self.initialize_weights()
+
+    def initialize_weights(self):
+        for m in self.modules():
+            if isinstance(m, nn.Conv1d):
+                nn.init.kaiming_normal_(m.weight, nonlinearity="relu")
+                if m.bias is not None:
+                    nn.init.constant_(m.bias, 0)
+            elif isinstance(m, nn.GroupNorm):
+                nn.init.constant_(m.weight, 1)
+                nn.init.constant_(m.bias, 0)
+            elif isinstance(m, nn.Linear):
+                nn.init.kaiming_normal_(m.weight, nonlinearity="relu")
+                if m.bias is not None:
+                    nn.init.constant_(m.bias, 0)
+
+    def forward(self, x, mask, mu, t, spks=None, cond=None):
+        """Forward pass of the UNet1DConditional model.
+
+        Args:
+            x (torch.Tensor): shape (batch_size, in_channels, time)
+            mask (_type_): shape (batch_size, 1, time)
+            t (_type_): shape (batch_size)
+            spks (_type_, optional): shape: (batch_size, condition_channels). Defaults to None.
+            cond (_type_, optional): placeholder for future use. Defaults to None.
+
+        Raises:
+            ValueError: _description_
+            ValueError: _description_
+
+        Returns:
+            _type_: _description_
+        """
+
+        t = self.time_embeddings(t).to(t.dtype)
+        t = self.time_mlp(t)
+
+        x = pack([x, mu], "b * t")[0]
+
+        if spks is not None:
+            spks = repeat(spks, "b c -> b c t", t=x.shape[-1])
+            x = pack([x, spks], "b * t")[0]
+        if cond is not None:
+            x = pack([x, cond], "b * t")[0]
+
+        hiddens = []
+        masks = [mask]
+        for resnet, transformer_blocks, downsample in self.down_blocks:
+            mask_down = masks[-1]
+            x = resnet(x, mask_down, t)
+            x = rearrange(x, "b c t -> b t c").contiguous()
+            # attn_mask = torch.matmul(mask_down.transpose(1, 2).contiguous(), mask_down)
+            attn_mask = add_optional_chunk_mask(x, mask_down.bool(), False, False, 0, self.static_chunk_size, -1)
+            attn_mask = mask_to_bias(attn_mask == 1, x.dtype)
+            for transformer_block in transformer_blocks:
+                x = transformer_block(
+                    hidden_states=x,
+                    attention_mask=attn_mask,
+                    timestep=t,
+                )
+            x = rearrange(x, "b t c -> b c t").contiguous()
+            hiddens.append(x)  # Save hidden states for skip connections
+            x = downsample(x * mask_down)
+            masks.append(mask_down[:, :, ::2])
+        masks = masks[:-1]
+        mask_mid = masks[-1]
+
+        for resnet, transformer_blocks in self.mid_blocks:
+            x = resnet(x, mask_mid, t)
+            x = rearrange(x, "b c t -> b t c").contiguous()
+            # attn_mask = torch.matmul(mask_mid.transpose(1, 2).contiguous(), mask_mid)
+            attn_mask = add_optional_chunk_mask(x, mask_mid.bool(), False, False, 0, self.static_chunk_size, -1)
+            attn_mask = mask_to_bias(attn_mask == 1, x.dtype)
+            for transformer_block in transformer_blocks:
+                x = transformer_block(
+                    hidden_states=x,
+                    attention_mask=attn_mask,
+                    timestep=t,
+                )
+            x = rearrange(x, "b t c -> b c t").contiguous()
+
+        for resnet, transformer_blocks, upsample in self.up_blocks:
+            mask_up = masks.pop()
+            skip = hiddens.pop()
+            x = pack([x[:, :, :skip.shape[-1]], skip], "b * t")[0]
+            x = resnet(x, mask_up, t)
+            x = rearrange(x, "b c t -> b t c").contiguous()
+            # attn_mask = torch.matmul(mask_up.transpose(1, 2).contiguous(), mask_up)
+            attn_mask = add_optional_chunk_mask(x, mask_up.bool(), False, False, 0, self.static_chunk_size, -1)
+            attn_mask = mask_to_bias(attn_mask == 1, x.dtype)
+            for transformer_block in transformer_blocks:
+                x = transformer_block(
+                    hidden_states=x,
+                    attention_mask=attn_mask,
+                    timestep=t,
+                )
+            x = rearrange(x, "b t c -> b c t").contiguous()
+            x = upsample(x * mask_up)
+        x = self.final_block(x, mask_up)
+        output = self.final_proj(x * mask_up)
+        return output * mask

engines/chatterbox_official_23lang/models/s3gen/f0_predictor.py

@@ -0,0 +1,55 @@
+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Kai Hu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import torch
+import torch.nn as nn
+from torch.nn.utils.parametrizations import weight_norm
+
+
+class ConvRNNF0Predictor(nn.Module):
+    def __init__(self,
+                 num_class: int = 1,
+                 in_channels: int = 80,
+                 cond_channels: int = 512
+                 ):
+        super().__init__()
+
+        self.num_class = num_class
+        self.condnet = nn.Sequential(
+            weight_norm(
+                nn.Conv1d(in_channels, cond_channels, kernel_size=3, padding=1)
+            ),
+            nn.ELU(),
+            weight_norm(
+                nn.Conv1d(cond_channels, cond_channels, kernel_size=3, padding=1)
+            ),
+            nn.ELU(),
+            weight_norm(
+                nn.Conv1d(cond_channels, cond_channels, kernel_size=3, padding=1)
+            ),
+            nn.ELU(),
+            weight_norm(
+                nn.Conv1d(cond_channels, cond_channels, kernel_size=3, padding=1)
+            ),
+            nn.ELU(),
+            weight_norm(
+                nn.Conv1d(cond_channels, cond_channels, kernel_size=3, padding=1)
+            ),
+            nn.ELU(),
+        )
+        self.classifier = nn.Linear(in_features=cond_channels, out_features=self.num_class)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        x = self.condnet(x)
+        x = x.transpose(1, 2)
+        return torch.abs(self.classifier(x).squeeze(-1))

engines/chatterbox_official_23lang/models/s3gen/flow.py

@@ -0,0 +1,290 @@
+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Zhihao Du)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import logging
+import random
+from typing import Dict, Optional
+
+logger = logging.getLogger(__name__)
+import torch
+import torch.nn as nn
+from torch.nn import functional as F
+from .utils.mask import make_pad_mask
+from .configs import CFM_PARAMS
+
+
+class MaskedDiffWithXvec(torch.nn.Module):
+    def __init__(
+        self,
+        input_size: int = 512,
+        output_size: int = 80,
+        spk_embed_dim: int = 192,
+        output_type: str = "mel",
+        vocab_size: int = 4096,
+        input_frame_rate: int = 50,
+        only_mask_loss: bool = True,
+        encoder: torch.nn.Module = None,
+        length_regulator: torch.nn.Module = None,
+        decoder: torch.nn.Module = None,
+        decoder_conf: Dict = {
+            'in_channels': 240,
+            'out_channel': 80,
+            'spk_emb_dim': 80,
+            'n_spks': 1,
+            'cfm_params': CFM_PARAMS,
+            'decoder_params': {
+                'channels': [256, 256],
+                'dropout': 0.0,
+                'attention_head_dim': 64,
+                'n_blocks': 4,
+                'num_mid_blocks': 12,
+                'num_heads': 8,
+                'act_fn': 'gelu',
+            }
+        },
+        mel_feat_conf: Dict = {
+            'n_fft': 1024,
+            'num_mels': 80,
+            'sampling_rate': 22050,
+            'hop_size': 256,
+            'win_size': 1024,
+            'fmin': 0,
+            'fmax': 8000
+        }
+    ):
+        super().__init__()
+        self.input_size = input_size
+        self.output_size = output_size
+        self.decoder_conf = decoder_conf
+        self.mel_feat_conf = mel_feat_conf
+        self.vocab_size = vocab_size
+        self.output_type = output_type
+        self.input_frame_rate = input_frame_rate
+        logging.info(f"input frame rate={self.input_frame_rate}")
+        self.input_embedding = nn.Embedding(vocab_size, input_size)
+        self.spk_embed_affine_layer = torch.nn.Linear(spk_embed_dim, output_size)
+        self.encoder = encoder
+        self.encoder_proj = torch.nn.Linear(self.encoder.output_size(), output_size)
+        self.decoder = decoder
+        self.length_regulator = length_regulator
+        self.only_mask_loss = only_mask_loss
+
+    def forward(
+            self,
+            batch: dict,
+            device: torch.device,
+    ) -> Dict[str, Optional[torch.Tensor]]:
+        token = batch['speech_token'].to(device)
+        token_len = batch['speech_token_len'].to(device)
+        feat = batch['speech_feat'].to(device)
+        feat_len = batch['speech_feat_len'].to(device)
+        embedding = batch['embedding'].to(device)
+
+        # xvec projection
+        embedding = F.normalize(embedding, dim=1)
+        embedding = self.spk_embed_affine_layer(embedding)
+
+        # concat text and prompt_text
+        mask = (~make_pad_mask(token_len)).float().unsqueeze(-1).to(device)
+        token = self.input_embedding(torch.clamp(token, min=0, max=self.input_embedding.num_embeddings-1)) * mask
+
+        # text encode
+        h, h_lengths = self.encoder(token, token_len)
+        h = self.encoder_proj(h)
+        h, h_lengths = self.length_regulator(h, feat_len)
+
+        # get conditions
+        conds = torch.zeros(feat.shape, device=token.device)
+        for i, j in enumerate(feat_len):
+            if random.random() < 0.5:
+                continue
+            index = random.randint(0, int(0.3 * j))
+            conds[i, :index] = feat[i, :index]
+        conds = conds.transpose(1, 2)
+
+        mask = (~make_pad_mask(feat_len)).to(h)
+        feat = F.interpolate(feat.unsqueeze(dim=1), size=h.shape[1:], mode="nearest").squeeze(dim=1)
+        loss, _ = self.decoder.compute_loss(
+            feat.transpose(1, 2).contiguous(),
+            mask.unsqueeze(1),
+            h.transpose(1, 2).contiguous(),
+            embedding,
+            cond=conds
+        )
+        return {'loss': loss}
+
+    @torch.inference_mode()
+    def inference(self,
+                  token,
+                  token_len,
+                  prompt_token,
+                  prompt_token_len,
+                  prompt_feat,
+                  prompt_feat_len,
+                  embedding,
+                  flow_cache):
+        if self.fp16 is True:
+            prompt_feat = prompt_feat.half()
+            embedding = embedding.half()
+
+        assert token.shape[0] == 1
+        # xvec projection
+        embedding = F.normalize(embedding, dim=1)
+        embedding = self.spk_embed_affine_layer(embedding)
+
+        # concat text and prompt_text
+        token_len1, token_len2 = prompt_token.shape[1], token.shape[1]
+        token, token_len = torch.concat([prompt_token, token], dim=1), prompt_token_len + token_len
+        mask = (~make_pad_mask(token_len)).unsqueeze(-1).to(embedding)
+        
+        # Check for out-of-bounds token IDs
+        vocab_size = self.input_embedding.num_embeddings
+        if token.max() >= vocab_size or token.min() < 0:
+            logging.warning(f"S3Gen: Token IDs out of bounds: min={token.min().item()}, max={token.max().item()}, vocab_size={vocab_size}")
+        
+        token = self.input_embedding(torch.clamp(token, min=0, max=vocab_size-1)) * mask
+
+        # text encode
+        h, h_lengths = self.encoder(token, token_len)
+        h = self.encoder_proj(h)
+        mel_len1, mel_len2 = prompt_feat.shape[1], int(token_len2 / self.input_frame_rate * 22050 / 256)
+        h, h_lengths = self.length_regulator.inference(h[:, :token_len1], h[:, token_len1:], mel_len1, mel_len2, self.input_frame_rate)
+
+        # get conditions
+        conds = torch.zeros([1, mel_len1 + mel_len2, self.output_size], device=token.device).to(h.dtype)
+        conds[:, :mel_len1] = prompt_feat
+        conds = conds.transpose(1, 2)
+
+        mask = (~make_pad_mask(torch.tensor([mel_len1 + mel_len2]))).to(h)
+        feat, flow_cache = self.decoder(
+            mu=h.transpose(1, 2).contiguous(),
+            mask=mask.unsqueeze(1),
+            spks=embedding,
+            cond=conds,
+            n_timesteps=10,
+            prompt_len=mel_len1,
+            flow_cache=flow_cache
+        )
+        feat = feat[:, :, mel_len1:]
+        assert feat.shape[2] == mel_len2
+        return feat.float(), flow_cache
+
+
+class CausalMaskedDiffWithXvec(torch.nn.Module):
+    def __init__(
+        self,
+        input_size: int = 512,
+        output_size: int = 80,
+        spk_embed_dim: int = 192,
+        output_type: str = "mel",
+        vocab_size: int = 6561,
+        input_frame_rate: int = 25,
+        only_mask_loss: bool = True,
+        token_mel_ratio: int = 2,
+        pre_lookahead_len: int = 3,
+        encoder: torch.nn.Module = None,
+        decoder: torch.nn.Module = None,
+        decoder_conf: Dict = {
+            'in_channels': 240,
+            'out_channel': 80,
+            'spk_emb_dim': 80,
+            'n_spks': 1,
+            'cfm_params': CFM_PARAMS,
+            'decoder_params': {
+                'channels': [256, 256],
+                'dropout': 0.0,
+                'attention_head_dim': 64,
+                'n_blocks': 4,
+                'num_mid_blocks': 12,
+                'num_heads': 8,
+                'act_fn': 'gelu',
+            }
+        },
+        mel_feat_conf: Dict = {
+            'n_fft': 1024,
+            'num_mels': 80,
+            'sampling_rate': 22050,
+            'hop_size': 256,
+            'win_size': 1024,
+            'fmin': 0,
+            'fmax': 8000
+        }
+    ):
+        super().__init__()
+        self.input_size = input_size
+        self.output_size = output_size
+        self.decoder_conf = decoder_conf
+        self.mel_feat_conf = mel_feat_conf
+        self.vocab_size = vocab_size
+        self.output_type = output_type
+        self.input_frame_rate = input_frame_rate
+        logging.info(f"input frame rate={self.input_frame_rate}")
+        self.input_embedding = nn.Embedding(vocab_size, input_size)
+        self.spk_embed_affine_layer = torch.nn.Linear(spk_embed_dim, output_size)
+        self.encoder = encoder
+        self.encoder_proj = torch.nn.Linear(self.encoder.output_size(), output_size)
+        self.decoder = decoder
+        self.only_mask_loss = only_mask_loss
+        self.token_mel_ratio = token_mel_ratio
+        self.pre_lookahead_len = pre_lookahead_len
+
+        # FIXME: this was missing - just putting it in as false
+        self.fp16 = False
+
+    @torch.inference_mode()
+    def inference(self,
+                  token,
+                  token_len,
+                  prompt_token,
+                  prompt_token_len,
+                  prompt_feat,
+                  prompt_feat_len,
+                  embedding,
+                  finalize):
+        if self.fp16 is True:
+            prompt_feat = prompt_feat.half()
+            embedding = embedding.half()
+
+        assert token.shape[0] == 1
+        # xvec projection
+        embedding = F.normalize(embedding, dim=1)
+        embedding = self.spk_embed_affine_layer(embedding)
+
+        # concat text and prompt_text
+        token, token_len = torch.concat([prompt_token, token], dim=1), prompt_token_len + token_len
+        mask = (~make_pad_mask(token_len)).unsqueeze(-1).to(embedding)
+        token = self.input_embedding(torch.clamp(token, min=0, max=self.input_embedding.num_embeddings-1)) * mask
+
+        # text encode
+        h, h_lengths = self.encoder(token, token_len)
+        if finalize is False:
+            h = h[:, :-self.pre_lookahead_len * self.token_mel_ratio]
+        mel_len1, mel_len2 = prompt_feat.shape[1], h.shape[1] - prompt_feat.shape[1]
+        h = self.encoder_proj(h)
+
+        # get conditions
+        conds = torch.zeros([1, mel_len1 + mel_len2, self.output_size], device=token.device).to(h.dtype)
+        conds[:, :mel_len1] = prompt_feat
+        conds = conds.transpose(1, 2)
+
+        mask = (~make_pad_mask(torch.tensor([mel_len1 + mel_len2]))).to(h)
+        feat, _ = self.decoder(
+            mu=h.transpose(1, 2).contiguous(),
+            mask=mask.unsqueeze(1),
+            spks=embedding,
+            cond=conds,
+            n_timesteps=10
+        )
+        feat = feat[:, :, mel_len1:]
+        assert feat.shape[2] == mel_len2
+        return feat.float(), None  # NOTE jrm: why are they returning None here?

engines/chatterbox_official_23lang/models/s3gen/flow_matching.py

@@ -0,0 +1,218 @@
+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Zhihao Du)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import threading
+import torch
+import torch.nn.functional as F
+from .matcha.flow_matching import BASECFM
+from .configs import CFM_PARAMS
+
+
+class ConditionalCFM(BASECFM):
+    def __init__(self, in_channels, cfm_params, n_spks=1, spk_emb_dim=64, estimator: torch.nn.Module = None):
+        super().__init__(
+            n_feats=in_channels,
+            cfm_params=cfm_params,
+            n_spks=n_spks,
+            spk_emb_dim=spk_emb_dim,
+        )
+        self.t_scheduler = cfm_params.t_scheduler
+        self.training_cfg_rate = cfm_params.training_cfg_rate
+        self.inference_cfg_rate = cfm_params.inference_cfg_rate
+        in_channels = in_channels + (spk_emb_dim if n_spks > 0 else 0)
+        # Just change the architecture of the estimator here
+        self.estimator = estimator
+        self.lock = threading.Lock()
+
+    @torch.inference_mode()
+    def forward(self, mu, mask, n_timesteps, temperature=1.0, spks=None, cond=None, prompt_len=0, flow_cache=torch.zeros(1, 80, 0, 2)):
+        """Forward diffusion
+
+        Args:
+            mu (torch.Tensor): output of encoder
+                shape: (batch_size, n_feats, mel_timesteps)
+            mask (torch.Tensor): output_mask
+                shape: (batch_size, 1, mel_timesteps)
+            n_timesteps (int): number of diffusion steps
+            temperature (float, optional): temperature for scaling noise. Defaults to 1.0.
+            spks (torch.Tensor, optional): speaker ids. Defaults to None.
+                shape: (batch_size, spk_emb_dim)
+            cond: Not used but kept for future purposes
+
+        Returns:
+            sample: generated mel-spectrogram
+                shape: (batch_size, n_feats, mel_timesteps)
+        """
+
+        z = torch.randn_like(mu).to(mu.device).to(mu.dtype) * temperature
+        cache_size = flow_cache.shape[2]
+        # fix prompt and overlap part mu and z
+        if cache_size != 0:
+            z[:, :, :cache_size] = flow_cache[:, :, :, 0]
+            mu[:, :, :cache_size] = flow_cache[:, :, :, 1]
+        z_cache = torch.concat([z[:, :, :prompt_len], z[:, :, -34:]], dim=2)
+        mu_cache = torch.concat([mu[:, :, :prompt_len], mu[:, :, -34:]], dim=2)
+        flow_cache = torch.stack([z_cache, mu_cache], dim=-1)
+
+        t_span = torch.linspace(0, 1, n_timesteps + 1, device=mu.device, dtype=mu.dtype)
+        if self.t_scheduler == 'cosine':
+            t_span = 1 - torch.cos(t_span * 0.5 * torch.pi)
+        return self.solve_euler(z, t_span=t_span, mu=mu, mask=mask, spks=spks, cond=cond), flow_cache
+
+    def solve_euler(self, x, t_span, mu, mask, spks, cond):
+        """
+        Fixed euler solver for ODEs.
+        Args:
+            x (torch.Tensor): random noise
+            t_span (torch.Tensor): n_timesteps interpolated
+                shape: (n_timesteps + 1,)
+            mu (torch.Tensor): output of encoder
+                shape: (batch_size, n_feats, mel_timesteps)
+            mask (torch.Tensor): output_mask
+                shape: (batch_size, 1, mel_timesteps)
+            spks (torch.Tensor, optional): speaker ids. Defaults to None.
+                shape: (batch_size, spk_emb_dim)
+            cond: Not used but kept for future purposes
+        """
+        t, _, dt = t_span[0], t_span[-1], t_span[1] - t_span[0]
+        t = t.unsqueeze(dim=0)
+
+        # I am storing this because I can later plot it by putting a debugger here and saving it to a file
+        # Or in future might add like a return_all_steps flag
+        sol = []
+
+        # Do not use concat, it may cause memory format changed and trt infer with wrong results!
+        x_in = torch.zeros([2, 80, x.size(2)], device=x.device, dtype=x.dtype)
+        mask_in = torch.zeros([2, 1, x.size(2)], device=x.device, dtype=x.dtype)
+        mu_in = torch.zeros([2, 80, x.size(2)], device=x.device, dtype=x.dtype)
+        t_in = torch.zeros([2], device=x.device, dtype=x.dtype)
+        spks_in = torch.zeros([2, 80], device=x.device, dtype=x.dtype)
+        cond_in = torch.zeros([2, 80, x.size(2)], device=x.device, dtype=x.dtype)
+        for step in range(1, len(t_span)):
+            # Classifier-Free Guidance inference introduced in VoiceBox
+            x_in[:] = x
+            mask_in[:] = mask
+            mu_in[0] = mu
+            t_in[:] = t.unsqueeze(0)
+            spks_in[0] = spks
+            cond_in[0] = cond
+            dphi_dt = self.forward_estimator(
+                x_in, mask_in,
+                mu_in, t_in,
+                spks_in,
+                cond_in
+            )
+            dphi_dt, cfg_dphi_dt = torch.split(dphi_dt, [x.size(0), x.size(0)], dim=0)
+            dphi_dt = ((1.0 + self.inference_cfg_rate) * dphi_dt - self.inference_cfg_rate * cfg_dphi_dt)
+            x = x + dt * dphi_dt
+            t = t + dt
+            sol.append(x)
+            if step < len(t_span) - 1:
+                dt = t_span[step + 1] - t
+
+        return sol[-1].float()
+
+    def forward_estimator(self, x, mask, mu, t, spks, cond):
+        if isinstance(self.estimator, torch.nn.Module):
+            return self.estimator.forward(x, mask, mu, t, spks, cond)
+        else:
+            with self.lock:
+                self.estimator.set_input_shape('x', (2, 80, x.size(2)))
+                self.estimator.set_input_shape('mask', (2, 1, x.size(2)))
+                self.estimator.set_input_shape('mu', (2, 80, x.size(2)))
+                self.estimator.set_input_shape('t', (2,))
+                self.estimator.set_input_shape('spks', (2, 80))
+                self.estimator.set_input_shape('cond', (2, 80, x.size(2)))
+                # run trt engine
+                self.estimator.execute_v2([x.contiguous().data_ptr(),
+                                           mask.contiguous().data_ptr(),
+                                           mu.contiguous().data_ptr(),
+                                           t.contiguous().data_ptr(),
+                                           spks.contiguous().data_ptr(),
+                                           cond.contiguous().data_ptr(),
+                                           x.data_ptr()])
+            return x
+
+    def compute_loss(self, x1, mask, mu, spks=None, cond=None):
+        """Computes diffusion loss
+
+        Args:
+            x1 (torch.Tensor): Target
+                shape: (batch_size, n_feats, mel_timesteps)
+            mask (torch.Tensor): target mask
+                shape: (batch_size, 1, mel_timesteps)
+            mu (torch.Tensor): output of encoder
+                shape: (batch_size, n_feats, mel_timesteps)
+            spks (torch.Tensor, optional): speaker embedding. Defaults to None.
+                shape: (batch_size, spk_emb_dim)
+
+        Returns:
+            loss: conditional flow matching loss
+            y: conditional flow
+                shape: (batch_size, n_feats, mel_timesteps)
+        """
+        b, _, t = mu.shape
+
+        # random timestep
+        t = torch.rand([b, 1, 1], device=mu.device, dtype=mu.dtype)
+        if self.t_scheduler == 'cosine':
+            t = 1 - torch.cos(t * 0.5 * torch.pi)
+        # sample noise p(x_0)
+        z = torch.randn_like(x1)
+
+        y = (1 - (1 - self.sigma_min) * t) * z + t * x1
+        u = x1 - (1 - self.sigma_min) * z
+
+        # during training, we randomly drop condition to trade off mode coverage and sample fidelity
+        if self.training_cfg_rate > 0:
+            cfg_mask = torch.rand(b, device=x1.device) > self.training_cfg_rate
+            mu = mu * cfg_mask.view(-1, 1, 1)
+            spks = spks * cfg_mask.view(-1, 1)
+            cond = cond * cfg_mask.view(-1, 1, 1)
+
+        pred = self.estimator(y, mask, mu, t.squeeze(), spks, cond)
+        loss = F.mse_loss(pred * mask, u * mask, reduction="sum") / (torch.sum(mask) * u.shape[1])
+        return loss, y
+
+
+class CausalConditionalCFM(ConditionalCFM):
+    def __init__(self, in_channels=240, cfm_params=CFM_PARAMS, n_spks=1, spk_emb_dim=80, estimator=None):
+        super().__init__(in_channels, cfm_params, n_spks, spk_emb_dim, estimator)
+        self.rand_noise = torch.randn([1, 80, 50 * 300])
+
+    @torch.inference_mode()
+    def forward(self, mu, mask, n_timesteps, temperature=1.0, spks=None, cond=None):
+        """Forward diffusion
+
+        Args:
+            mu (torch.Tensor): output of encoder
+                shape: (batch_size, n_feats, mel_timesteps)
+            mask (torch.Tensor): output_mask
+                shape: (batch_size, 1, mel_timesteps)
+            n_timesteps (int): number of diffusion steps
+            temperature (float, optional): temperature for scaling noise. Defaults to 1.0.
+            spks (torch.Tensor, optional): speaker ids. Defaults to None.
+                shape: (batch_size, spk_emb_dim)
+            cond: Not used but kept for future purposes
+
+        Returns:
+            sample: generated mel-spectrogram
+                shape: (batch_size, n_feats, mel_timesteps)
+        """
+
+        z = self.rand_noise[:, :, :mu.size(2)].to(mu.device).to(mu.dtype) * temperature
+        # fix prompt and overlap part mu and z
+        t_span = torch.linspace(0, 1, n_timesteps + 1, device=mu.device, dtype=mu.dtype)
+        if self.t_scheduler == 'cosine':
+            t_span = 1 - torch.cos(t_span * 0.5 * torch.pi)
+        return self.solve_euler(z, t_span=t_span, mu=mu, mask=mask, spks=spks, cond=cond), None

engines/chatterbox_official_23lang/models/s3gen/hifigan.py

@@ -0,0 +1,474 @@
+# jrm: adapted from CosyVoice/cosyvoice/hifigan/generator.py
+#      most modules should be reusable, but I found their SineGen changed a git.
+
+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Kai Hu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""HIFI-GAN"""
+
+from typing import Dict, Optional, List
+import numpy as np
+from scipy.signal import get_window
+import torch
+import torch.nn.functional as F
+from torch.nn import Conv1d
+from torch.nn import ConvTranspose1d
+from torch.nn.utils import remove_weight_norm
+from torch.nn.utils.parametrizations import weight_norm
+from torch.distributions.uniform import Uniform
+from torch import nn, sin, pow
+from torch.nn import Parameter
+
+
+class Snake(nn.Module):
+    '''
+    Implementation of a sine-based periodic activation function
+    Shape:
+        - Input: (B, C, T)
+        - Output: (B, C, T), same shape as the input
+    Parameters:
+        - alpha - trainable parameter
+    References:
+        - This activation function is from this paper by Liu Ziyin, Tilman Hartwig, Masahito Ueda:
+        https://arxiv.org/abs/2006.08195
+    Examples:
+        >>> a1 = snake(256)
+        >>> x = torch.randn(256)
+        >>> x = a1(x)
+    '''
+    def __init__(self, in_features, alpha=1.0, alpha_trainable=True, alpha_logscale=False):
+        '''
+        Initialization.
+        INPUT:
+            - in_features: shape of the input
+            - alpha: trainable parameter
+            alpha is initialized to 1 by default, higher values = higher-frequency.
+            alpha will be trained along with the rest of your model.
+        '''
+        super(Snake, self).__init__()
+        self.in_features = in_features
+
+        # initialize alpha
+        self.alpha_logscale = alpha_logscale
+        if self.alpha_logscale: # log scale alphas initialized to zeros
+            self.alpha = Parameter(torch.zeros(in_features) * alpha)
+        else: # linear scale alphas initialized to ones
+            self.alpha = Parameter(torch.ones(in_features) * alpha)
+
+        self.alpha.requires_grad = alpha_trainable
+
+        self.no_div_by_zero = 0.000000001
+
+    def forward(self, x):
+        '''
+        Forward pass of the function.
+        Applies the function to the input elementwise.
+        Snake ∶= x + 1/a * sin^2 (xa)
+        '''
+        alpha = self.alpha.unsqueeze(0).unsqueeze(-1) # line up with x to [B, C, T]
+        if self.alpha_logscale:
+            alpha = torch.exp(alpha)
+        x = x + (1.0 / (alpha + self.no_div_by_zero)) * pow(sin(x * alpha), 2)
+
+        return x
+
+
+
+def get_padding(kernel_size, dilation=1):
+    return int((kernel_size * dilation - dilation) / 2)
+
+def init_weights(m, mean=0.0, std=0.01):
+    classname = m.__class__.__name__
+    if classname.find("Conv") != -1:
+        m.weight.data.normal_(mean, std)
+
+
+"""hifigan based generator implementation.
+
+This code is modified from https://github.com/jik876/hifi-gan
+ ,https://github.com/kan-bayashi/ParallelWaveGAN and
+ https://github.com/NVIDIA/BigVGAN
+
+"""
+
+
+class ResBlock(torch.nn.Module):
+    """Residual block module in HiFiGAN/BigVGAN."""
+    def __init__(
+        self,
+        channels: int = 512,
+        kernel_size: int = 3,
+        dilations: List[int] = [1, 3, 5],
+    ):
+        super(ResBlock, self).__init__()
+        self.convs1 = nn.ModuleList()
+        self.convs2 = nn.ModuleList()
+
+        for dilation in dilations:
+            self.convs1.append(
+                weight_norm(
+                    Conv1d(
+                        channels,
+                        channels,
+                        kernel_size,
+                        1,
+                        dilation=dilation,
+                        padding=get_padding(kernel_size, dilation)
+                    )
+                )
+            )
+            self.convs2.append(
+                weight_norm(
+                    Conv1d(
+                        channels,
+                        channels,
+                        kernel_size,
+                        1,
+                        dilation=1,
+                        padding=get_padding(kernel_size, 1)
+                    )
+                )
+            )
+        self.convs1.apply(init_weights)
+        self.convs2.apply(init_weights)
+        self.activations1 = nn.ModuleList([
+            Snake(channels, alpha_logscale=False)
+            for _ in range(len(self.convs1))
+        ])
+        self.activations2 = nn.ModuleList([
+            Snake(channels, alpha_logscale=False)
+            for _ in range(len(self.convs2))
+        ])
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        for idx in range(len(self.convs1)):
+            xt = self.activations1[idx](x)
+            xt = self.convs1[idx](xt)
+            xt = self.activations2[idx](xt)
+            xt = self.convs2[idx](xt)
+            x = xt + x
+        return x
+
+    def remove_weight_norm(self):
+        for idx in range(len(self.convs1)):
+            remove_weight_norm(self.convs1[idx])
+            remove_weight_norm(self.convs2[idx])
+
+
+class SineGen(torch.nn.Module):
+    """ Definition of sine generator
+    SineGen(samp_rate, harmonic_num = 0,
+            sine_amp = 0.1, noise_std = 0.003,
+            voiced_threshold = 0,
+            flag_for_pulse=False)
+    samp_rate: sampling rate in Hz
+    harmonic_num: number of harmonic overtones (default 0)
+    sine_amp: amplitude of sine-wavefrom (default 0.1)
+    noise_std: std of Gaussian noise (default 0.003)
+    voiced_thoreshold: F0 threshold for U/V classification (default 0)
+    flag_for_pulse: this SinGen is used inside PulseGen (default False)
+    Note: when flag_for_pulse is True, the first time step of a voiced
+        segment is always sin(np.pi) or cos(0)
+    """
+
+    def __init__(self, samp_rate, harmonic_num=0,
+                 sine_amp=0.1, noise_std=0.003,
+                 voiced_threshold=0):
+        super(SineGen, self).__init__()
+        self.sine_amp = sine_amp
+        self.noise_std = noise_std
+        self.harmonic_num = harmonic_num
+        self.sampling_rate = samp_rate
+        self.voiced_threshold = voiced_threshold
+
+    def _f02uv(self, f0):
+        # generate uv signal
+        uv = (f0 > self.voiced_threshold).type(torch.float32)
+        return uv
+
+    @torch.no_grad()
+    def forward(self, f0):
+        """
+        :param f0: [B, 1, sample_len], Hz
+        :return: [B, 1, sample_len]
+        """
+
+        F_mat = torch.zeros((f0.size(0), self.harmonic_num + 1, f0.size(-1))).to(f0.device)
+        for i in range(self.harmonic_num + 1):
+            F_mat[:, i: i + 1, :] = f0 * (i + 1) / self.sampling_rate
+
+        theta_mat = 2 * np.pi * (torch.cumsum(F_mat, dim=-1) % 1)
+        u_dist = Uniform(low=-np.pi, high=np.pi)
+        phase_vec = u_dist.sample(sample_shape=(f0.size(0), self.harmonic_num + 1, 1)).to(F_mat.device)
+        phase_vec[:, 0, :] = 0
+
+        # generate sine waveforms
+        sine_waves = self.sine_amp * torch.sin(theta_mat + phase_vec)
+
+        # generate uv signal
+        uv = self._f02uv(f0)
+
+        # noise: for unvoiced should be similar to sine_amp
+        #        std = self.sine_amp/3 -> max value ~ self.sine_amp
+        # .       for voiced regions is self.noise_std
+        noise_amp = uv * self.noise_std + (1 - uv) * self.sine_amp / 3
+        noise = noise_amp * torch.randn_like(sine_waves)
+
+        # first: set the unvoiced part to 0 by uv
+        # then: additive noise
+        sine_waves = sine_waves * uv + noise
+        return sine_waves, uv, noise
+
+
+class SourceModuleHnNSF(torch.nn.Module):
+    """ SourceModule for hn-nsf
+    SourceModule(sampling_rate, harmonic_num=0, sine_amp=0.1,
+                 add_noise_std=0.003, voiced_threshod=0)
+    sampling_rate: sampling_rate in Hz
+    harmonic_num: number of harmonic above F0 (default: 0)
+    sine_amp: amplitude of sine source signal (default: 0.1)
+    add_noise_std: std of additive Gaussian noise (default: 0.003)
+        note that amplitude of noise in unvoiced is decided
+        by sine_amp
+    voiced_threshold: threhold to set U/V given F0 (default: 0)
+    Sine_source, noise_source = SourceModuleHnNSF(F0_sampled)
+    F0_sampled (batchsize, length, 1)
+    Sine_source (batchsize, length, 1)
+    noise_source (batchsize, length 1)
+    uv (batchsize, length, 1)
+    """
+
+    def __init__(self, sampling_rate, upsample_scale, harmonic_num=0, sine_amp=0.1,
+                 add_noise_std=0.003, voiced_threshod=0):
+        super(SourceModuleHnNSF, self).__init__()
+
+        self.sine_amp = sine_amp
+        self.noise_std = add_noise_std
+
+        # to produce sine waveforms
+        self.l_sin_gen = SineGen(sampling_rate, harmonic_num,
+                                 sine_amp, add_noise_std, voiced_threshod)
+
+        # to merge source harmonics into a single excitation
+        self.l_linear = torch.nn.Linear(harmonic_num + 1, 1)
+        self.l_tanh = torch.nn.Tanh()
+
+    def forward(self, x):
+        """
+        Sine_source, noise_source = SourceModuleHnNSF(F0_sampled)
+        F0_sampled (batchsize, length, 1)
+        Sine_source (batchsize, length, 1)
+        noise_source (batchsize, length 1)
+        """
+        # source for harmonic branch
+        with torch.no_grad():
+            sine_wavs, uv, _ = self.l_sin_gen(x.transpose(1, 2))
+            sine_wavs = sine_wavs.transpose(1, 2)
+            uv = uv.transpose(1, 2)
+        sine_merge = self.l_tanh(self.l_linear(sine_wavs))
+
+        # source for noise branch, in the same shape as uv
+        noise = torch.randn_like(uv) * self.sine_amp / 3
+        return sine_merge, noise, uv
+
+
+class HiFTGenerator(nn.Module):
+    """
+    HiFTNet Generator: Neural Source Filter + ISTFTNet
+    https://arxiv.org/abs/2309.09493
+    """
+    def __init__(
+            self,
+            in_channels: int = 80,
+            base_channels: int = 512,
+            nb_harmonics: int = 8,
+            sampling_rate: int = 22050,
+            nsf_alpha: float = 0.1,
+            nsf_sigma: float = 0.003,
+            nsf_voiced_threshold: float = 10,
+            upsample_rates: List[int] = [8, 8],
+            upsample_kernel_sizes: List[int] = [16, 16],
+            istft_params: Dict[str, int] = {"n_fft": 16, "hop_len": 4},
+            resblock_kernel_sizes: List[int] = [3, 7, 11],
+            resblock_dilation_sizes: List[List[int]] = [[1, 3, 5], [1, 3, 5], [1, 3, 5]],
+            source_resblock_kernel_sizes: List[int] = [7, 11],
+            source_resblock_dilation_sizes: List[List[int]] = [[1, 3, 5], [1, 3, 5]],
+            lrelu_slope: float = 0.1,
+            audio_limit: float = 0.99,
+            f0_predictor: torch.nn.Module = None,
+    ):
+        super(HiFTGenerator, self).__init__()
+
+        self.out_channels = 1
+        self.nb_harmonics = nb_harmonics
+        self.sampling_rate = sampling_rate
+        self.istft_params = istft_params
+        self.lrelu_slope = lrelu_slope
+        self.audio_limit = audio_limit
+
+        self.num_kernels = len(resblock_kernel_sizes)
+        self.num_upsamples = len(upsample_rates)
+        self.m_source = SourceModuleHnNSF(
+            sampling_rate=sampling_rate,
+            upsample_scale=np.prod(upsample_rates) * istft_params["hop_len"],
+            harmonic_num=nb_harmonics,
+            sine_amp=nsf_alpha,
+            add_noise_std=nsf_sigma,
+            voiced_threshod=nsf_voiced_threshold)
+        self.f0_upsamp = torch.nn.Upsample(scale_factor=np.prod(upsample_rates) * istft_params["hop_len"])
+
+        self.conv_pre = weight_norm(
+            Conv1d(in_channels, base_channels, 7, 1, padding=3)
+        )
+
+        # Up
+        self.ups = nn.ModuleList()
+        for i, (u, k) in enumerate(zip(upsample_rates, upsample_kernel_sizes)):
+            self.ups.append(
+                weight_norm(
+                    ConvTranspose1d(
+                        base_channels // (2**i),
+                        base_channels // (2**(i + 1)),
+                        k,
+                        u,
+                        padding=(k - u) // 2,
+                    )
+                )
+            )
+
+        # Down
+        self.source_downs = nn.ModuleList()
+        self.source_resblocks = nn.ModuleList()
+        downsample_rates = [1] + upsample_rates[::-1][:-1]
+        downsample_cum_rates = np.cumprod(downsample_rates)
+        for i, (u, k, d) in enumerate(zip(downsample_cum_rates[::-1], source_resblock_kernel_sizes, source_resblock_dilation_sizes)):
+            if u == 1:
+                self.source_downs.append(
+                    Conv1d(istft_params["n_fft"] + 2, base_channels // (2 ** (i + 1)), 1, 1)
+                )
+            else:
+                self.source_downs.append(
+                    Conv1d(istft_params["n_fft"] + 2, base_channels // (2 ** (i + 1)), u * 2, u, padding=(u // 2))
+                )
+
+            self.source_resblocks.append(
+                ResBlock(base_channels // (2 ** (i + 1)), k, d)
+            )
+
+        self.resblocks = nn.ModuleList()
+        for i in range(len(self.ups)):
+            ch = base_channels // (2**(i + 1))
+            for _, (k, d) in enumerate(zip(resblock_kernel_sizes, resblock_dilation_sizes)):
+                self.resblocks.append(ResBlock(ch, k, d))
+
+        self.conv_post = weight_norm(Conv1d(ch, istft_params["n_fft"] + 2, 7, 1, padding=3))
+        self.ups.apply(init_weights)
+        self.conv_post.apply(init_weights)
+        self.reflection_pad = nn.ReflectionPad1d((1, 0))
+        self.stft_window = torch.from_numpy(get_window("hann", istft_params["n_fft"], fftbins=True).astype(np.float32))
+        self.f0_predictor = f0_predictor
+
+    def remove_weight_norm(self):
+        print('Removing weight norm...')
+        for l in self.ups:
+            remove_weight_norm(l)
+        for l in self.resblocks:
+            l.remove_weight_norm()
+        remove_weight_norm(self.conv_pre)
+        remove_weight_norm(self.conv_post)
+        self.m_source.remove_weight_norm()
+        for l in self.source_downs:
+            remove_weight_norm(l)
+        for l in self.source_resblocks:
+            l.remove_weight_norm()
+
+    def _stft(self, x):
+        spec = torch.stft(
+            x,
+            self.istft_params["n_fft"], self.istft_params["hop_len"], self.istft_params["n_fft"], window=self.stft_window.to(x.device),
+            return_complex=True)
+        spec = torch.view_as_real(spec)  # [B, F, TT, 2]
+        return spec[..., 0], spec[..., 1]
+
+    def _istft(self, magnitude, phase):
+        magnitude = torch.clip(magnitude, max=1e2)
+        real = magnitude * torch.cos(phase)
+        img = magnitude * torch.sin(phase)
+        inverse_transform = torch.istft(torch.complex(real, img), self.istft_params["n_fft"], self.istft_params["hop_len"],
+                                        self.istft_params["n_fft"], window=self.stft_window.to(magnitude.device))
+        return inverse_transform
+
+    def decode(self, x: torch.Tensor, s: torch.Tensor = torch.zeros(1, 1, 0)) -> torch.Tensor:
+        s_stft_real, s_stft_imag = self._stft(s.squeeze(1))
+        s_stft = torch.cat([s_stft_real, s_stft_imag], dim=1)
+
+        x = self.conv_pre(x)
+        for i in range(self.num_upsamples):
+            x = F.leaky_relu(x, self.lrelu_slope)
+            x = self.ups[i](x)
+
+            if i == self.num_upsamples - 1:
+                x = self.reflection_pad(x)
+
+            # fusion
+            si = self.source_downs[i](s_stft)
+            si = self.source_resblocks[i](si)
+            x = x + si
+
+            xs = None
+            for j in range(self.num_kernels):
+                if xs is None:
+                    xs = self.resblocks[i * self.num_kernels + j](x)
+                else:
+                    xs += self.resblocks[i * self.num_kernels + j](x)
+            x = xs / self.num_kernels
+
+        x = F.leaky_relu(x)
+        x = self.conv_post(x)
+        magnitude = torch.exp(x[:, :self.istft_params["n_fft"] // 2 + 1, :])
+        phase = torch.sin(x[:, self.istft_params["n_fft"] // 2 + 1:, :])  # actually, sin is redundancy
+
+        x = self._istft(magnitude, phase)
+        x = torch.clamp(x, -self.audio_limit, self.audio_limit)
+        return x
+
+    def forward(
+            self,
+            batch: dict,
+            device: torch.device,
+    ) -> Dict[str, Optional[torch.Tensor]]:
+        speech_feat = batch['speech_feat'].transpose(1, 2).to(device)
+        # mel->f0
+        f0 = self.f0_predictor(speech_feat)
+        # f0->source
+        s = self.f0_upsamp(f0[:, None]).transpose(1, 2)  # bs,n,t
+        s, _, _ = self.m_source(s)
+        s = s.transpose(1, 2)
+        # mel+source->speech
+        generated_speech = self.decode(x=speech_feat, s=s)
+        return generated_speech, f0
+
+    @torch.inference_mode()
+    def inference(self, speech_feat: torch.Tensor, cache_source: torch.Tensor = torch.zeros(1, 1, 0)) -> torch.Tensor:
+        # mel->f0
+        f0 = self.f0_predictor(speech_feat)
+        # f0->source
+        s = self.f0_upsamp(f0[:, None]).transpose(1, 2)  # bs,n,t
+        s, _, _ = self.m_source(s)
+        s = s.transpose(1, 2)
+        # use cache_source to avoid glitch
+        if cache_source.shape[2] != 0:
+            s[:, :, :cache_source.shape[2]] = cache_source
+        generated_speech = self.decode(x=speech_feat, s=s)
+        return generated_speech, s

engines/chatterbox_official_23lang/models/s3gen/matcha/decoder.py

@@ -0,0 +1,443 @@
+import math
+from typing import Optional
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from conformer import ConformerBlock
+from diffusers.models.activations import get_activation
+from einops import pack, rearrange, repeat
+
+from .transformer import BasicTransformerBlock
+
+
+class SinusoidalPosEmb(torch.nn.Module):
+    def __init__(self, dim):
+        super().__init__()
+        self.dim = dim
+        assert self.dim % 2 == 0, "SinusoidalPosEmb requires dim to be even"
+
+    def forward(self, x, scale=1000):
+        if x.ndim < 1:
+            x = x.unsqueeze(0)
+        device = x.device
+        half_dim = self.dim // 2
+        emb = math.log(10000) / (half_dim - 1)
+        emb = torch.exp(torch.arange(half_dim, device=device).float() * -emb)
+        emb = scale * x.unsqueeze(1) * emb.unsqueeze(0)
+        emb = torch.cat((emb.sin(), emb.cos()), dim=-1)
+        return emb
+
+
+class Block1D(torch.nn.Module):
+    def __init__(self, dim, dim_out, groups=8):
+        super().__init__()
+        self.block = torch.nn.Sequential(
+            torch.nn.Conv1d(dim, dim_out, 3, padding=1),
+            torch.nn.GroupNorm(groups, dim_out),
+            nn.Mish(),
+        )
+
+    def forward(self, x, mask):
+        output = self.block(x * mask)
+        return output * mask
+
+
+class ResnetBlock1D(torch.nn.Module):
+    def __init__(self, dim, dim_out, time_emb_dim, groups=8):
+        super().__init__()
+        self.mlp = torch.nn.Sequential(nn.Mish(), torch.nn.Linear(time_emb_dim, dim_out))
+
+        self.block1 = Block1D(dim, dim_out, groups=groups)
+        self.block2 = Block1D(dim_out, dim_out, groups=groups)
+
+        self.res_conv = torch.nn.Conv1d(dim, dim_out, 1)
+
+    def forward(self, x, mask, time_emb):
+        h = self.block1(x, mask)
+        h += self.mlp(time_emb).unsqueeze(-1)
+        h = self.block2(h, mask)
+        output = h + self.res_conv(x * mask)
+        return output
+
+
+class Downsample1D(nn.Module):
+    def __init__(self, dim):
+        super().__init__()
+        self.conv = torch.nn.Conv1d(dim, dim, 3, 2, 1)
+
+    def forward(self, x):
+        return self.conv(x)
+
+
+class TimestepEmbedding(nn.Module):
+    def __init__(
+        self,
+        in_channels: int,
+        time_embed_dim: int,
+        act_fn: str = "silu",
+        out_dim: int = None,
+        post_act_fn: Optional[str] = None,
+        cond_proj_dim=None,
+    ):
+        super().__init__()
+
+        self.linear_1 = nn.Linear(in_channels, time_embed_dim)
+
+        if cond_proj_dim is not None:
+            self.cond_proj = nn.Linear(cond_proj_dim, in_channels, bias=False)
+        else:
+            self.cond_proj = None
+
+        self.act = get_activation(act_fn)
+
+        if out_dim is not None:
+            time_embed_dim_out = out_dim
+        else:
+            time_embed_dim_out = time_embed_dim
+        self.linear_2 = nn.Linear(time_embed_dim, time_embed_dim_out)
+
+        if post_act_fn is None:
+            self.post_act = None
+        else:
+            self.post_act = get_activation(post_act_fn)
+
+    def forward(self, sample, condition=None):
+        if condition is not None:
+            sample = sample + self.cond_proj(condition)
+        sample = self.linear_1(sample)
+
+        if self.act is not None:
+            sample = self.act(sample)
+
+        sample = self.linear_2(sample)
+
+        if self.post_act is not None:
+            sample = self.post_act(sample)
+        return sample
+
+
+class Upsample1D(nn.Module):
+    """A 1D upsampling layer with an optional convolution.
+
+    Parameters:
+        channels (`int`):
+            number of channels in the inputs and outputs.
+        use_conv (`bool`, default `False`):
+            option to use a convolution.
+        use_conv_transpose (`bool`, default `False`):
+            option to use a convolution transpose.
+        out_channels (`int`, optional):
+            number of output channels. Defaults to `channels`.
+    """
+
+    def __init__(self, channels, use_conv=False, use_conv_transpose=True, out_channels=None, name="conv"):
+        super().__init__()
+        self.channels = channels
+        self.out_channels = out_channels or channels
+        self.use_conv = use_conv
+        self.use_conv_transpose = use_conv_transpose
+        self.name = name
+
+        self.conv = None
+        if use_conv_transpose:
+            self.conv = nn.ConvTranspose1d(channels, self.out_channels, 4, 2, 1)
+        elif use_conv:
+            self.conv = nn.Conv1d(self.channels, self.out_channels, 3, padding=1)
+
+    def forward(self, inputs):
+        assert inputs.shape[1] == self.channels
+        if self.use_conv_transpose:
+            return self.conv(inputs)
+
+        outputs = F.interpolate(inputs, scale_factor=2.0, mode="nearest")
+
+        if self.use_conv:
+            outputs = self.conv(outputs)
+
+        return outputs
+
+
+class ConformerWrapper(ConformerBlock):
+    def __init__(  # pylint: disable=useless-super-delegation
+        self,
+        *,
+        dim,
+        dim_head=64,
+        heads=8,
+        ff_mult=4,
+        conv_expansion_factor=2,
+        conv_kernel_size=31,
+        attn_dropout=0,
+        ff_dropout=0,
+        conv_dropout=0,
+        conv_causal=False,
+    ):
+        super().__init__(
+            dim=dim,
+            dim_head=dim_head,
+            heads=heads,
+            ff_mult=ff_mult,
+            conv_expansion_factor=conv_expansion_factor,
+            conv_kernel_size=conv_kernel_size,
+            attn_dropout=attn_dropout,
+            ff_dropout=ff_dropout,
+            conv_dropout=conv_dropout,
+            conv_causal=conv_causal,
+        )
+
+    def forward(
+        self,
+        hidden_states,
+        attention_mask,
+        encoder_hidden_states=None,
+        encoder_attention_mask=None,
+        timestep=None,
+    ):
+        return super().forward(x=hidden_states, mask=attention_mask.bool())
+
+
+class Decoder(nn.Module):
+    def __init__(
+        self,
+        in_channels,
+        out_channels,
+        channels=(256, 256),
+        dropout=0.05,
+        attention_head_dim=64,
+        n_blocks=1,
+        num_mid_blocks=2,
+        num_heads=4,
+        act_fn="snake",
+        down_block_type="transformer",
+        mid_block_type="transformer",
+        up_block_type="transformer",
+    ):
+        super().__init__()
+        channels = tuple(channels)
+        self.in_channels = in_channels
+        self.out_channels = out_channels
+
+        self.time_embeddings = SinusoidalPosEmb(in_channels)
+        time_embed_dim = channels[0] * 4
+        self.time_mlp = TimestepEmbedding(
+            in_channels=in_channels,
+            time_embed_dim=time_embed_dim,
+            act_fn="silu",
+        )
+
+        self.down_blocks = nn.ModuleList([])
+        self.mid_blocks = nn.ModuleList([])
+        self.up_blocks = nn.ModuleList([])
+
+        output_channel = in_channels
+        for i in range(len(channels)):  # pylint: disable=consider-using-enumerate
+            input_channel = output_channel
+            output_channel = channels[i]
+            is_last = i == len(channels) - 1
+            resnet = ResnetBlock1D(dim=input_channel, dim_out=output_channel, time_emb_dim=time_embed_dim)
+            transformer_blocks = nn.ModuleList(
+                [
+                    self.get_block(
+                        down_block_type,
+                        output_channel,
+                        attention_head_dim,
+                        num_heads,
+                        dropout,
+                        act_fn,
+                    )
+                    for _ in range(n_blocks)
+                ]
+            )
+            downsample = (
+                Downsample1D(output_channel) if not is_last else nn.Conv1d(output_channel, output_channel, 3, padding=1)
+            )
+
+            self.down_blocks.append(nn.ModuleList([resnet, transformer_blocks, downsample]))
+
+        for i in range(num_mid_blocks):
+            input_channel = channels[-1]
+            out_channels = channels[-1]
+
+            resnet = ResnetBlock1D(dim=input_channel, dim_out=output_channel, time_emb_dim=time_embed_dim)
+
+            transformer_blocks = nn.ModuleList(
+                [
+                    self.get_block(
+                        mid_block_type,
+                        output_channel,
+                        attention_head_dim,
+                        num_heads,
+                        dropout,
+                        act_fn,
+                    )
+                    for _ in range(n_blocks)
+                ]
+            )
+
+            self.mid_blocks.append(nn.ModuleList([resnet, transformer_blocks]))
+
+        channels = channels[::-1] + (channels[0],)
+        for i in range(len(channels) - 1):
+            input_channel = channels[i]
+            output_channel = channels[i + 1]
+            is_last = i == len(channels) - 2
+
+            resnet = ResnetBlock1D(
+                dim=2 * input_channel,
+                dim_out=output_channel,
+                time_emb_dim=time_embed_dim,
+            )
+            transformer_blocks = nn.ModuleList(
+                [
+                    self.get_block(
+                        up_block_type,
+                        output_channel,
+                        attention_head_dim,
+                        num_heads,
+                        dropout,
+                        act_fn,
+                    )
+                    for _ in range(n_blocks)
+                ]
+            )
+            upsample = (
+                Upsample1D(output_channel, use_conv_transpose=True)
+                if not is_last
+                else nn.Conv1d(output_channel, output_channel, 3, padding=1)
+            )
+
+            self.up_blocks.append(nn.ModuleList([resnet, transformer_blocks, upsample]))
+
+        self.final_block = Block1D(channels[-1], channels[-1])
+        self.final_proj = nn.Conv1d(channels[-1], self.out_channels, 1)
+
+        self.initialize_weights()
+        # nn.init.normal_(self.final_proj.weight)
+
+    @staticmethod
+    def get_block(block_type, dim, attention_head_dim, num_heads, dropout, act_fn):
+        if block_type == "conformer":
+            block = ConformerWrapper(
+                dim=dim,
+                dim_head=attention_head_dim,
+                heads=num_heads,
+                ff_mult=1,
+                conv_expansion_factor=2,
+                ff_dropout=dropout,
+                attn_dropout=dropout,
+                conv_dropout=dropout,
+                conv_kernel_size=31,
+            )
+        elif block_type == "transformer":
+            block = BasicTransformerBlock(
+                dim=dim,
+                num_attention_heads=num_heads,
+                attention_head_dim=attention_head_dim,
+                dropout=dropout,
+                activation_fn=act_fn,
+            )
+        else:
+            raise ValueError(f"Unknown block type {block_type}")
+
+        return block
+
+    def initialize_weights(self):
+        for m in self.modules():
+            if isinstance(m, nn.Conv1d):
+                nn.init.kaiming_normal_(m.weight, nonlinearity="relu")
+
+                if m.bias is not None:
+                    nn.init.constant_(m.bias, 0)
+
+            elif isinstance(m, nn.GroupNorm):
+                nn.init.constant_(m.weight, 1)
+                nn.init.constant_(m.bias, 0)
+
+            elif isinstance(m, nn.Linear):
+                nn.init.kaiming_normal_(m.weight, nonlinearity="relu")
+
+                if m.bias is not None:
+                    nn.init.constant_(m.bias, 0)
+
+    def forward(self, x, mask, mu, t, spks=None, cond=None):
+        """Forward pass of the UNet1DConditional model.
+
+        Args:
+            x (torch.Tensor): shape (batch_size, in_channels, time)
+            mask (_type_): shape (batch_size, 1, time)
+            t (_type_): shape (batch_size)
+            spks (_type_, optional): shape: (batch_size, condition_channels). Defaults to None.
+            cond (_type_, optional): placeholder for future use. Defaults to None.
+
+        Raises:
+            ValueError: _description_
+            ValueError: _description_
+
+        Returns:
+            _type_: _description_
+        """
+
+        t = self.time_embeddings(t)
+        t = self.time_mlp(t)
+
+        x = pack([x, mu], "b * t")[0]
+
+        if spks is not None:
+            spks = repeat(spks, "b c -> b c t", t=x.shape[-1])
+            x = pack([x, spks], "b * t")[0]
+
+        hiddens = []
+        masks = [mask]
+        for resnet, transformer_blocks, downsample in self.down_blocks:
+            mask_down = masks[-1]
+            x = resnet(x, mask_down, t)
+            x = rearrange(x, "b c t -> b t c")
+            mask_down = rearrange(mask_down, "b 1 t -> b t")
+            for transformer_block in transformer_blocks:
+                x = transformer_block(
+                    hidden_states=x,
+                    attention_mask=mask_down,
+                    timestep=t,
+                )
+            x = rearrange(x, "b t c -> b c t")
+            mask_down = rearrange(mask_down, "b t -> b 1 t")
+            hiddens.append(x)  # Save hidden states for skip connections
+            x = downsample(x * mask_down)
+            masks.append(mask_down[:, :, ::2])
+
+        masks = masks[:-1]
+        mask_mid = masks[-1]
+
+        for resnet, transformer_blocks in self.mid_blocks:
+            x = resnet(x, mask_mid, t)
+            x = rearrange(x, "b c t -> b t c")
+            mask_mid = rearrange(mask_mid, "b 1 t -> b t")
+            for transformer_block in transformer_blocks:
+                x = transformer_block(
+                    hidden_states=x,
+                    attention_mask=mask_mid,
+                    timestep=t,
+                )
+            x = rearrange(x, "b t c -> b c t")
+            mask_mid = rearrange(mask_mid, "b t -> b 1 t")
+
+        for resnet, transformer_blocks, upsample in self.up_blocks:
+            mask_up = masks.pop()
+            x = resnet(pack([x, hiddens.pop()], "b * t")[0], mask_up, t)
+            x = rearrange(x, "b c t -> b t c")
+            mask_up = rearrange(mask_up, "b 1 t -> b t")
+            for transformer_block in transformer_blocks:
+                x = transformer_block(
+                    hidden_states=x,
+                    attention_mask=mask_up,
+                    timestep=t,
+                )
+            x = rearrange(x, "b t c -> b c t")
+            mask_up = rearrange(mask_up, "b t -> b 1 t")
+            x = upsample(x * mask_up)
+
+        x = self.final_block(x, mask_up)
+        output = self.final_proj(x * mask_up)
+
+        return output * mask

engines/chatterbox_official_23lang/models/s3gen/matcha/flow_matching.py

@@ -0,0 +1,129 @@
+from abc import ABC
+
+import torch
+import torch.nn.functional as F
+
+from .decoder import Decoder
+
+
+class BASECFM(torch.nn.Module, ABC):
+    def __init__(
+        self,
+        n_feats,
+        cfm_params,
+        n_spks=1,
+        spk_emb_dim=128,
+    ):
+        super().__init__()
+        self.n_feats = n_feats
+        self.n_spks = n_spks
+        self.spk_emb_dim = spk_emb_dim
+        self.solver = cfm_params.solver
+        if hasattr(cfm_params, "sigma_min"):
+            self.sigma_min = cfm_params.sigma_min
+        else:
+            self.sigma_min = 1e-4
+
+        self.estimator = None
+
+    @torch.inference_mode()
+    def forward(self, mu, mask, n_timesteps, temperature=1.0, spks=None, cond=None):
+        """Forward diffusion
+
+        Args:
+            mu (torch.Tensor): output of encoder
+                shape: (batch_size, n_feats, mel_timesteps)
+            mask (torch.Tensor): output_mask
+                shape: (batch_size, 1, mel_timesteps)
+            n_timesteps (int): number of diffusion steps
+            temperature (float, optional): temperature for scaling noise. Defaults to 1.0.
+            spks (torch.Tensor, optional): speaker ids. Defaults to None.
+                shape: (batch_size, spk_emb_dim)
+            cond: Not used but kept for future purposes
+
+        Returns:
+            sample: generated mel-spectrogram
+                shape: (batch_size, n_feats, mel_timesteps)
+        """
+        z = torch.randn_like(mu) * temperature
+        t_span = torch.linspace(0, 1, n_timesteps + 1, device=mu.device)
+        return self.solve_euler(z, t_span=t_span, mu=mu, mask=mask, spks=spks, cond=cond)
+
+    def solve_euler(self, x, t_span, mu, mask, spks, cond):
+        """
+        Fixed euler solver for ODEs.
+        Args:
+            x (torch.Tensor): random noise
+            t_span (torch.Tensor): n_timesteps interpolated
+                shape: (n_timesteps + 1,)
+            mu (torch.Tensor): output of encoder
+                shape: (batch_size, n_feats, mel_timesteps)
+            mask (torch.Tensor): output_mask
+                shape: (batch_size, 1, mel_timesteps)
+            spks (torch.Tensor, optional): speaker ids. Defaults to None.
+                shape: (batch_size, spk_emb_dim)
+            cond: Not used but kept for future purposes
+        """
+        t, _, dt = t_span[0], t_span[-1], t_span[1] - t_span[0]
+
+        # I am storing this because I can later plot it by putting a debugger here and saving it to a file
+        # Or in future might add like a return_all_steps flag
+        sol = []
+
+        for step in range(1, len(t_span)):
+            dphi_dt = self.estimator(x, mask, mu, t, spks, cond)
+
+            x = x + dt * dphi_dt
+            t = t + dt
+            sol.append(x)
+            if step < len(t_span) - 1:
+                dt = t_span[step + 1] - t
+
+        return sol[-1]
+
+    def compute_loss(self, x1, mask, mu, spks=None, cond=None):
+        """Computes diffusion loss
+
+        Args:
+            x1 (torch.Tensor): Target
+                shape: (batch_size, n_feats, mel_timesteps)
+            mask (torch.Tensor): target mask
+                shape: (batch_size, 1, mel_timesteps)
+            mu (torch.Tensor): output of encoder
+                shape: (batch_size, n_feats, mel_timesteps)
+            spks (torch.Tensor, optional): speaker embedding. Defaults to None.
+                shape: (batch_size, spk_emb_dim)
+
+        Returns:
+            loss: conditional flow matching loss
+            y: conditional flow
+                shape: (batch_size, n_feats, mel_timesteps)
+        """
+        b, _, t = mu.shape
+
+        # random timestep
+        t = torch.rand([b, 1, 1], device=mu.device, dtype=mu.dtype)
+        # sample noise p(x_0)
+        z = torch.randn_like(x1)
+
+        y = (1 - (1 - self.sigma_min) * t) * z + t * x1
+        u = x1 - (1 - self.sigma_min) * z
+
+        loss = F.mse_loss(self.estimator(y, mask, mu, t.squeeze(), spks), u, reduction="sum") / (
+            torch.sum(mask) * u.shape[1]
+        )
+        return loss, y
+
+
+class CFM(BASECFM):
+    def __init__(self, in_channels, out_channel, cfm_params, decoder_params, n_spks=1, spk_emb_dim=64):
+        super().__init__(
+            n_feats=in_channels,
+            cfm_params=cfm_params,
+            n_spks=n_spks,
+            spk_emb_dim=spk_emb_dim,
+        )
+
+        in_channels = in_channels + (spk_emb_dim if n_spks > 1 else 0)
+        # Just change the architecture of the estimator here
+        self.estimator = Decoder(in_channels=in_channels, out_channels=out_channel, **decoder_params)

engines/chatterbox_official_23lang/models/s3gen/matcha/text_encoder.py

@@ -0,0 +1,413 @@
+""" from https://github.com/jaywalnut310/glow-tts """
+
+import math
+
+import torch
+import torch.nn as nn
+from einops import rearrange
+
+
+def sequence_mask(length, max_length=None):
+    if max_length is None:
+        max_length = length.max()
+    x = torch.arange(max_length, dtype=length.dtype, device=length.device)
+    return x.unsqueeze(0) < length.unsqueeze(1)
+
+
+
+class LayerNorm(nn.Module):
+    def __init__(self, channels, eps=1e-4):
+        super().__init__()
+        self.channels = channels
+        self.eps = eps
+
+        self.gamma = torch.nn.Parameter(torch.ones(channels))
+        self.beta = torch.nn.Parameter(torch.zeros(channels))
+
+    def forward(self, x):
+        n_dims = len(x.shape)
+        mean = torch.mean(x, 1, keepdim=True)
+        variance = torch.mean((x - mean) ** 2, 1, keepdim=True)
+
+        x = (x - mean) * torch.rsqrt(variance + self.eps)
+
+        shape = [1, -1] + [1] * (n_dims - 2)
+        x = x * self.gamma.view(*shape) + self.beta.view(*shape)
+        return x
+
+
+class ConvReluNorm(nn.Module):
+    def __init__(self, in_channels, hidden_channels, out_channels, kernel_size, n_layers, p_dropout):
+        super().__init__()
+        self.in_channels = in_channels
+        self.hidden_channels = hidden_channels
+        self.out_channels = out_channels
+        self.kernel_size = kernel_size
+        self.n_layers = n_layers
+        self.p_dropout = p_dropout
+
+        self.conv_layers = torch.nn.ModuleList()
+        self.norm_layers = torch.nn.ModuleList()
+        self.conv_layers.append(torch.nn.Conv1d(in_channels, hidden_channels, kernel_size, padding=kernel_size // 2))
+        self.norm_layers.append(LayerNorm(hidden_channels))
+        self.relu_drop = torch.nn.Sequential(torch.nn.ReLU(), torch.nn.Dropout(p_dropout))
+        for _ in range(n_layers - 1):
+            self.conv_layers.append(
+                torch.nn.Conv1d(hidden_channels, hidden_channels, kernel_size, padding=kernel_size // 2)
+            )
+            self.norm_layers.append(LayerNorm(hidden_channels))
+        self.proj = torch.nn.Conv1d(hidden_channels, out_channels, 1)
+        self.proj.weight.data.zero_()
+        self.proj.bias.data.zero_()
+
+    def forward(self, x, x_mask):
+        x_org = x
+        for i in range(self.n_layers):
+            x = self.conv_layers[i](x * x_mask)
+            x = self.norm_layers[i](x)
+            x = self.relu_drop(x)
+        x = x_org + self.proj(x)
+        return x * x_mask
+
+
+class DurationPredictor(nn.Module):
+    def __init__(self, in_channels, filter_channels, kernel_size, p_dropout):
+        super().__init__()
+        self.in_channels = in_channels
+        self.filter_channels = filter_channels
+        self.p_dropout = p_dropout
+
+        self.drop = torch.nn.Dropout(p_dropout)
+        self.conv_1 = torch.nn.Conv1d(in_channels, filter_channels, kernel_size, padding=kernel_size // 2)
+        self.norm_1 = LayerNorm(filter_channels)
+        self.conv_2 = torch.nn.Conv1d(filter_channels, filter_channels, kernel_size, padding=kernel_size // 2)
+        self.norm_2 = LayerNorm(filter_channels)
+        self.proj = torch.nn.Conv1d(filter_channels, 1, 1)
+
+    def forward(self, x, x_mask):
+        x = self.conv_1(x * x_mask)
+        x = torch.relu(x)
+        x = self.norm_1(x)
+        x = self.drop(x)
+        x = self.conv_2(x * x_mask)
+        x = torch.relu(x)
+        x = self.norm_2(x)
+        x = self.drop(x)
+        x = self.proj(x * x_mask)
+        return x * x_mask
+
+
+class RotaryPositionalEmbeddings(nn.Module):
+    """
+    ## RoPE module
+
+    Rotary encoding transforms pairs of features by rotating in the 2D plane.
+    That is, it organizes the $d$ features as $\frac{d}{2}$ pairs.
+    Each pair can be considered a coordinate in a 2D plane, and the encoding will rotate it
+    by an angle depending on the position of the token.
+    """
+
+    def __init__(self, d: int, base: int = 10_000):
+        r"""
+        * `d` is the number of features $d$
+        * `base` is the constant used for calculating $\Theta$
+        """
+        super().__init__()
+
+        self.base = base
+        self.d = int(d)
+        self.cos_cached = None
+        self.sin_cached = None
+
+    def _build_cache(self, x: torch.Tensor):
+        r"""
+        Cache $\cos$ and $\sin$ values
+        """
+        # Return if cache is already built
+        if self.cos_cached is not None and x.shape[0] <= self.cos_cached.shape[0]:
+            return
+
+        # Get sequence length
+        seq_len = x.shape[0]
+
+        # $\Theta = {\theta_i = 10000^{-\frac{2(i-1)}{d}}, i \in [1, 2, ..., \frac{d}{2}]}$
+        theta = 1.0 / (self.base ** (torch.arange(0, self.d, 2).float() / self.d)).to(x.device)
+
+        # Create position indexes `[0, 1, ..., seq_len - 1]`
+        seq_idx = torch.arange(seq_len, device=x.device).float().to(x.device)
+
+        # Calculate the product of position index and $\theta_i$
+        idx_theta = torch.einsum("n,d->nd", seq_idx, theta)
+
+        # Concatenate so that for row $m$ we have
+        # $[m \theta_0, m \theta_1, ..., m \theta_{\frac{d}{2}}, m \theta_0, m \theta_1, ..., m \theta_{\frac{d}{2}}]$
+        idx_theta2 = torch.cat([idx_theta, idx_theta], dim=1)
+
+        # Cache them
+        self.cos_cached = idx_theta2.cos()[:, None, None, :]
+        self.sin_cached = idx_theta2.sin()[:, None, None, :]
+
+    def _neg_half(self, x: torch.Tensor):
+        # $\frac{d}{2}$
+        d_2 = self.d // 2
+
+        # Calculate $[-x^{(\frac{d}{2} + 1)}, -x^{(\frac{d}{2} + 2)}, ..., -x^{(d)}, x^{(1)}, x^{(2)}, ..., x^{(\frac{d}{2})}]$
+        return torch.cat([-x[:, :, :, d_2:], x[:, :, :, :d_2]], dim=-1)
+
+    def forward(self, x: torch.Tensor):
+        """
+        * `x` is the Tensor at the head of a key or a query with shape `[seq_len, batch_size, n_heads, d]`
+        """
+        # Cache $\cos$ and $\sin$ values
+        x = rearrange(x, "b h t d -> t b h d")
+
+        self._build_cache(x)
+
+        # Split the features, we can choose to apply rotary embeddings only to a partial set of features.
+        x_rope, x_pass = x[..., : self.d], x[..., self.d :]
+
+        # Calculate
+        # $[-x^{(\frac{d}{2} + 1)}, -x^{(\frac{d}{2} + 2)}, ..., -x^{(d)}, x^{(1)}, x^{(2)}, ..., x^{(\frac{d}{2})}]$
+        neg_half_x = self._neg_half(x_rope)
+
+        x_rope = (x_rope * self.cos_cached[: x.shape[0]]) + (neg_half_x * self.sin_cached[: x.shape[0]])
+
+        return rearrange(torch.cat((x_rope, x_pass), dim=-1), "t b h d -> b h t d")
+
+
+class MultiHeadAttention(nn.Module):
+    def __init__(
+        self,
+        channels,
+        out_channels,
+        n_heads,
+        heads_share=True,
+        p_dropout=0.0,
+        proximal_bias=False,
+        proximal_init=False,
+    ):
+        super().__init__()
+        assert channels % n_heads == 0
+
+        self.channels = channels
+        self.out_channels = out_channels
+        self.n_heads = n_heads
+        self.heads_share = heads_share
+        self.proximal_bias = proximal_bias
+        self.p_dropout = p_dropout
+        self.attn = None
+
+        self.k_channels = channels // n_heads
+        self.conv_q = torch.nn.Conv1d(channels, channels, 1)
+        self.conv_k = torch.nn.Conv1d(channels, channels, 1)
+        self.conv_v = torch.nn.Conv1d(channels, channels, 1)
+
+        # from https://nn.labml.ai/transformers/rope/index.html
+        self.query_rotary_pe = RotaryPositionalEmbeddings(self.k_channels * 0.5)
+        self.key_rotary_pe = RotaryPositionalEmbeddings(self.k_channels * 0.5)
+
+        self.conv_o = torch.nn.Conv1d(channels, out_channels, 1)
+        self.drop = torch.nn.Dropout(p_dropout)
+
+        torch.nn.init.xavier_uniform_(self.conv_q.weight)
+        torch.nn.init.xavier_uniform_(self.conv_k.weight)
+        if proximal_init:
+            self.conv_k.weight.data.copy_(self.conv_q.weight.data)
+            self.conv_k.bias.data.copy_(self.conv_q.bias.data)
+        torch.nn.init.xavier_uniform_(self.conv_v.weight)
+
+    def forward(self, x, c, attn_mask=None):
+        q = self.conv_q(x)
+        k = self.conv_k(c)
+        v = self.conv_v(c)
+
+        x, self.attn = self.attention(q, k, v, mask=attn_mask)
+
+        x = self.conv_o(x)
+        return x
+
+    def attention(self, query, key, value, mask=None):
+        b, d, t_s, t_t = (*key.size(), query.size(2))
+        query = rearrange(query, "b (h c) t-> b h t c", h=self.n_heads)
+        key = rearrange(key, "b (h c) t-> b h t c", h=self.n_heads)
+        value = rearrange(value, "b (h c) t-> b h t c", h=self.n_heads)
+
+        query = self.query_rotary_pe(query)
+        key = self.key_rotary_pe(key)
+
+        scores = torch.matmul(query, key.transpose(-2, -1)) / math.sqrt(self.k_channels)
+
+        if self.proximal_bias:
+            assert t_s == t_t, "Proximal bias is only available for self-attention."
+            scores = scores + self._attention_bias_proximal(t_s).to(device=scores.device, dtype=scores.dtype)
+        if mask is not None:
+            scores = scores.masked_fill(mask == 0, -1e4)
+        p_attn = torch.nn.functional.softmax(scores, dim=-1)
+        p_attn = self.drop(p_attn)
+        output = torch.matmul(p_attn, value)
+        output = output.transpose(2, 3).contiguous().view(b, d, t_t)
+        return output, p_attn
+
+    @staticmethod
+    def _attention_bias_proximal(length):
+        r = torch.arange(length, dtype=torch.float32)
+        diff = torch.unsqueeze(r, 0) - torch.unsqueeze(r, 1)
+        return torch.unsqueeze(torch.unsqueeze(-torch.log1p(torch.abs(diff)), 0), 0)
+
+
+class FFN(nn.Module):
+    def __init__(self, in_channels, out_channels, filter_channels, kernel_size, p_dropout=0.0):
+        super().__init__()
+        self.in_channels = in_channels
+        self.out_channels = out_channels
+        self.filter_channels = filter_channels
+        self.kernel_size = kernel_size
+        self.p_dropout = p_dropout
+
+        self.conv_1 = torch.nn.Conv1d(in_channels, filter_channels, kernel_size, padding=kernel_size // 2)
+        self.conv_2 = torch.nn.Conv1d(filter_channels, out_channels, kernel_size, padding=kernel_size // 2)
+        self.drop = torch.nn.Dropout(p_dropout)
+
+    def forward(self, x, x_mask):
+        x = self.conv_1(x * x_mask)
+        x = torch.relu(x)
+        x = self.drop(x)
+        x = self.conv_2(x * x_mask)
+        return x * x_mask
+
+
+class Encoder(nn.Module):
+    def __init__(
+        self,
+        hidden_channels,
+        filter_channels,
+        n_heads,
+        n_layers,
+        kernel_size=1,
+        p_dropout=0.0,
+        **kwargs,
+    ):
+        super().__init__()
+        self.hidden_channels = hidden_channels
+        self.filter_channels = filter_channels
+        self.n_heads = n_heads
+        self.n_layers = n_layers
+        self.kernel_size = kernel_size
+        self.p_dropout = p_dropout
+
+        self.drop = torch.nn.Dropout(p_dropout)
+        self.attn_layers = torch.nn.ModuleList()
+        self.norm_layers_1 = torch.nn.ModuleList()
+        self.ffn_layers = torch.nn.ModuleList()
+        self.norm_layers_2 = torch.nn.ModuleList()
+        for _ in range(self.n_layers):
+            self.attn_layers.append(MultiHeadAttention(hidden_channels, hidden_channels, n_heads, p_dropout=p_dropout))
+            self.norm_layers_1.append(LayerNorm(hidden_channels))
+            self.ffn_layers.append(
+                FFN(
+                    hidden_channels,
+                    hidden_channels,
+                    filter_channels,
+                    kernel_size,
+                    p_dropout=p_dropout,
+                )
+            )
+            self.norm_layers_2.append(LayerNorm(hidden_channels))
+
+    def forward(self, x, x_mask):
+        attn_mask = x_mask.unsqueeze(2) * x_mask.unsqueeze(-1)
+        for i in range(self.n_layers):
+            x = x * x_mask
+            y = self.attn_layers[i](x, x, attn_mask)
+            y = self.drop(y)
+            x = self.norm_layers_1[i](x + y)
+            y = self.ffn_layers[i](x, x_mask)
+            y = self.drop(y)
+            x = self.norm_layers_2[i](x + y)
+        x = x * x_mask
+        return x
+
+
+class TextEncoder(nn.Module):
+    def __init__(
+        self,
+        encoder_type,
+        encoder_params,
+        duration_predictor_params,
+        n_vocab,
+        n_spks=1,
+        spk_emb_dim=128,
+    ):
+        super().__init__()
+        self.encoder_type = encoder_type
+        self.n_vocab = n_vocab
+        self.n_feats = encoder_params.n_feats
+        self.n_channels = encoder_params.n_channels
+        self.spk_emb_dim = spk_emb_dim
+        self.n_spks = n_spks
+
+        self.emb = torch.nn.Embedding(n_vocab, self.n_channels)
+        torch.nn.init.normal_(self.emb.weight, 0.0, self.n_channels**-0.5)
+
+        if encoder_params.prenet:
+            self.prenet = ConvReluNorm(
+                self.n_channels,
+                self.n_channels,
+                self.n_channels,
+                kernel_size=5,
+                n_layers=3,
+                p_dropout=0.5,
+            )
+        else:
+            self.prenet = lambda x, x_mask: x
+
+        self.encoder = Encoder(
+            encoder_params.n_channels + (spk_emb_dim if n_spks > 1 else 0),
+            encoder_params.filter_channels,
+            encoder_params.n_heads,
+            encoder_params.n_layers,
+            encoder_params.kernel_size,
+            encoder_params.p_dropout,
+        )
+
+        self.proj_m = torch.nn.Conv1d(self.n_channels + (spk_emb_dim if n_spks > 1 else 0), self.n_feats, 1)
+        self.proj_w = DurationPredictor(
+            self.n_channels + (spk_emb_dim if n_spks > 1 else 0),
+            duration_predictor_params.filter_channels_dp,
+            duration_predictor_params.kernel_size,
+            duration_predictor_params.p_dropout,
+        )
+
+    def forward(self, x, x_lengths, spks=None):
+        """Run forward pass to the transformer based encoder and duration predictor
+
+        Args:
+            x (torch.Tensor): text input
+                shape: (batch_size, max_text_length)
+            x_lengths (torch.Tensor): text input lengths
+                shape: (batch_size,)
+            spks (torch.Tensor, optional): speaker ids. Defaults to None.
+                shape: (batch_size,)
+
+        Returns:
+            mu (torch.Tensor): average output of the encoder
+                shape: (batch_size, n_feats, max_text_length)
+            logw (torch.Tensor): log duration predicted by the duration predictor
+                shape: (batch_size, 1, max_text_length)
+            x_mask (torch.Tensor): mask for the text input
+                shape: (batch_size, 1, max_text_length)
+        """
+        x = self.emb(x) * math.sqrt(self.n_channels)
+        x = torch.transpose(x, 1, -1)
+        x_mask = torch.unsqueeze(sequence_mask(x_lengths, x.size(2)), 1).to(x.dtype)
+
+        x = self.prenet(x, x_mask)
+        if self.n_spks > 1:
+            x = torch.cat([x, spks.unsqueeze(-1).repeat(1, 1, x.shape[-1])], dim=1)
+        x = self.encoder(x, x_mask)
+        mu = self.proj_m(x) * x_mask
+
+        x_dp = torch.detach(x)
+        logw = self.proj_w(x_dp, x_mask)
+
+        return mu, logw, x_mask

engines/chatterbox_official_23lang/models/s3gen/matcha/transformer.py

@@ -0,0 +1,316 @@
+from typing import Any, Dict, Optional
+
+import torch
+import torch.nn as nn
+from diffusers.models.attention import (
+    GEGLU,
+    GELU,
+    AdaLayerNorm,
+    AdaLayerNormZero,
+    ApproximateGELU,
+)
+from diffusers.models.attention_processor import Attention
+from diffusers.models.lora import LoRACompatibleLinear
+from diffusers.utils.torch_utils import maybe_allow_in_graph
+
+
+class SnakeBeta(nn.Module):
+    """
+    A modified Snake function which uses separate parameters for the magnitude of the periodic components
+    Shape:
+        - Input: (B, C, T)
+        - Output: (B, C, T), same shape as the input
+    Parameters:
+        - alpha - trainable parameter that controls frequency
+        - beta - trainable parameter that controls magnitude
+    References:
+        - This activation function is a modified version based on this paper by Liu Ziyin, Tilman Hartwig, Masahito Ueda:
+        https://arxiv.org/abs/2006.08195
+    Examples:
+        >>> a1 = snakebeta(256)
+        >>> x = torch.randn(256)
+        >>> x = a1(x)
+    """
+
+    def __init__(self, in_features, out_features, alpha=1.0, alpha_trainable=True, alpha_logscale=True):
+        """
+        Initialization.
+        INPUT:
+            - in_features: shape of the input
+            - alpha - trainable parameter that controls frequency
+            - beta - trainable parameter that controls magnitude
+            alpha is initialized to 1 by default, higher values = higher-frequency.
+            beta is initialized to 1 by default, higher values = higher-magnitude.
+            alpha will be trained along with the rest of your model.
+        """
+        super().__init__()
+        self.in_features = out_features if isinstance(out_features, list) else [out_features]
+        self.proj = LoRACompatibleLinear(in_features, out_features)
+
+        # initialize alpha
+        self.alpha_logscale = alpha_logscale
+        if self.alpha_logscale:  # log scale alphas initialized to zeros
+            self.alpha = nn.Parameter(torch.zeros(self.in_features) * alpha)
+            self.beta = nn.Parameter(torch.zeros(self.in_features) * alpha)
+        else:  # linear scale alphas initialized to ones
+            self.alpha = nn.Parameter(torch.ones(self.in_features) * alpha)
+            self.beta = nn.Parameter(torch.ones(self.in_features) * alpha)
+
+        self.alpha.requires_grad = alpha_trainable
+        self.beta.requires_grad = alpha_trainable
+
+        self.no_div_by_zero = 0.000000001
+
+    def forward(self, x):
+        """
+        Forward pass of the function.
+        Applies the function to the input elementwise.
+        SnakeBeta ∶= x + 1/b * sin^2 (xa)
+        """
+        x = self.proj(x)
+        if self.alpha_logscale:
+            alpha = torch.exp(self.alpha)
+            beta = torch.exp(self.beta)
+        else:
+            alpha = self.alpha
+            beta = self.beta
+
+        x = x + (1.0 / (beta + self.no_div_by_zero)) * torch.pow(torch.sin(x * alpha), 2)
+
+        return x
+
+
+class FeedForward(nn.Module):
+    r"""
+    A feed-forward layer.
+
+    Parameters:
+        dim (`int`): The number of channels in the input.
+        dim_out (`int`, *optional*): The number of channels in the output. If not given, defaults to `dim`.
+        mult (`int`, *optional*, defaults to 4): The multiplier to use for the hidden dimension.
+        dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.
+        activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward.
+        final_dropout (`bool` *optional*, defaults to False): Apply a final dropout.
+    """
+
+    def __init__(
+        self,
+        dim: int,
+        dim_out: Optional[int] = None,
+        mult: int = 4,
+        dropout: float = 0.0,
+        activation_fn: str = "geglu",
+        final_dropout: bool = False,
+    ):
+        super().__init__()
+        inner_dim = int(dim * mult)
+        dim_out = dim_out if dim_out is not None else dim
+
+        if activation_fn == "gelu":
+            act_fn = GELU(dim, inner_dim)
+        if activation_fn == "gelu-approximate":
+            act_fn = GELU(dim, inner_dim, approximate="tanh")
+        elif activation_fn == "geglu":
+            act_fn = GEGLU(dim, inner_dim)
+        elif activation_fn == "geglu-approximate":
+            act_fn = ApproximateGELU(dim, inner_dim)
+        elif activation_fn == "snakebeta":
+            act_fn = SnakeBeta(dim, inner_dim)
+
+        self.net = nn.ModuleList([])
+        # project in
+        self.net.append(act_fn)
+        # project dropout
+        self.net.append(nn.Dropout(dropout))
+        # project out
+        self.net.append(LoRACompatibleLinear(inner_dim, dim_out))
+        # FF as used in Vision Transformer, MLP-Mixer, etc. have a final dropout
+        if final_dropout:
+            self.net.append(nn.Dropout(dropout))
+
+    def forward(self, hidden_states):
+        for module in self.net:
+            hidden_states = module(hidden_states)
+        return hidden_states
+
+
+@maybe_allow_in_graph
+class BasicTransformerBlock(nn.Module):
+    r"""
+    A basic Transformer block.
+
+    Parameters:
+        dim (`int`): The number of channels in the input and output.
+        num_attention_heads (`int`): The number of heads to use for multi-head attention.
+        attention_head_dim (`int`): The number of channels in each head.
+        dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.
+        cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention.
+        only_cross_attention (`bool`, *optional*):
+            Whether to use only cross-attention layers. In this case two cross attention layers are used.
+        double_self_attention (`bool`, *optional*):
+            Whether to use two self-attention layers. In this case no cross attention layers are used.
+        activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward.
+        num_embeds_ada_norm (:
+            obj: `int`, *optional*): The number of diffusion steps used during training. See `Transformer2DModel`.
+        attention_bias (:
+            obj: `bool`, *optional*, defaults to `False`): Configure if the attentions should contain a bias parameter.
+    """
+
+    def __init__(
+        self,
+        dim: int,
+        num_attention_heads: int,
+        attention_head_dim: int,
+        dropout=0.0,
+        cross_attention_dim: Optional[int] = None,
+        activation_fn: str = "geglu",
+        num_embeds_ada_norm: Optional[int] = None,
+        attention_bias: bool = False,
+        only_cross_attention: bool = False,
+        double_self_attention: bool = False,
+        upcast_attention: bool = False,
+        norm_elementwise_affine: bool = True,
+        norm_type: str = "layer_norm",
+        final_dropout: bool = False,
+    ):
+        super().__init__()
+        self.only_cross_attention = only_cross_attention
+
+        self.use_ada_layer_norm_zero = (num_embeds_ada_norm is not None) and norm_type == "ada_norm_zero"
+        self.use_ada_layer_norm = (num_embeds_ada_norm is not None) and norm_type == "ada_norm"
+
+        if norm_type in ("ada_norm", "ada_norm_zero") and num_embeds_ada_norm is None:
+            raise ValueError(
+                f"`norm_type` is set to {norm_type}, but `num_embeds_ada_norm` is not defined. Please make sure to"
+                f" define `num_embeds_ada_norm` if setting `norm_type` to {norm_type}."
+            )
+
+        # Define 3 blocks. Each block has its own normalization layer.
+        # 1. Self-Attn
+        if self.use_ada_layer_norm:
+            self.norm1 = AdaLayerNorm(dim, num_embeds_ada_norm)
+        elif self.use_ada_layer_norm_zero:
+            self.norm1 = AdaLayerNormZero(dim, num_embeds_ada_norm)
+        else:
+            self.norm1 = nn.LayerNorm(dim, elementwise_affine=norm_elementwise_affine)
+        self.attn1 = Attention(
+            query_dim=dim,
+            heads=num_attention_heads,
+            dim_head=attention_head_dim,
+            dropout=dropout,
+            bias=attention_bias,
+            cross_attention_dim=cross_attention_dim if only_cross_attention else None,
+            upcast_attention=upcast_attention,
+        )
+
+        # 2. Cross-Attn
+        if cross_attention_dim is not None or double_self_attention:
+            # We currently only use AdaLayerNormZero for self attention where there will only be one attention block.
+            # I.e. the number of returned modulation chunks from AdaLayerZero would not make sense if returned during
+            # the second cross attention block.
+            self.norm2 = (
+                AdaLayerNorm(dim, num_embeds_ada_norm)
+                if self.use_ada_layer_norm
+                else nn.LayerNorm(dim, elementwise_affine=norm_elementwise_affine)
+            )
+            self.attn2 = Attention(
+                query_dim=dim,
+                cross_attention_dim=cross_attention_dim if not double_self_attention else None,
+                heads=num_attention_heads,
+                dim_head=attention_head_dim,
+                dropout=dropout,
+                bias=attention_bias,
+                upcast_attention=upcast_attention,
+                # scale_qk=False, # uncomment this to not to use flash attention
+            )  # is self-attn if encoder_hidden_states is none
+        else:
+            self.norm2 = None
+            self.attn2 = None
+
+        # 3. Feed-forward
+        self.norm3 = nn.LayerNorm(dim, elementwise_affine=norm_elementwise_affine)
+        self.ff = FeedForward(dim, dropout=dropout, activation_fn=activation_fn, final_dropout=final_dropout)
+
+        # let chunk size default to None
+        self._chunk_size = None
+        self._chunk_dim = 0
+
+    def set_chunk_feed_forward(self, chunk_size: Optional[int], dim: int):
+        # Sets chunk feed-forward
+        self._chunk_size = chunk_size
+        self._chunk_dim = dim
+
+    def forward(
+        self,
+        hidden_states: torch.FloatTensor,
+        attention_mask: Optional[torch.FloatTensor] = None,
+        encoder_hidden_states: Optional[torch.FloatTensor] = None,
+        encoder_attention_mask: Optional[torch.FloatTensor] = None,
+        timestep: Optional[torch.LongTensor] = None,
+        cross_attention_kwargs: Dict[str, Any] = None,
+        class_labels: Optional[torch.LongTensor] = None,
+    ):
+        # Notice that normalization is always applied before the real computation in the following blocks.
+        # 1. Self-Attention
+        if self.use_ada_layer_norm:
+            norm_hidden_states = self.norm1(hidden_states, timestep)
+        elif self.use_ada_layer_norm_zero:
+            norm_hidden_states, gate_msa, shift_mlp, scale_mlp, gate_mlp = self.norm1(
+                hidden_states, timestep, class_labels, hidden_dtype=hidden_states.dtype
+            )
+        else:
+            norm_hidden_states = self.norm1(hidden_states)
+
+        cross_attention_kwargs = cross_attention_kwargs if cross_attention_kwargs is not None else {}
+
+        attn_output = self.attn1(
+            norm_hidden_states,
+            encoder_hidden_states=encoder_hidden_states if self.only_cross_attention else None,
+            attention_mask=encoder_attention_mask if self.only_cross_attention else attention_mask,
+            **cross_attention_kwargs,
+        )
+        if self.use_ada_layer_norm_zero:
+            attn_output = gate_msa.unsqueeze(1) * attn_output
+        hidden_states = attn_output + hidden_states
+
+        # 2. Cross-Attention
+        if self.attn2 is not None:
+            norm_hidden_states = (
+                self.norm2(hidden_states, timestep) if self.use_ada_layer_norm else self.norm2(hidden_states)
+            )
+
+            attn_output = self.attn2(
+                norm_hidden_states,
+                encoder_hidden_states=encoder_hidden_states,
+                attention_mask=encoder_attention_mask,
+                **cross_attention_kwargs,
+            )
+            hidden_states = attn_output + hidden_states
+
+        # 3. Feed-forward
+        norm_hidden_states = self.norm3(hidden_states)
+
+        if self.use_ada_layer_norm_zero:
+            norm_hidden_states = norm_hidden_states * (1 + scale_mlp[:, None]) + shift_mlp[:, None]
+
+        if self._chunk_size is not None:
+            # "feed_forward_chunk_size" can be used to save memory
+            if norm_hidden_states.shape[self._chunk_dim] % self._chunk_size != 0:
+                raise ValueError(
+                    f"`hidden_states` dimension to be chunked: {norm_hidden_states.shape[self._chunk_dim]} has to be divisible by chunk size: {self._chunk_size}. Make sure to set an appropriate `chunk_size` when calling `unet.enable_forward_chunking`."
+                )
+
+            num_chunks = norm_hidden_states.shape[self._chunk_dim] // self._chunk_size
+            ff_output = torch.cat(
+                [self.ff(hid_slice) for hid_slice in norm_hidden_states.chunk(num_chunks, dim=self._chunk_dim)],
+                dim=self._chunk_dim,
+            )
+        else:
+            ff_output = self.ff(norm_hidden_states)
+
+        if self.use_ada_layer_norm_zero:
+            ff_output = gate_mlp.unsqueeze(1) * ff_output
+
+        hidden_states = ff_output + hidden_states
+
+        return hidden_states

engines/chatterbox_official_23lang/models/s3gen/s3gen.py

@@ -0,0 +1,301 @@
+# Modified from CosyVoice https://github.com/FunAudioLLM/CosyVoice
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+import numpy as np
+import torch
+import torchaudio as ta
+from functools import lru_cache
+from typing import Optional
+
+from ..s3tokenizer import S3_SR, SPEECH_VOCAB_SIZE, S3Tokenizer
+from .const import S3GEN_SR
+from .flow import CausalMaskedDiffWithXvec
+from .xvector import CAMPPlus
+from .utils.mel import mel_spectrogram
+from .f0_predictor import ConvRNNF0Predictor
+from .hifigan import HiFTGenerator
+from .transformer.upsample_encoder import UpsampleConformerEncoder
+from .flow_matching import CausalConditionalCFM
+from .decoder import ConditionalDecoder
+from .configs import CFM_PARAMS
+
+
+def drop_invalid_tokens(x):
+    assert len(x.shape) <= 2 and x.shape[0] == 1, "only batch size of one allowed for now"
+    return x[x < SPEECH_VOCAB_SIZE]
+
+
+# TODO: global resampler cache
+@lru_cache(100)
+def get_resampler(src_sr, dst_sr, device):
+    return ta.transforms.Resample(src_sr, dst_sr).to(device)
+
+
+class S3Token2Mel(torch.nn.Module):
+    """
+    CosyVoice2's CFM decoder maps S3 speech tokens to mel-spectrograms.
+
+    TODO: make these modules configurable?
+    """
+    def __init__(self):
+        super().__init__()
+        self.tokenizer = S3Tokenizer("speech_tokenizer_v2_25hz")
+        self.mel_extractor = mel_spectrogram # TODO: make it a torch module?
+        self.speaker_encoder = CAMPPlus()  # use default args
+
+        encoder = UpsampleConformerEncoder(
+            output_size=512,
+            attention_heads=8,
+            linear_units=2048,
+            num_blocks=6,
+            dropout_rate=0.1,
+            positional_dropout_rate=0.1,
+            attention_dropout_rate=0.1,
+            normalize_before=True,
+            input_layer='linear',
+            pos_enc_layer_type='rel_pos_espnet',
+            selfattention_layer_type='rel_selfattn',
+            input_size=512,
+            use_cnn_module=False,
+            macaron_style=False,
+        )
+
+        estimator = ConditionalDecoder(
+            in_channels=320,
+            out_channels=80,
+            causal=True,
+            channels=[256],
+            dropout=0.0,
+            attention_head_dim=64,
+            n_blocks=4,
+            num_mid_blocks=12,
+            num_heads=8,
+            act_fn='gelu',
+        )
+        cfm_params = CFM_PARAMS
+        decoder = CausalConditionalCFM(
+            spk_emb_dim=80,
+            cfm_params=cfm_params,
+            estimator=estimator,
+        )
+
+        self.flow = CausalMaskedDiffWithXvec(
+            encoder=encoder,
+            decoder=decoder
+        )
+
+        self.resamplers = {}
+
+    @property
+    def device(self):
+        params = self.tokenizer.parameters()
+        return next(params).device
+
+    def embed_ref(
+        self,
+        ref_wav: torch.Tensor,
+        ref_sr: int,
+        device="auto",
+        ref_fade_out=True,
+    ):
+        device = self.device if device == "auto" else device
+        if isinstance(ref_wav, np.ndarray):
+            ref_wav = torch.from_numpy(ref_wav).float()
+
+        if ref_wav.device != device:
+            ref_wav = ref_wav.to(device)
+
+        if len(ref_wav.shape) == 1:
+            ref_wav = ref_wav.unsqueeze(0)  # (B, L)
+
+        if ref_wav.size(1) > 10 * ref_sr:
+            print("WARNING: cosydec received ref longer than 10s")
+
+        ref_wav_24 = ref_wav
+        if ref_sr != S3GEN_SR:
+            ref_wav_24 = get_resampler(ref_sr, S3GEN_SR, device)(ref_wav)
+
+        ref_mels_24 = self.mel_extractor(ref_wav_24).transpose(1, 2).to(device)
+        ref_mels_24_len = None
+
+        # Resample to 16kHz
+        ref_wav_16 = get_resampler(ref_sr, S3_SR, device)(ref_wav).to(device)
+
+        # Speaker embedding
+        # Ensure speaker_encoder is on the correct device
+        if next(self.speaker_encoder.parameters()).device != device:
+            self.speaker_encoder = self.speaker_encoder.to(device)
+        ref_x_vector = self.speaker_encoder.inference(ref_wav_16)
+
+        # Tokenize 16khz reference
+        ref_speech_tokens, ref_speech_token_lens = self.tokenizer(ref_wav_16)
+
+        # Make sure mel_len = 2 * stoken_len (happens when the input is not padded to multiple of 40ms)
+        if ref_mels_24.shape[1] != 2 * ref_speech_tokens.shape[1]:
+            logging.warning(
+                "Reference mel length is not equal to 2 * reference token length.\n"
+            )
+            ref_speech_tokens = ref_speech_tokens[:, :ref_mels_24.shape[1] // 2]
+            ref_speech_token_lens[0] = ref_speech_tokens.shape[1]
+
+        return dict(
+            prompt_token=ref_speech_tokens.to(device),
+            prompt_token_len=ref_speech_token_lens,
+            prompt_feat=ref_mels_24,
+            prompt_feat_len=ref_mels_24_len,
+            embedding=ref_x_vector,
+        )
+
+    def forward(
+        self,
+        speech_tokens: torch.LongTensor,
+        # locally-computed ref embedding (mutex with ref_dict)
+        ref_wav: Optional[torch.Tensor],
+        ref_sr: Optional[int],
+        # pre-computed ref embedding (prod API)
+        ref_dict: Optional[dict] = None,
+        finalize: bool = False,
+    ):
+        """
+        Generate waveforms from S3 speech tokens and a reference waveform, which the speaker timbre is inferred from.
+
+        NOTE:
+        - The speaker encoder accepts 16 kHz waveform.
+        - S3TokenizerV2 accepts 16 kHz waveform.
+        - The mel-spectrogram for the reference assumes 24 kHz input signal.
+        - This function is designed for batch_size=1 only.
+
+        Args
+        ----
+        - `speech_tokens`: S3 speech tokens [B=1, T]
+        - `ref_wav`: reference waveform (`torch.Tensor` with shape=[B=1, T])
+        - `ref_sr`: reference sample rate
+        - `finalize`: whether streaming is finished or not. Note that if False, the last 3 tokens will be ignored.
+        """
+        assert (ref_wav is None) ^ (ref_dict is None), f"Must provide exactly one of ref_wav or ref_dict (got {ref_wav} and {ref_dict})"
+
+        if ref_dict is None:
+            ref_dict = self.embed_ref(ref_wav, ref_sr)
+        else:
+            # type/device casting (all values will be numpy if it's from a prod API call)
+            for rk in list(ref_dict):
+                if isinstance(ref_dict[rk], np.ndarray):
+                    ref_dict[rk] = torch.from_numpy(ref_dict[rk])
+                if torch.is_tensor(ref_dict[rk]):
+                    ref_dict[rk] = ref_dict[rk].to(self.device)
+
+        if len(speech_tokens.shape) == 1:
+            speech_tokens = speech_tokens.unsqueeze(0)
+
+        # assert speech_tokens.shape[0] == 1, "only batch size of one allowed for now"
+        speech_token_lens = torch.LongTensor([speech_tokens.size(1)]).to(self.device)
+
+        output_mels, _ = self.flow.inference(
+            token=speech_tokens,
+            token_len=speech_token_lens,
+            finalize=finalize,
+            **ref_dict,
+        )
+        return output_mels
+
+
+class S3Token2Wav(S3Token2Mel):
+    """
+    The decoder of CosyVoice2 is a concat of token-to-mel (CFM) and a mel-to-waveform (HiFiGAN) modules.
+
+    TODO: make these modules configurable?
+    """
+
+    def __init__(self):
+        super().__init__()
+
+        f0_predictor = ConvRNNF0Predictor()
+        self.mel2wav = HiFTGenerator(
+            sampling_rate=S3GEN_SR,
+            upsample_rates=[8, 5, 3],
+            upsample_kernel_sizes=[16, 11, 7],
+            source_resblock_kernel_sizes=[7, 7, 11],
+            source_resblock_dilation_sizes=[[1, 3, 5], [1, 3, 5], [1, 3, 5]],
+            f0_predictor=f0_predictor,
+        )
+
+        # silence out a few ms and fade audio in to reduce artifacts
+        n_trim = S3GEN_SR // 50  # 20ms = half of a frame
+        trim_fade = torch.zeros(2 * n_trim)
+        trim_fade[n_trim:] = (torch.cos(torch.linspace(torch.pi, 0, n_trim)) + 1) / 2
+        self.register_buffer("trim_fade", trim_fade, persistent=False) # (buffers get automatic device casting)
+
+    def forward(
+        self,
+        speech_tokens,
+        # locally-computed ref embedding (mutex with ref_dict)
+        ref_wav: Optional[torch.Tensor],
+        ref_sr: Optional[int],
+        # pre-computed ref embedding (prod API)
+        ref_dict: Optional[dict] = None,
+        finalize: bool = False
+    ):
+        output_mels = super().forward(speech_tokens, ref_wav=ref_wav, ref_sr=ref_sr, ref_dict=ref_dict, finalize=finalize)
+
+        # TODO jrm: ignoring the speed control (mel interpolation) and the HiFTGAN caching mechanisms for now.
+        hift_cache_source = torch.zeros(1, 1, 0).to(self.device)
+
+        output_wavs, *_ = self.mel2wav.inference(speech_feat=output_mels, cache_source=hift_cache_source)
+
+        if not self.training:
+            # NOTE: ad-hoc method to reduce "spillover" from the reference clip.
+            output_wavs[:, :len(self.trim_fade)] *= self.trim_fade
+
+        return output_wavs
+
+    @torch.inference_mode()
+    def flow_inference(
+        self,
+        speech_tokens,
+        # locally-computed ref embedding (mutex with ref_dict)
+        ref_wav: Optional[torch.Tensor] = None,
+        ref_sr: Optional[int] = None,
+        # pre-computed ref embedding (prod API)
+        ref_dict: Optional[dict] = None,
+        finalize: bool = False,
+    ):
+        return super().forward(speech_tokens, ref_wav=ref_wav, ref_sr=ref_sr, ref_dict=ref_dict, finalize=finalize)
+
+    @torch.inference_mode()
+    def hift_inference(self, speech_feat, cache_source: torch.Tensor = None):
+        if cache_source is None:
+            cache_source = torch.zeros(1, 1, 0).to(self.device)
+        return self.mel2wav.inference(speech_feat=speech_feat, cache_source=cache_source)
+
+    @torch.inference_mode()
+    def inference(
+        self,
+        speech_tokens,
+        # locally-computed ref embedding (mutex with ref_dict)
+        ref_wav: Optional[torch.Tensor] = None,
+        ref_sr: Optional[int] = None,
+        # pre-computed ref embedding (prod API)
+        ref_dict: Optional[dict] = None,
+        cache_source: torch.Tensor = None, # NOTE: this arg is for streaming, it can probably be removed here
+        finalize: bool = True,
+    ):
+        output_mels = self.flow_inference(speech_tokens, ref_wav=ref_wav, ref_sr=ref_sr, ref_dict=ref_dict, finalize=finalize)
+        output_wavs, output_sources = self.hift_inference(output_mels, cache_source)
+
+        # NOTE: ad-hoc method to reduce "spillover" from the reference clip.
+        output_wavs[:, :len(self.trim_fade)] *= self.trim_fade
+
+        return output_wavs, output_sources

engines/chatterbox_official_23lang/models/s3gen/transformer/activation.py

@@ -0,0 +1,84 @@
+# Copyright (c) 2020 Johns Hopkins University (Shinji Watanabe)
+#               2020 Northwestern Polytechnical University (Pengcheng Guo)
+#               2020 Mobvoi Inc (Binbin Zhang)
+#               2024 Alibaba Inc (Xiang Lyu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Swish() activation function for Conformer."""
+
+import torch
+from torch import nn, sin, pow
+from torch.nn import Parameter
+
+
+class Swish(torch.nn.Module):
+    """Construct an Swish object."""
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        """Return Swish activation function."""
+        return x * torch.sigmoid(x)
+
+
+# Implementation adapted from https://github.com/EdwardDixon/snake under the MIT license.
+#   LICENSE is in incl_licenses directory.
+class Snake(nn.Module):
+    '''
+    Implementation of a sine-based periodic activation function
+    Shape:
+        - Input: (B, C, T)
+        - Output: (B, C, T), same shape as the input
+    Parameters:
+        - alpha - trainable parameter
+    References:
+        - This activation function is from this paper by Liu Ziyin, Tilman Hartwig, Masahito Ueda:
+        https://arxiv.org/abs/2006.08195
+    Examples:
+        >>> a1 = snake(256)
+        >>> x = torch.randn(256)
+        >>> x = a1(x)
+    '''
+    def __init__(self, in_features, alpha=1.0, alpha_trainable=True, alpha_logscale=False):
+        '''
+        Initialization.
+        INPUT:
+            - in_features: shape of the input
+            - alpha: trainable parameter
+            alpha is initialized to 1 by default, higher values = higher-frequency.
+            alpha will be trained along with the rest of your model.
+        '''
+        super(Snake, self).__init__()
+        self.in_features = in_features
+
+        # initialize alpha
+        self.alpha_logscale = alpha_logscale
+        if self.alpha_logscale:  # log scale alphas initialized to zeros
+            self.alpha = Parameter(torch.zeros(in_features) * alpha)
+        else:  # linear scale alphas initialized to ones
+            self.alpha = Parameter(torch.ones(in_features) * alpha)
+
+        self.alpha.requires_grad = alpha_trainable
+
+        self.no_div_by_zero = 0.000000001
+
+    def forward(self, x):
+        '''
+        Forward pass of the function.
+        Applies the function to the input elementwise.
+        Snake ∶= x + 1/a * sin^2 (xa)
+        '''
+        alpha = self.alpha.unsqueeze(0).unsqueeze(-1)  # line up with x to [B, C, T]
+        if self.alpha_logscale:
+            alpha = torch.exp(alpha)
+        x = x + (1.0 / (alpha + self.no_div_by_zero)) * pow(sin(x * alpha), 2)
+
+        return x

engines/chatterbox_official_23lang/models/s3gen/transformer/attention.py

@@ -0,0 +1,330 @@
+# Copyright (c) 2019 Shigeki Karita
+#               2020 Mobvoi Inc (Binbin Zhang)
+#               2022 Xingchen Song (sxc19@mails.tsinghua.edu.cn)
+#               2024 Alibaba Inc (Xiang Lyu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Multi-Head Attention layer definition."""
+
+import math
+from typing import Tuple
+
+import torch
+from torch import nn
+
+
+class MultiHeadedAttention(nn.Module):
+    """Multi-Head Attention layer.
+
+    Args:
+        n_head (int): The number of heads.
+        n_feat (int): The number of features.
+        dropout_rate (float): Dropout rate.
+
+    """
+
+    def __init__(self,
+                 n_head: int,
+                 n_feat: int,
+                 dropout_rate: float,
+                 key_bias: bool = True):
+        """Construct an MultiHeadedAttention object."""
+        super().__init__()
+        assert n_feat % n_head == 0
+        # We assume d_v always equals d_k
+        self.d_k = n_feat // n_head
+        self.h = n_head
+        self.linear_q = nn.Linear(n_feat, n_feat)
+        self.linear_k = nn.Linear(n_feat, n_feat, bias=key_bias)
+        self.linear_v = nn.Linear(n_feat, n_feat)
+        self.linear_out = nn.Linear(n_feat, n_feat)
+        self.dropout = nn.Dropout(p=dropout_rate)
+
+    def forward_qkv(
+        self, query: torch.Tensor, key: torch.Tensor, value: torch.Tensor
+    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        """Transform query, key and value.
+
+        Args:
+            query (torch.Tensor): Query tensor (#batch, time1, size).
+            key (torch.Tensor): Key tensor (#batch, time2, size).
+            value (torch.Tensor): Value tensor (#batch, time2, size).
+
+        Returns:
+            torch.Tensor: Transformed query tensor, size
+                (#batch, n_head, time1, d_k).
+            torch.Tensor: Transformed key tensor, size
+                (#batch, n_head, time2, d_k).
+            torch.Tensor: Transformed value tensor, size
+                (#batch, n_head, time2, d_k).
+
+        """
+        n_batch = query.size(0)
+        q = self.linear_q(query).view(n_batch, -1, self.h, self.d_k)
+        k = self.linear_k(key).view(n_batch, -1, self.h, self.d_k)
+        v = self.linear_v(value).view(n_batch, -1, self.h, self.d_k)
+        q = q.transpose(1, 2)  # (batch, head, time1, d_k)
+        k = k.transpose(1, 2)  # (batch, head, time2, d_k)
+        v = v.transpose(1, 2)  # (batch, head, time2, d_k)
+
+        return q, k, v
+
+    def forward_attention(
+        self,
+        value: torch.Tensor,
+        scores: torch.Tensor,
+        mask: torch.Tensor = torch.ones((0, 0, 0), dtype=torch.bool)
+    ) -> torch.Tensor:
+        """Compute attention context vector.
+
+        Args:
+            value (torch.Tensor): Transformed value, size
+                (#batch, n_head, time2, d_k).
+            scores (torch.Tensor): Attention score, size
+                (#batch, n_head, time1, time2).
+            mask (torch.Tensor): Mask, size (#batch, 1, time2) or
+                (#batch, time1, time2), (0, 0, 0) means fake mask.
+
+        Returns:
+            torch.Tensor: Transformed value (#batch, time1, d_model)
+                weighted by the attention score (#batch, time1, time2).
+
+        """
+        n_batch = value.size(0)
+        # NOTE(xcsong): When will `if mask.size(2) > 0` be True?
+        #   1. onnx(16/4) [WHY? Because we feed real cache & real mask for the
+        #           1st chunk to ease the onnx export.]
+        #   2. pytorch training
+        if mask.size(2) > 0:  # time2 > 0
+            mask = mask.unsqueeze(1).eq(0)  # (batch, 1, *, time2)
+            # For last chunk, time2 might be larger than scores.size(-1)
+            mask = mask[:, :, :, :scores.size(-1)]  # (batch, 1, *, time2)
+            scores = scores.masked_fill(mask, -float('inf'))
+            attn = torch.softmax(scores, dim=-1).masked_fill(
+                mask, 0.0)  # (batch, head, time1, time2)
+        # NOTE(xcsong): When will `if mask.size(2) > 0` be False?
+        #   1. onnx(16/-1, -1/-1, 16/0)
+        #   2. jit (16/-1, -1/-1, 16/0, 16/4)
+        else:
+            attn = torch.softmax(scores, dim=-1)  # (batch, head, time1, time2)
+
+        p_attn = self.dropout(attn)
+        x = torch.matmul(p_attn, value)  # (batch, head, time1, d_k)
+        x = (x.transpose(1, 2).contiguous().view(n_batch, -1,
+                                                 self.h * self.d_k)
+             )  # (batch, time1, d_model)
+
+        return self.linear_out(x)  # (batch, time1, d_model)
+
+    def forward(
+        self,
+        query: torch.Tensor,
+        key: torch.Tensor,
+        value: torch.Tensor,
+        mask: torch.Tensor = torch.ones((0, 0, 0), dtype=torch.bool),
+        pos_emb: torch.Tensor = torch.empty(0),
+        cache: torch.Tensor = torch.zeros((0, 0, 0, 0))
+    ) -> Tuple[torch.Tensor, torch.Tensor]:
+        """Compute scaled dot product attention.
+
+        Args:
+            query (torch.Tensor): Query tensor (#batch, time1, size).
+            key (torch.Tensor): Key tensor (#batch, time2, size).
+            value (torch.Tensor): Value tensor (#batch, time2, size).
+            mask (torch.Tensor): Mask tensor (#batch, 1, time2) or
+                (#batch, time1, time2).
+                1.When applying cross attention between decoder and encoder,
+                the batch padding mask for input is in (#batch, 1, T) shape.
+                2.When applying self attention of encoder,
+                the mask is in (#batch, T, T)  shape.
+                3.When applying self attention of decoder,
+                the mask is in (#batch, L, L)  shape.
+                4.If the different position in decoder see different block
+                of the encoder, such as Mocha, the passed in mask could be
+                in (#batch, L, T) shape. But there is no such case in current
+                CosyVoice.
+            cache (torch.Tensor): Cache tensor (1, head, cache_t, d_k * 2),
+                where `cache_t == chunk_size * num_decoding_left_chunks`
+                and `head * d_k == size`
+
+
+        Returns:
+            torch.Tensor: Output tensor (#batch, time1, d_model).
+            torch.Tensor: Cache tensor (1, head, cache_t + time1, d_k * 2)
+                where `cache_t == chunk_size * num_decoding_left_chunks`
+                and `head * d_k == size`
+
+        """
+        q, k, v = self.forward_qkv(query, key, value)
+
+        # NOTE(xcsong):
+        #   when export onnx model, for 1st chunk, we feed
+        #       cache(1, head, 0, d_k * 2) (16/-1, -1/-1, 16/0 mode)
+        #       or cache(1, head, real_cache_t, d_k * 2) (16/4 mode).
+        #       In all modes, `if cache.size(0) > 0` will alwayse be `True`
+        #       and we will always do splitting and
+        #       concatnation(this will simplify onnx export). Note that
+        #       it's OK to concat & split zero-shaped tensors(see code below).
+        #   when export jit  model, for 1st chunk, we always feed
+        #       cache(0, 0, 0, 0) since jit supports dynamic if-branch.
+        # >>> a = torch.ones((1, 2, 0, 4))
+        # >>> b = torch.ones((1, 2, 3, 4))
+        # >>> c = torch.cat((a, b), dim=2)
+        # >>> torch.equal(b, c)        # True
+        # >>> d = torch.split(a, 2, dim=-1)
+        # >>> torch.equal(d[0], d[1])  # True
+        if cache.size(0) > 0:
+            key_cache, value_cache = torch.split(cache,
+                                                 cache.size(-1) // 2,
+                                                 dim=-1)
+            k = torch.cat([key_cache, k], dim=2)
+            v = torch.cat([value_cache, v], dim=2)
+        # NOTE(xcsong): We do cache slicing in encoder.forward_chunk, since it's
+        #   non-trivial to calculate `next_cache_start` here.
+        new_cache = torch.cat((k, v), dim=-1)
+
+        scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.d_k)
+        return self.forward_attention(v, scores, mask), new_cache
+
+
+class RelPositionMultiHeadedAttention(MultiHeadedAttention):
+    """Multi-Head Attention layer with relative position encoding.
+    Paper: https://arxiv.org/abs/1901.02860
+    Args:
+        n_head (int): The number of heads.
+        n_feat (int): The number of features.
+        dropout_rate (float): Dropout rate.
+    """
+
+    def __init__(self,
+                 n_head: int,
+                 n_feat: int,
+                 dropout_rate: float,
+                 key_bias: bool = True):
+        """Construct an RelPositionMultiHeadedAttention object."""
+        super().__init__(n_head, n_feat, dropout_rate, key_bias)
+        # linear transformation for positional encoding
+        self.linear_pos = nn.Linear(n_feat, n_feat, bias=False)
+        # these two learnable bias are used in matrix c and matrix d
+        # as described in https://arxiv.org/abs/1901.02860 Section 3.3
+        self.pos_bias_u = nn.Parameter(torch.Tensor(self.h, self.d_k))
+        self.pos_bias_v = nn.Parameter(torch.Tensor(self.h, self.d_k))
+        torch.nn.init.xavier_uniform_(self.pos_bias_u)
+        torch.nn.init.xavier_uniform_(self.pos_bias_v)
+
+    def rel_shift(self, x: torch.Tensor) -> torch.Tensor:
+        """Compute relative positional encoding.
+
+        Args:
+            x (torch.Tensor): Input tensor (batch, head, time1, 2*time1-1).
+            time1 means the length of query vector.
+
+        Returns:
+            torch.Tensor: Output tensor.
+
+        """
+        zero_pad = torch.zeros((x.size()[0], x.size()[1], x.size()[2], 1),
+                               device=x.device,
+                               dtype=x.dtype)
+        x_padded = torch.cat([zero_pad, x], dim=-1)
+
+        x_padded = x_padded.view(x.size()[0],
+                                 x.size()[1],
+                                 x.size(3) + 1, x.size(2))
+        x = x_padded[:, :, 1:].view_as(x)[
+            :, :, :, : x.size(-1) // 2 + 1
+        ]  # only keep the positions from 0 to time2
+        return x
+
+    def forward(
+        self,
+        query: torch.Tensor,
+        key: torch.Tensor,
+        value: torch.Tensor,
+        mask: torch.Tensor = torch.ones((0, 0, 0), dtype=torch.bool),
+        pos_emb: torch.Tensor = torch.empty(0),
+        cache: torch.Tensor = torch.zeros((0, 0, 0, 0))
+    ) -> Tuple[torch.Tensor, torch.Tensor]:
+        """Compute 'Scaled Dot Product Attention' with rel. positional encoding.
+        Args:
+            query (torch.Tensor): Query tensor (#batch, time1, size).
+            key (torch.Tensor): Key tensor (#batch, time2, size).
+            value (torch.Tensor): Value tensor (#batch, time2, size).
+            mask (torch.Tensor): Mask tensor (#batch, 1, time2) or
+                (#batch, time1, time2), (0, 0, 0) means fake mask.
+            pos_emb (torch.Tensor): Positional embedding tensor
+                (#batch, time2, size).
+            cache (torch.Tensor): Cache tensor (1, head, cache_t, d_k * 2),
+                where `cache_t == chunk_size * num_decoding_left_chunks`
+                and `head * d_k == size`
+        Returns:
+            torch.Tensor: Output tensor (#batch, time1, d_model).
+            torch.Tensor: Cache tensor (1, head, cache_t + time1, d_k * 2)
+                where `cache_t == chunk_size * num_decoding_left_chunks`
+                and `head * d_k == size`
+        """
+        q, k, v = self.forward_qkv(query, key, value)
+        q = q.transpose(1, 2)  # (batch, time1, head, d_k)
+
+        # NOTE(xcsong):
+        #   when export onnx model, for 1st chunk, we feed
+        #       cache(1, head, 0, d_k * 2) (16/-1, -1/-1, 16/0 mode)
+        #       or cache(1, head, real_cache_t, d_k * 2) (16/4 mode).
+        #       In all modes, `if cache.size(0) > 0` will alwayse be `True`
+        #       and we will always do splitting and
+        #       concatnation(this will simplify onnx export). Note that
+        #       it's OK to concat & split zero-shaped tensors(see code below).
+        #   when export jit  model, for 1st chunk, we always feed
+        #       cache(0, 0, 0, 0) since jit supports dynamic if-branch.
+        # >>> a = torch.ones((1, 2, 0, 4))
+        # >>> b = torch.ones((1, 2, 3, 4))
+        # >>> c = torch.cat((a, b), dim=2)
+        # >>> torch.equal(b, c)        # True
+        # >>> d = torch.split(a, 2, dim=-1)
+        # >>> torch.equal(d[0], d[1])  # True
+        if cache.size(0) > 0:
+            key_cache, value_cache = torch.split(cache,
+                                                 cache.size(-1) // 2,
+                                                 dim=-1)
+            k = torch.cat([key_cache, k], dim=2)
+            v = torch.cat([value_cache, v], dim=2)
+        # NOTE(xcsong): We do cache slicing in encoder.forward_chunk, since it's
+        #   non-trivial to calculate `next_cache_start` here.
+        new_cache = torch.cat((k, v), dim=-1)
+
+        n_batch_pos = pos_emb.size(0)
+        p = self.linear_pos(pos_emb).view(n_batch_pos, -1, self.h, self.d_k)
+        p = p.transpose(1, 2)  # (batch, head, time1, d_k)
+
+        # (batch, head, time1, d_k)
+        q_with_bias_u = (q + self.pos_bias_u.to(q.device)).transpose(1, 2)
+        # (batch, head, time1, d_k)
+        q_with_bias_v = (q + self.pos_bias_v.to(q.device)).transpose(1, 2)
+
+        # compute attention score
+        # first compute matrix a and matrix c
+        # as described in https://arxiv.org/abs/1901.02860 Section 3.3
+        # (batch, head, time1, time2)
+        matrix_ac = torch.matmul(q_with_bias_u, k.transpose(-2, -1))
+
+        # compute matrix b and matrix d
+        # (batch, head, time1, time2)
+        matrix_bd = torch.matmul(q_with_bias_v, p.transpose(-2, -1))
+        # NOTE(Xiang Lyu): Keep rel_shift since espnet rel_pos_emb is used
+        if matrix_ac.shape != matrix_bd.shape:
+            matrix_bd = self.rel_shift(matrix_bd)
+
+        scores = (matrix_ac + matrix_bd) / math.sqrt(
+            self.d_k)  # (batch, head, time1, time2)
+
+        return self.forward_attention(v, scores, mask), new_cache

engines/chatterbox_official_23lang/models/s3gen/transformer/convolution.py

@@ -0,0 +1,145 @@
+# Copyright (c) 2020 Mobvoi Inc. (authors: Binbin Zhang, Di Wu)
+#               2024 Alibaba Inc (Xiang Lyu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Modified from ESPnet(https://github.com/espnet/espnet)
+"""ConvolutionModule definition."""
+
+from typing import Tuple
+
+import torch
+from torch import nn
+
+
+class ConvolutionModule(nn.Module):
+    """ConvolutionModule in Conformer model."""
+
+    def __init__(self,
+                 channels: int,
+                 kernel_size: int = 15,
+                 activation: nn.Module = nn.ReLU(),
+                 norm: str = "batch_norm",
+                 causal: bool = False,
+                 bias: bool = True):
+        """Construct an ConvolutionModule object.
+        Args:
+            channels (int): The number of channels of conv layers.
+            kernel_size (int): Kernel size of conv layers.
+            causal (int): Whether use causal convolution or not
+        """
+        super().__init__()
+
+        self.pointwise_conv1 = nn.Conv1d(
+            channels,
+            2 * channels,
+            kernel_size=1,
+            stride=1,
+            padding=0,
+            bias=bias,
+        )
+        # self.lorder is used to distinguish if it's a causal convolution,
+        # if self.lorder > 0: it's a causal convolution, the input will be
+        #    padded with self.lorder frames on the left in forward.
+        # else: it's a symmetrical convolution
+        if causal:
+            padding = 0
+            self.lorder = kernel_size - 1
+        else:
+            # kernel_size should be an odd number for none causal convolution
+            assert (kernel_size - 1) % 2 == 0
+            padding = (kernel_size - 1) // 2
+            self.lorder = 0
+        self.depthwise_conv = nn.Conv1d(
+            channels,
+            channels,
+            kernel_size,
+            stride=1,
+            padding=padding,
+            groups=channels,
+            bias=bias,
+        )
+
+        assert norm in ['batch_norm', 'layer_norm']
+        if norm == "batch_norm":
+            self.use_layer_norm = False
+            self.norm = nn.BatchNorm1d(channels)
+        else:
+            self.use_layer_norm = True
+            self.norm = nn.LayerNorm(channels)
+
+        self.pointwise_conv2 = nn.Conv1d(
+            channels,
+            channels,
+            kernel_size=1,
+            stride=1,
+            padding=0,
+            bias=bias,
+        )
+        self.activation = activation
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        mask_pad: torch.Tensor = torch.ones((0, 0, 0), dtype=torch.bool),
+        cache: torch.Tensor = torch.zeros((0, 0, 0)),
+    ) -> Tuple[torch.Tensor, torch.Tensor]:
+        """Compute convolution module.
+        Args:
+            x (torch.Tensor): Input tensor (#batch, time, channels).
+            mask_pad (torch.Tensor): used for batch padding (#batch, 1, time),
+                (0, 0, 0) means fake mask.
+            cache (torch.Tensor): left context cache, it is only
+                used in causal convolution (#batch, channels, cache_t),
+                (0, 0, 0) meas fake cache.
+        Returns:
+            torch.Tensor: Output tensor (#batch, time, channels).
+        """
+        # exchange the temporal dimension and the feature dimension
+        x = x.transpose(1, 2)  # (#batch, channels, time)
+
+        # mask batch padding
+        if mask_pad.size(2) > 0:  # time > 0
+            x.masked_fill_(~mask_pad, 0.0)
+
+        if self.lorder > 0:
+            if cache.size(2) == 0:  # cache_t == 0
+                x = nn.functional.pad(x, (self.lorder, 0), 'constant', 0.0)
+            else:
+                assert cache.size(0) == x.size(0)  # equal batch
+                assert cache.size(1) == x.size(1)  # equal channel
+                x = torch.cat((cache, x), dim=2)
+            assert (x.size(2) > self.lorder)
+            new_cache = x[:, :, -self.lorder:]
+        else:
+            # It's better we just return None if no cache is required,
+            # However, for JIT export, here we just fake one tensor instead of
+            # None.
+            new_cache = torch.zeros((0, 0, 0), dtype=x.dtype, device=x.device)
+
+        # GLU mechanism
+        x = self.pointwise_conv1(x)  # (batch, 2*channel, dim)
+        x = nn.functional.glu(x, dim=1)  # (batch, channel, dim)
+
+        # 1D Depthwise Conv
+        x = self.depthwise_conv(x)
+        if self.use_layer_norm:
+            x = x.transpose(1, 2)
+        x = self.activation(self.norm(x))
+        if self.use_layer_norm:
+            x = x.transpose(1, 2)
+        x = self.pointwise_conv2(x)
+        # mask batch padding
+        if mask_pad.size(2) > 0:  # time > 0
+            x.masked_fill_(~mask_pad, 0.0)
+
+        return x.transpose(1, 2), new_cache

engines/chatterbox_official_23lang/models/s3gen/transformer/embedding.py

@@ -0,0 +1,294 @@
+# Copyright (c) 2020 Mobvoi Inc. (authors: Binbin Zhang, Di Wu)
+#               2024 Alibaba Inc (Xiang Lyu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Modified from ESPnet(https://github.com/espnet/espnet)
+"""Positonal Encoding Module."""
+
+import math
+from typing import Tuple, Union
+
+import torch
+import torch.nn.functional as F
+import numpy as np
+
+
+class PositionalEncoding(torch.nn.Module):
+    """Positional encoding.
+
+    :param int d_model: embedding dim
+    :param float dropout_rate: dropout rate
+    :param int max_len: maximum input length
+
+    PE(pos, 2i)   = sin(pos/(10000^(2i/dmodel)))
+    PE(pos, 2i+1) = cos(pos/(10000^(2i/dmodel)))
+    """
+
+    def __init__(self,
+                 d_model: int,
+                 dropout_rate: float,
+                 max_len: int = 5000,
+                 reverse: bool = False):
+        """Construct an PositionalEncoding object."""
+        super().__init__()
+        self.d_model = d_model
+        self.xscale = math.sqrt(self.d_model)
+        self.dropout = torch.nn.Dropout(p=dropout_rate)
+        self.max_len = max_len
+
+        self.pe = torch.zeros(self.max_len, self.d_model)
+        position = torch.arange(0, self.max_len,
+                                dtype=torch.float32).unsqueeze(1)
+        div_term = torch.exp(
+            torch.arange(0, self.d_model, 2, dtype=torch.float32) *
+            -(math.log(10000.0) / self.d_model))
+        self.pe[:, 0::2] = torch.sin(position * div_term)
+        self.pe[:, 1::2] = torch.cos(position * div_term)
+        self.pe = self.pe.unsqueeze(0)
+
+    def forward(self,
+                x: torch.Tensor,
+                offset: Union[int, torch.Tensor] = 0) \
+            -> Tuple[torch.Tensor, torch.Tensor]:
+        """Add positional encoding.
+
+        Args:
+            x (torch.Tensor): Input. Its shape is (batch, time, ...)
+            offset (int, torch.tensor): position offset
+
+        Returns:
+            torch.Tensor: Encoded tensor. Its shape is (batch, time, ...)
+            torch.Tensor: for compatibility to RelPositionalEncoding
+        """
+
+        self.pe = self.pe.to(x.device)
+        pos_emb = self.position_encoding(offset, x.size(1), False)
+        x = x * self.xscale + pos_emb
+        return self.dropout(x), self.dropout(pos_emb)
+
+    def position_encoding(self,
+                          offset: Union[int, torch.Tensor],
+                          size: int,
+                          apply_dropout: bool = True) -> torch.Tensor:
+        """ For getting encoding in a streaming fashion
+
+        Attention!!!!!
+        we apply dropout only once at the whole utterance level in a none
+        streaming way, but will call this function several times with
+        increasing input size in a streaming scenario, so the dropout will
+        be applied several times.
+
+        Args:
+            offset (int or torch.tensor): start offset
+            size (int): required size of position encoding
+
+        Returns:
+            torch.Tensor: Corresponding encoding
+        """
+        # How to subscript a Union type:
+        #   https://github.com/pytorch/pytorch/issues/69434
+        if isinstance(offset, int):
+            assert offset + size <= self.max_len
+            pos_emb = self.pe[:, offset:offset + size]
+        elif isinstance(offset, torch.Tensor) and offset.dim() == 0:  # scalar
+            assert offset + size <= self.max_len
+            pos_emb = self.pe[:, offset:offset + size]
+        else:  # for batched streaming decoding on GPU
+            assert torch.max(offset) + size <= self.max_len
+            index = offset.unsqueeze(1) + \
+                torch.arange(0, size).to(offset.device)  # B X T
+            flag = index > 0
+            # remove negative offset
+            index = index * flag
+            pos_emb = F.embedding(index, self.pe[0])  # B X T X d_model
+
+        if apply_dropout:
+            pos_emb = self.dropout(pos_emb)
+        return pos_emb
+
+
+class RelPositionalEncoding(PositionalEncoding):
+    """Relative positional encoding module.
+    See : Appendix B in https://arxiv.org/abs/1901.02860
+    Args:
+        d_model (int): Embedding dimension.
+        dropout_rate (float): Dropout rate.
+        max_len (int): Maximum input length.
+    """
+
+    def __init__(self, d_model: int, dropout_rate: float, max_len: int = 5000):
+        """Initialize class."""
+        super().__init__(d_model, dropout_rate, max_len, reverse=True)
+
+    def forward(self,
+                x: torch.Tensor,
+                offset: Union[int, torch.Tensor] = 0) \
+            -> Tuple[torch.Tensor, torch.Tensor]:
+        """Compute positional encoding.
+        Args:
+            x (torch.Tensor): Input tensor (batch, time, `*`).
+        Returns:
+            torch.Tensor: Encoded tensor (batch, time, `*`).
+            torch.Tensor: Positional embedding tensor (1, time, `*`).
+        """
+        self.pe = self.pe.to(x.device)
+        x = x * self.xscale
+        pos_emb = self.position_encoding(offset, x.size(1), False)
+        return self.dropout(x), self.dropout(pos_emb)
+
+
+class WhisperPositionalEncoding(PositionalEncoding):
+    """ Sinusoids position encoding used in openai-whisper.encoder
+    """
+
+    def __init__(self, d_model: int, dropout_rate: float, max_len: int = 1500):
+        super().__init__(d_model, dropout_rate, max_len)
+        self.xscale = 1.0
+        log_timescale_increment = np.log(10000) / (d_model // 2 - 1)
+        inv_timescales = torch.exp(-log_timescale_increment *
+                                   torch.arange(d_model // 2))
+        scaled_time = torch.arange(max_len)[:, np.newaxis] * \
+            inv_timescales[np.newaxis, :]
+        pe = torch.cat([torch.sin(scaled_time), torch.cos(scaled_time)], dim=1)
+        delattr(self, "pe")
+        self.register_buffer("pe", pe.unsqueeze(0))
+
+
+class LearnablePositionalEncoding(PositionalEncoding):
+    """ Learnable position encoding used in openai-whisper.decoder
+    """
+
+    def __init__(self, d_model: int, dropout_rate: float, max_len: int = 448):
+        super().__init__(d_model, dropout_rate, max_len)
+        # NOTE(xcsong): overwrite self.pe & self.xscale
+        self.pe = torch.nn.Parameter(torch.empty(1, max_len, d_model))
+        self.xscale = 1.0
+
+
+class NoPositionalEncoding(torch.nn.Module):
+    """ No position encoding
+    """
+
+    def __init__(self, d_model: int, dropout_rate: float):
+        super().__init__()
+        self.d_model = d_model
+        self.dropout = torch.nn.Dropout(p=dropout_rate)
+
+    def forward(self,
+                x: torch.Tensor,
+                offset: Union[int, torch.Tensor] = 0) \
+            -> Tuple[torch.Tensor, torch.Tensor]:
+        """ Just return zero vector for interface compatibility
+        """
+        pos_emb = torch.zeros(1, x.size(1), self.d_model).to(x.device)
+        return self.dropout(x), pos_emb
+
+    def position_encoding(self, offset: Union[int, torch.Tensor],
+                          size: int) -> torch.Tensor:
+        return torch.zeros(1, size, self.d_model)
+
+
+class EspnetRelPositionalEncoding(torch.nn.Module):
+    """Relative positional encoding module (new implementation).
+
+    Details can be found in https://github.com/espnet/espnet/pull/2816.
+
+    See : Appendix B in https://arxiv.org/abs/1901.02860
+
+    Args:
+        d_model (int): Embedding dimension.
+        dropout_rate (float): Dropout rate.
+        max_len (int): Maximum input length.
+
+    """
+
+    def __init__(self, d_model: int, dropout_rate: float, max_len: int = 5000):
+        """Construct an PositionalEncoding object."""
+        super(EspnetRelPositionalEncoding, self).__init__()
+        self.d_model = d_model
+        self.xscale = math.sqrt(self.d_model)
+        self.dropout = torch.nn.Dropout(p=dropout_rate)
+        self.pe = None
+        self.extend_pe(torch.tensor(0.0).expand(1, max_len))
+
+    def extend_pe(self, x: torch.Tensor):
+        """Reset the positional encodings."""
+        if self.pe is not None:
+            # self.pe contains both positive and negative parts
+            # the length of self.pe is 2 * input_len - 1
+            if self.pe.size(1) >= x.size(1) * 2 - 1:
+                if self.pe.dtype != x.dtype or self.pe.device != x.device:
+                    self.pe = self.pe.to(dtype=x.dtype, device=x.device)
+                return
+        # Suppose `i` means to the position of query vecotr and `j` means the
+        # position of key vector. We use position relative positions when keys
+        # are to the left (i>j) and negative relative positions otherwise (i<j).
+        pe_positive = torch.zeros(x.size(1), self.d_model)
+        pe_negative = torch.zeros(x.size(1), self.d_model)
+        position = torch.arange(0, x.size(1), dtype=torch.float32).unsqueeze(1)
+        div_term = torch.exp(
+            torch.arange(0, self.d_model, 2, dtype=torch.float32)
+            * -(math.log(10000.0) / self.d_model)
+        )
+        pe_positive[:, 0::2] = torch.sin(position * div_term)
+        pe_positive[:, 1::2] = torch.cos(position * div_term)
+        pe_negative[:, 0::2] = torch.sin(-1 * position * div_term)
+        pe_negative[:, 1::2] = torch.cos(-1 * position * div_term)
+
+        # Reserve the order of positive indices and concat both positive and
+        # negative indices. This is used to support the shifting trick
+        # as in https://arxiv.org/abs/1901.02860
+        pe_positive = torch.flip(pe_positive, [0]).unsqueeze(0)
+        pe_negative = pe_negative[1:].unsqueeze(0)
+        pe = torch.cat([pe_positive, pe_negative], dim=1)
+        self.pe = pe.to(device=x.device, dtype=x.dtype)
+
+    def forward(self, x: torch.Tensor, offset: Union[int, torch.Tensor] = 0) \
+            -> Tuple[torch.Tensor, torch.Tensor]:
+        """Add positional encoding.
+
+        Args:
+            x (torch.Tensor): Input tensor (batch, time, `*`).
+
+        Returns:
+            torch.Tensor: Encoded tensor (batch, time, `*`).
+
+        """
+        self.extend_pe(x)
+        x = x * self.xscale
+        pos_emb = self.position_encoding(size=x.size(1), offset=offset)
+        return self.dropout(x), self.dropout(pos_emb)
+
+    def position_encoding(self,
+                          offset: Union[int, torch.Tensor],
+                          size: int) -> torch.Tensor:
+        """ For getting encoding in a streaming fashion
+
+        Attention!!!!!
+        we apply dropout only once at the whole utterance level in a none
+        streaming way, but will call this function several times with
+        increasing input size in a streaming scenario, so the dropout will
+        be applied several times.
+
+        Args:
+            offset (int or torch.tensor): start offset
+            size (int): required size of position encoding
+
+        Returns:
+            torch.Tensor: Corresponding encoding
+        """
+        pos_emb = self.pe[
+            :,
+            self.pe.size(1) // 2 - size + 1: self.pe.size(1) // 2 + size,
+        ]
+        return pos_emb

engines/chatterbox_official_23lang/models/s3gen/transformer/encoder_layer.py

@@ -0,0 +1,236 @@
+# Copyright (c) 2021 Mobvoi Inc (Binbin Zhang, Di Wu)
+#               2022 Xingchen Song (sxc19@mails.tsinghua.edu.cn)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Modified from ESPnet(https://github.com/espnet/espnet)
+"""Encoder self-attention layer definition."""
+
+from typing import Optional, Tuple
+
+import torch
+from torch import nn
+
+
+class TransformerEncoderLayer(nn.Module):
+    """Encoder layer module.
+
+    Args:
+        size (int): Input dimension.
+        self_attn (torch.nn.Module): Self-attention module instance.
+            `MultiHeadedAttention` or `RelPositionMultiHeadedAttention`
+            instance can be used as the argument.
+        feed_forward (torch.nn.Module): Feed-forward module instance.
+            `PositionwiseFeedForward`, instance can be used as the argument.
+        dropout_rate (float): Dropout rate.
+        normalize_before (bool):
+            True: use layer_norm before each sub-block.
+            False: to use layer_norm after each sub-block.
+    """
+
+    def __init__(
+        self,
+        size: int,
+        self_attn: torch.nn.Module,
+        feed_forward: torch.nn.Module,
+        dropout_rate: float,
+        normalize_before: bool = True,
+    ):
+        """Construct an EncoderLayer object."""
+        super().__init__()
+        self.self_attn = self_attn
+        self.feed_forward = feed_forward
+        self.norm1 = nn.LayerNorm(size, eps=1e-12)
+        self.norm2 = nn.LayerNorm(size, eps=1e-12)
+        self.dropout = nn.Dropout(dropout_rate)
+        self.size = size
+        self.normalize_before = normalize_before
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        mask: torch.Tensor,
+        pos_emb: torch.Tensor,
+        mask_pad: torch.Tensor = torch.ones((0, 0, 0), dtype=torch.bool),
+        att_cache: torch.Tensor = torch.zeros((0, 0, 0, 0)),
+        cnn_cache: torch.Tensor = torch.zeros((0, 0, 0, 0)),
+    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
+        """Compute encoded features.
+
+        Args:
+            x (torch.Tensor): (#batch, time, size)
+            mask (torch.Tensor): Mask tensor for the input (#batch, time，time),
+                (0, 0, 0) means fake mask.
+            pos_emb (torch.Tensor): just for interface compatibility
+                to ConformerEncoderLayer
+            mask_pad (torch.Tensor): does not used in transformer layer,
+                just for unified api with conformer.
+            att_cache (torch.Tensor): Cache tensor of the KEY & VALUE
+                (#batch=1, head, cache_t1, d_k * 2), head * d_k == size.
+            cnn_cache (torch.Tensor): Convolution cache in conformer layer
+                (#batch=1, size, cache_t2), not used here, it's for interface
+                compatibility to ConformerEncoderLayer.
+        Returns:
+            torch.Tensor: Output tensor (#batch, time, size).
+            torch.Tensor: Mask tensor (#batch, time, time).
+            torch.Tensor: att_cache tensor,
+                (#batch=1, head, cache_t1 + time, d_k * 2).
+            torch.Tensor: cnn_cahce tensor (#batch=1, size, cache_t2).
+
+        """
+        residual = x
+        if self.normalize_before:
+            x = self.norm1(x)
+        x_att, new_att_cache = self.self_attn(x, x, x, mask, pos_emb=pos_emb, cache=att_cache)
+        x = residual + self.dropout(x_att)
+        if not self.normalize_before:
+            x = self.norm1(x)
+
+        residual = x
+        if self.normalize_before:
+            x = self.norm2(x)
+        x = residual + self.dropout(self.feed_forward(x))
+        if not self.normalize_before:
+            x = self.norm2(x)
+
+        fake_cnn_cache = torch.zeros((0, 0, 0), dtype=x.dtype, device=x.device)
+        return x, mask, new_att_cache, fake_cnn_cache
+
+
+class ConformerEncoderLayer(nn.Module):
+    """Encoder layer module.
+    Args:
+        size (int): Input dimension.
+        self_attn (torch.nn.Module): Self-attention module instance.
+            `MultiHeadedAttention` or `RelPositionMultiHeadedAttention`
+            instance can be used as the argument.
+        feed_forward (torch.nn.Module): Feed-forward module instance.
+            `PositionwiseFeedForward` instance can be used as the argument.
+        feed_forward_macaron (torch.nn.Module): Additional feed-forward module
+             instance.
+            `PositionwiseFeedForward` instance can be used as the argument.
+        conv_module (torch.nn.Module): Convolution module instance.
+            `ConvlutionModule` instance can be used as the argument.
+        dropout_rate (float): Dropout rate.
+        normalize_before (bool):
+            True: use layer_norm before each sub-block.
+            False: use layer_norm after each sub-block.
+    """
+
+    def __init__(
+        self,
+        size: int,
+        self_attn: torch.nn.Module,
+        feed_forward: Optional[nn.Module] = None,
+        feed_forward_macaron: Optional[nn.Module] = None,
+        conv_module: Optional[nn.Module] = None,
+        dropout_rate: float = 0.1,
+        normalize_before: bool = True,
+    ):
+        """Construct an EncoderLayer object."""
+        super().__init__()
+        self.self_attn = self_attn
+        self.feed_forward = feed_forward
+        self.feed_forward_macaron = feed_forward_macaron
+        self.conv_module = conv_module
+        self.norm_ff = nn.LayerNorm(size, eps=1e-12)  # for the FNN module
+        self.norm_mha = nn.LayerNorm(size, eps=1e-12)  # for the MHA module
+        if feed_forward_macaron is not None:
+            self.norm_ff_macaron = nn.LayerNorm(size, eps=1e-12)
+            self.ff_scale = 0.5
+        else:
+            self.ff_scale = 1.0
+        if self.conv_module is not None:
+            self.norm_conv = nn.LayerNorm(size, eps=1e-12)  # for the CNN module
+            self.norm_final = nn.LayerNorm(
+                size, eps=1e-12)  # for the final output of the block
+        self.dropout = nn.Dropout(dropout_rate)
+        self.size = size
+        self.normalize_before = normalize_before
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        mask: torch.Tensor,
+        pos_emb: torch.Tensor,
+        mask_pad: torch.Tensor = torch.ones((0, 0, 0), dtype=torch.bool),
+        att_cache: torch.Tensor = torch.zeros((0, 0, 0, 0)),
+        cnn_cache: torch.Tensor = torch.zeros((0, 0, 0, 0)),
+    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
+        """Compute encoded features.
+
+        Args:
+            x (torch.Tensor): (#batch, time, size)
+            mask (torch.Tensor): Mask tensor for the input (#batch, time，time),
+                (0, 0, 0) means fake mask.
+            pos_emb (torch.Tensor): positional encoding, must not be None
+                for ConformerEncoderLayer.
+            mask_pad (torch.Tensor): batch padding mask used for conv module.
+                (#batch, 1，time), (0, 0, 0) means fake mask.
+            att_cache (torch.Tensor): Cache tensor of the KEY & VALUE
+                (#batch=1, head, cache_t1, d_k * 2), head * d_k == size.
+            cnn_cache (torch.Tensor): Convolution cache in conformer layer
+                (#batch=1, size, cache_t2)
+        Returns:
+            torch.Tensor: Output tensor (#batch, time, size).
+            torch.Tensor: Mask tensor (#batch, time, time).
+            torch.Tensor: att_cache tensor,
+                (#batch=1, head, cache_t1 + time, d_k * 2).
+            torch.Tensor: cnn_cahce tensor (#batch, size, cache_t2).
+        """
+
+        # whether to use macaron style
+        if self.feed_forward_macaron is not None:
+            residual = x
+            if self.normalize_before:
+                x = self.norm_ff_macaron(x)
+            x = residual + self.ff_scale * self.dropout(
+                self.feed_forward_macaron(x))
+            if not self.normalize_before:
+                x = self.norm_ff_macaron(x)
+
+        # multi-headed self-attention module
+        residual = x
+        if self.normalize_before:
+            x = self.norm_mha(x)
+        x_att, new_att_cache = self.self_attn(x, x, x, mask, pos_emb,
+                                              att_cache)
+        x = residual + self.dropout(x_att)
+        if not self.normalize_before:
+            x = self.norm_mha(x)
+
+        # convolution module
+        # Fake new cnn cache here, and then change it in conv_module
+        new_cnn_cache = torch.zeros((0, 0, 0), dtype=x.dtype, device=x.device)
+        if self.conv_module is not None:
+            residual = x
+            if self.normalize_before:
+                x = self.norm_conv(x)
+            x, new_cnn_cache = self.conv_module(x, mask_pad, cnn_cache)
+            x = residual + self.dropout(x)
+
+            if not self.normalize_before:
+                x = self.norm_conv(x)
+
+        # feed forward module
+        residual = x
+        if self.normalize_before:
+            x = self.norm_ff(x)
+
+        x = residual + self.ff_scale * self.dropout(self.feed_forward(x))
+        if not self.normalize_before:
+            x = self.norm_ff(x)
+
+        if self.conv_module is not None:
+            x = self.norm_final(x)
+
+        return x, mask, new_att_cache, new_cnn_cache

engines/chatterbox_official_23lang/models/s3gen/transformer/positionwise_feed_forward.py

@@ -0,0 +1,115 @@
+# Copyright (c) 2019 Shigeki Karita
+#               2020 Mobvoi Inc (Binbin Zhang)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Positionwise feed forward layer definition."""
+
+import torch
+
+
+class PositionwiseFeedForward(torch.nn.Module):
+    """Positionwise feed forward layer.
+
+    FeedForward are appied on each position of the sequence.
+    The output dim is same with the input dim.
+
+    Args:
+        idim (int): Input dimenstion.
+        hidden_units (int): The number of hidden units.
+        dropout_rate (float): Dropout rate.
+        activation (torch.nn.Module): Activation function
+    """
+
+    def __init__(
+            self,
+            idim: int,
+            hidden_units: int,
+            dropout_rate: float,
+            activation: torch.nn.Module = torch.nn.ReLU(),
+    ):
+        """Construct a PositionwiseFeedForward object."""
+        super(PositionwiseFeedForward, self).__init__()
+        self.w_1 = torch.nn.Linear(idim, hidden_units)
+        self.activation = activation
+        self.dropout = torch.nn.Dropout(dropout_rate)
+        self.w_2 = torch.nn.Linear(hidden_units, idim)
+
+    def forward(self, xs: torch.Tensor) -> torch.Tensor:
+        """Forward function.
+
+        Args:
+            xs: input tensor (B, L, D)
+        Returns:
+            output tensor, (B, L, D)
+        """
+        return self.w_2(self.dropout(self.activation(self.w_1(xs))))
+
+
+class MoEFFNLayer(torch.nn.Module):
+    """
+    Mixture of expert with Positionwise feed forward layer
+    See also figure 1 in https://arxiv.org/pdf/2305.15663.pdf
+    The output dim is same with the input dim.
+
+    Modified from https://github.com/Lightning-AI/lit-gpt/pull/823
+                  https://github.com/mistralai/mistral-src/blob/b46d6/moe_one_file_ref.py#L203-L219
+    Args:
+        n_expert: number of expert.
+        n_expert_per_token: The actual number of experts used for each frame
+        idim (int): Input dimenstion.
+        hidden_units (int): The number of hidden units.
+        dropout_rate (float): Dropout rate.
+        activation (torch.nn.Module): Activation function
+    """
+
+    def __init__(
+            self,
+            n_expert: int,
+            n_expert_per_token: int,
+            idim: int,
+            hidden_units: int,
+            dropout_rate: float,
+            activation: torch.nn.Module = torch.nn.ReLU(),
+    ):
+        super(MoEFFNLayer, self).__init__()
+        self.gate = torch.nn.Linear(idim, n_expert, bias=False)
+        self.experts = torch.nn.ModuleList(
+            PositionwiseFeedForward(idim, hidden_units, dropout_rate,
+                                    activation) for _ in range(n_expert))
+        self.n_expert_per_token = n_expert_per_token
+
+    def forward(self, xs: torch.Tensor) -> torch.Tensor:
+        """Foward function.
+        Args:
+            xs: input tensor (B, L, D)
+        Returns:
+            output tensor, (B, L, D)
+
+        """
+        B, L, D = xs.size(
+        )  # batch size, sequence length, embedding dimension (idim)
+        xs = xs.view(-1, D)  # (B*L, D)
+        router = self.gate(xs)  # (B*L, n_expert)
+        logits, indices = torch.topk(
+            router, self.n_expert_per_token
+        )  # probs:(B*L, n_expert), indices: (B*L, n_expert)
+        weights = torch.nn.functional.softmax(
+            logits, dim=1,
+            dtype=torch.float).to(dtype=xs.dtype)  # (B*L, n_expert_per_token)
+        output = torch.zeros_like(xs)  # (B*L, D)
+        for i, expert in enumerate(self.experts):
+            mask = indices == i
+            batch_idx, ith_expert = torch.where(mask)
+            output[batch_idx] += weights[batch_idx, ith_expert, None] * expert(
+                xs[batch_idx])
+        return output.view(B, L, D)

engines/chatterbox_official_23lang/models/s3gen/transformer/subsampling.py

@@ -0,0 +1,383 @@
+# Copyright (c) 2021 Mobvoi Inc (Binbin Zhang, Di Wu)
+#               2024 Alibaba Inc (Xiang Lyu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Modified from ESPnet(https://github.com/espnet/espnet)
+"""Subsampling layer definition."""
+
+from typing import Tuple, Union
+
+import torch
+
+
+class BaseSubsampling(torch.nn.Module):
+
+    def __init__(self):
+        super().__init__()
+        self.right_context = 0
+        self.subsampling_rate = 1
+
+    def position_encoding(self, offset: Union[int, torch.Tensor],
+                          size: int) -> torch.Tensor:
+        return self.pos_enc.position_encoding(offset, size)
+
+
+class EmbedinigNoSubsampling(BaseSubsampling):
+    """Embedding input without subsampling
+    """
+
+    def __init__(self, idim: int, odim: int, dropout_rate: float,
+                 pos_enc_class: torch.nn.Module):
+        super().__init__()
+        self.embed = torch.nn.Embedding(idim, odim)
+        self.pos_enc = pos_enc_class
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        x_mask: torch.Tensor,
+        offset: Union[int, torch.Tensor] = 0
+    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        """Input x.
+
+        Args:
+            x (torch.Tensor): Input tensor (#batch, time, idim).
+            x_mask (torch.Tensor): Input mask (#batch, 1, time).
+
+        Returns:
+            torch.Tensor: linear input tensor (#batch, time', odim),
+                where time' = time .
+            torch.Tensor: linear input mask (#batch, 1, time'),
+                where time' = time .
+
+        """
+        x = self.embed(x)
+        x, pos_emb = self.pos_enc(x, offset)
+        return x, pos_emb, x_mask
+
+
+class LinearNoSubsampling(BaseSubsampling):
+    """Linear transform the input without subsampling
+
+    Args:
+        idim (int): Input dimension.
+        odim (int): Output dimension.
+        dropout_rate (float): Dropout rate.
+
+    """
+
+    def __init__(self, idim: int, odim: int, dropout_rate: float,
+                 pos_enc_class: torch.nn.Module):
+        """Construct an linear object."""
+        super().__init__()
+        self.out = torch.nn.Sequential(
+            torch.nn.Linear(idim, odim),
+            torch.nn.LayerNorm(odim, eps=1e-5),
+            torch.nn.Dropout(dropout_rate),
+        )
+        self.pos_enc = pos_enc_class
+        self.right_context = 0
+        self.subsampling_rate = 1
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        x_mask: torch.Tensor,
+        offset: Union[int, torch.Tensor] = 0
+    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        """Input x.
+
+        Args:
+            x (torch.Tensor): Input tensor (#batch, time, idim).
+            x_mask (torch.Tensor): Input mask (#batch, 1, time).
+
+        Returns:
+            torch.Tensor: linear input tensor (#batch, time', odim),
+                where time' = time .
+            torch.Tensor: linear input mask (#batch, 1, time'),
+                where time' = time .
+
+        """
+        x = self.out(x)
+        x, pos_emb = self.pos_enc(x, offset)
+        return x, pos_emb, x_mask
+
+
+class Conv1dSubsampling2(BaseSubsampling):
+    """Convolutional 1D subsampling (to 1/2 length).
+       It is designed for Whisper, ref:
+       https://github.com/openai/whisper/blob/main/whisper/model.py
+
+    Args:
+        idim (int): Input dimension.
+        odim (int): Output dimension.
+        dropout_rate (float): Dropout rate.
+
+    """
+
+    def __init__(self, idim: int, odim: int, dropout_rate: float,
+                 pos_enc_class: torch.nn.Module):
+        """Construct an Conv1dSubsampling2 object."""
+        super().__init__()
+        self.conv = torch.nn.Sequential(
+            torch.nn.Conv1d(idim, odim, kernel_size=3, padding=1),
+            torch.nn.GELU(),
+            torch.nn.Conv1d(odim, odim, kernel_size=3, stride=2, padding=1),
+            torch.nn.GELU(),
+        )
+        self.pos_enc = pos_enc_class
+        # The right context for every conv layer is computed by:
+        # (kernel_size - 1) * frame_rate_of_this_layer
+        self.subsampling_rate = 2
+        # 4 = (3 - 1) * 1 + (3 - 1) * 1
+        self.right_context = 4
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        x_mask: torch.Tensor,
+        offset: Union[int, torch.Tensor] = 0
+    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        """Subsample x.
+
+        Args:
+            x (torch.Tensor): Input tensor (#batch, time, idim).
+            x_mask (torch.Tensor): Input mask (#batch, 1, time).
+
+        Returns:
+            torch.Tensor: Subsampled tensor (#batch, time', odim),
+                where time' = time // 2.
+            torch.Tensor: Subsampled mask (#batch, 1, time'),
+                where time' = time // 2.
+            torch.Tensor: positional encoding
+
+        """
+        time = x.size(1)
+        x = x.transpose(1, 2)  # (b, f, t)
+        x = self.conv(x)
+        x = x.transpose(1, 2)  # (b, t, f)
+        x, pos_emb = self.pos_enc(x, offset)
+        return x, pos_emb, x_mask[:, :, (time + 1) % 2::2]
+
+
+class Conv2dSubsampling4(BaseSubsampling):
+    """Convolutional 2D subsampling (to 1/4 length).
+
+    Args:
+        idim (int): Input dimension.
+        odim (int): Output dimension.
+        dropout_rate (float): Dropout rate.
+
+    """
+
+    def __init__(self, idim: int, odim: int, dropout_rate: float,
+                 pos_enc_class: torch.nn.Module):
+        """Construct an Conv2dSubsampling4 object."""
+        super().__init__()
+        self.conv = torch.nn.Sequential(
+            torch.nn.Conv2d(1, odim, 3, 2),
+            torch.nn.ReLU(),
+            torch.nn.Conv2d(odim, odim, 3, 2),
+            torch.nn.ReLU(),
+        )
+        self.out = torch.nn.Sequential(
+            torch.nn.Linear(odim * (((idim - 1) // 2 - 1) // 2), odim))
+        self.pos_enc = pos_enc_class
+        # The right context for every conv layer is computed by:
+        # (kernel_size - 1) * frame_rate_of_this_layer
+        self.subsampling_rate = 4
+        # 6 = (3 - 1) * 1 + (3 - 1) * 2
+        self.right_context = 6
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        x_mask: torch.Tensor,
+        offset: Union[int, torch.Tensor] = 0
+    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        """Subsample x.
+
+        Args:
+            x (torch.Tensor): Input tensor (#batch, time, idim).
+            x_mask (torch.Tensor): Input mask (#batch, 1, time).
+
+        Returns:
+            torch.Tensor: Subsampled tensor (#batch, time', odim),
+                where time' = time // 4.
+            torch.Tensor: Subsampled mask (#batch, 1, time'),
+                where time' = time // 4.
+            torch.Tensor: positional encoding
+
+        """
+        x = x.unsqueeze(1)  # (b, c=1, t, f)
+        x = self.conv(x)
+        b, c, t, f = x.size()
+        x = self.out(x.transpose(1, 2).contiguous().view(b, t, c * f))
+        x, pos_emb = self.pos_enc(x, offset)
+        return x, pos_emb, x_mask[:, :, 2::2][:, :, 2::2]
+
+
+class Conv2dSubsampling6(BaseSubsampling):
+    """Convolutional 2D subsampling (to 1/6 length).
+    Args:
+        idim (int): Input dimension.
+        odim (int): Output dimension.
+        dropout_rate (float): Dropout rate.
+        pos_enc (torch.nn.Module): Custom position encoding layer.
+    """
+
+    def __init__(self, idim: int, odim: int, dropout_rate: float,
+                 pos_enc_class: torch.nn.Module):
+        """Construct an Conv2dSubsampling6 object."""
+        super().__init__()
+        self.conv = torch.nn.Sequential(
+            torch.nn.Conv2d(1, odim, 3, 2),
+            torch.nn.ReLU(),
+            torch.nn.Conv2d(odim, odim, 5, 3),
+            torch.nn.ReLU(),
+        )
+        self.linear = torch.nn.Linear(odim * (((idim - 1) // 2 - 2) // 3),
+                                      odim)
+        self.pos_enc = pos_enc_class
+        # 10 = (3 - 1) * 1 + (5 - 1) * 2
+        self.subsampling_rate = 6
+        self.right_context = 10
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        x_mask: torch.Tensor,
+        offset: Union[int, torch.Tensor] = 0
+    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        """Subsample x.
+        Args:
+            x (torch.Tensor): Input tensor (#batch, time, idim).
+            x_mask (torch.Tensor): Input mask (#batch, 1, time).
+
+        Returns:
+            torch.Tensor: Subsampled tensor (#batch, time', odim),
+                where time' = time // 6.
+            torch.Tensor: Subsampled mask (#batch, 1, time'),
+                where time' = time // 6.
+            torch.Tensor: positional encoding
+        """
+        x = x.unsqueeze(1)  # (b, c, t, f)
+        x = self.conv(x)
+        b, c, t, f = x.size()
+        x = self.linear(x.transpose(1, 2).contiguous().view(b, t, c * f))
+        x, pos_emb = self.pos_enc(x, offset)
+        return x, pos_emb, x_mask[:, :, 2::2][:, :, 4::3]
+
+
+class Conv2dSubsampling8(BaseSubsampling):
+    """Convolutional 2D subsampling (to 1/8 length).
+
+    Args:
+        idim (int): Input dimension.
+        odim (int): Output dimension.
+        dropout_rate (float): Dropout rate.
+
+    """
+
+    def __init__(self, idim: int, odim: int, dropout_rate: float,
+                 pos_enc_class: torch.nn.Module):
+        """Construct an Conv2dSubsampling8 object."""
+        super().__init__()
+        self.conv = torch.nn.Sequential(
+            torch.nn.Conv2d(1, odim, 3, 2),
+            torch.nn.ReLU(),
+            torch.nn.Conv2d(odim, odim, 3, 2),
+            torch.nn.ReLU(),
+            torch.nn.Conv2d(odim, odim, 3, 2),
+            torch.nn.ReLU(),
+        )
+        self.linear = torch.nn.Linear(
+            odim * ((((idim - 1) // 2 - 1) // 2 - 1) // 2), odim)
+        self.pos_enc = pos_enc_class
+        self.subsampling_rate = 8
+        # 14 = (3 - 1) * 1 + (3 - 1) * 2 + (3 - 1) * 4
+        self.right_context = 14
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        x_mask: torch.Tensor,
+        offset: Union[int, torch.Tensor] = 0
+    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        """Subsample x.
+
+        Args:
+            x (torch.Tensor): Input tensor (#batch, time, idim).
+            x_mask (torch.Tensor): Input mask (#batch, 1, time).
+
+        Returns:
+            torch.Tensor: Subsampled tensor (#batch, time', odim),
+                where time' = time // 8.
+            torch.Tensor: Subsampled mask (#batch, 1, time'),
+                where time' = time // 8.
+            torch.Tensor: positional encoding
+        """
+        x = x.unsqueeze(1)  # (b, c, t, f)
+        x = self.conv(x)
+        b, c, t, f = x.size()
+        x = self.linear(x.transpose(1, 2).contiguous().view(b, t, c * f))
+        x, pos_emb = self.pos_enc(x, offset)
+        return x, pos_emb, x_mask[:, :, 2::2][:, :, 2::2][:, :, 2::2]
+
+
+class LegacyLinearNoSubsampling(BaseSubsampling):
+    """Linear transform the input without subsampling
+
+    Args:
+        idim (int): Input dimension.
+        odim (int): Output dimension.
+        dropout_rate (float): Dropout rate.
+
+    """
+
+    def __init__(self, idim: int, odim: int, dropout_rate: float,
+                 pos_enc_class: torch.nn.Module):
+        """Construct an linear object."""
+        super().__init__()
+        self.out = torch.nn.Sequential(
+            torch.nn.Linear(idim, odim),
+            torch.nn.LayerNorm(odim, eps=1e-5),
+            torch.nn.Dropout(dropout_rate),
+            torch.nn.ReLU(),
+        )
+        self.pos_enc = pos_enc_class
+        self.right_context = 0
+        self.subsampling_rate = 1
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        x_mask: torch.Tensor,
+        offset: Union[int, torch.Tensor] = 0
+    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        """Input x.
+
+        Args:
+            x (torch.Tensor): Input tensor (#batch, time, idim).
+            x_mask (torch.Tensor): Input mask (#batch, 1, time).
+
+        Returns:
+            torch.Tensor: linear input tensor (#batch, time', odim),
+                where time' = time .
+            torch.Tensor: linear input mask (#batch, 1, time'),
+                where time' = time .
+
+        """
+        x = self.out(x)
+        x, pos_emb = self.pos_enc(x, offset)
+        return x, pos_emb, x_mask

engines/chatterbox_official_23lang/models/s3gen/transformer/upsample_encoder.py

@@ -0,0 +1,318 @@
+# Copyright (c) 2021 Mobvoi Inc (Binbin Zhang, Di Wu)
+#               2022 Xingchen Song (sxc19@mails.tsinghua.edu.cn)
+#               2024 Alibaba Inc (Xiang Lyu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Modified from ESPnet(https://github.com/espnet/espnet)
+"""Encoder definition."""
+from typing import Tuple
+
+import torch
+from torch import nn
+from torch.nn import functional as F
+
+from .convolution import ConvolutionModule
+from .encoder_layer import ConformerEncoderLayer
+from .positionwise_feed_forward import PositionwiseFeedForward
+from ..utils.class_utils import (
+    COSYVOICE_EMB_CLASSES,
+    COSYVOICE_SUBSAMPLE_CLASSES,
+    COSYVOICE_ATTENTION_CLASSES,
+    COSYVOICE_ACTIVATION_CLASSES,
+)
+from ..utils.mask import make_pad_mask
+from ..utils.mask import add_optional_chunk_mask
+
+
+class Upsample1D(nn.Module):
+    """A 1D upsampling layer with an optional convolution.
+
+    Parameters:
+        channels (`int`):
+            number of channels in the inputs and outputs.
+        use_conv (`bool`, default `False`):
+            option to use a convolution.
+        use_conv_transpose (`bool`, default `False`):
+            option to use a convolution transpose.
+        out_channels (`int`, optional):
+            number of output channels. Defaults to `channels`.
+    """
+
+    def __init__(self, channels: int, out_channels: int, stride: int = 2):
+        super().__init__()
+        self.channels = channels
+        self.out_channels = out_channels
+        self.stride = stride
+        # In this mode, first repeat interpolate, than conv with stride=1
+        self.conv = nn.Conv1d(self.channels, self.out_channels, stride * 2 + 1, stride=1, padding=0)
+
+    def forward(self, inputs: torch.Tensor, input_lengths: torch.Tensor):
+        outputs = F.interpolate(inputs, scale_factor=float(self.stride), mode="nearest")
+        outputs = F.pad(outputs, (self.stride * 2, 0), value=0.0)
+        outputs = self.conv(outputs)
+        return outputs, input_lengths * self.stride
+
+
+class PreLookaheadLayer(nn.Module):
+    def __init__(self, channels: int, pre_lookahead_len: int = 1):
+        super().__init__()
+        self.channels = channels
+        self.pre_lookahead_len = pre_lookahead_len
+        self.conv1 = nn.Conv1d(
+            channels, channels,
+            kernel_size=pre_lookahead_len + 1,
+            stride=1, padding=0,
+        )
+        self.conv2 = nn.Conv1d(
+            channels, channels,
+            kernel_size=3, stride=1, padding=0,
+        )
+
+    def forward(self, inputs: torch.Tensor) -> torch.Tensor:
+        """
+        inputs: (batch_size, seq_len, channels)
+        """
+        outputs = inputs.transpose(1, 2).contiguous()
+        # look ahead
+        outputs = F.pad(outputs, (0, self.pre_lookahead_len), mode='constant', value=0.0)
+        outputs = F.leaky_relu(self.conv1(outputs))
+        # outputs
+        outputs = F.pad(outputs, (2, 0), mode='constant', value=0.0)
+        outputs = self.conv2(outputs)
+        outputs = outputs.transpose(1, 2).contiguous()
+
+        # residual connection
+        outputs = outputs + inputs
+        return outputs
+
+
+class UpsampleConformerEncoder(torch.nn.Module):
+
+    def __init__(
+        self,
+        input_size: int = 512,
+        output_size: int = 512,
+        attention_heads: int = 8,
+        linear_units: int = 2048,
+        num_blocks: int = 6,
+        dropout_rate: float = 0.1,
+        positional_dropout_rate: float = 0.1,
+        attention_dropout_rate: float = 0.1,
+        input_layer: str = "linear",
+        pos_enc_layer_type: str = "rel_pos_espnet",
+        normalize_before: bool = True,
+        static_chunk_size: int = 0,
+        use_dynamic_chunk: bool = False,
+        global_cmvn: torch.nn.Module = None,
+        use_dynamic_left_chunk: bool = False,
+        positionwise_conv_kernel_size: int = 1,
+        macaron_style: bool = False,
+        selfattention_layer_type: str = "rel_selfattn",
+        activation_type: str = "swish",
+        use_cnn_module: bool = False,
+        cnn_module_kernel: int = 15,
+        causal: bool = False,
+        cnn_module_norm: str = "batch_norm",
+        key_bias: bool = True,
+        gradient_checkpointing: bool = False,
+    ):
+        """
+        Args:
+            input_size (int): input dim
+            output_size (int): dimension of attention
+            attention_heads (int): the number of heads of multi head attention
+            linear_units (int): the hidden units number of position-wise feed
+                forward
+            num_blocks (int): the number of decoder blocks
+            dropout_rate (float): dropout rate
+            attention_dropout_rate (float): dropout rate in attention
+            positional_dropout_rate (float): dropout rate after adding
+                positional encoding
+            input_layer (str): input layer type.
+                optional [linear, conv2d, conv2d6, conv2d8]
+            pos_enc_layer_type (str): Encoder positional encoding layer type.
+                opitonal [abs_pos, scaled_abs_pos, rel_pos, no_pos]
+            normalize_before (bool):
+                True: use layer_norm before each sub-block of a layer.
+                False: use layer_norm after each sub-block of a layer.
+            static_chunk_size (int): chunk size for static chunk training and
+                decoding
+            use_dynamic_chunk (bool): whether use dynamic chunk size for
+                training or not, You can only use fixed chunk(chunk_size > 0)
+                or dyanmic chunk size(use_dynamic_chunk = True)
+            global_cmvn (Optional[torch.nn.Module]): Optional GlobalCMVN module
+            use_dynamic_left_chunk (bool): whether use dynamic left chunk in
+                dynamic chunk training
+            key_bias: whether use bias in attention.linear_k, False for whisper models.
+            gradient_checkpointing: rerunning a forward-pass segment for each
+                checkpointed segment during backward.
+        """
+        super().__init__()
+        self._output_size = output_size
+
+        self.global_cmvn = global_cmvn
+        self.embed = COSYVOICE_SUBSAMPLE_CLASSES[input_layer](
+            input_size,
+            output_size,
+            dropout_rate,
+            COSYVOICE_EMB_CLASSES[pos_enc_layer_type](output_size,
+                                                      positional_dropout_rate),
+        )
+
+        self.normalize_before = normalize_before
+        self.after_norm = torch.nn.LayerNorm(output_size, eps=1e-5)
+        self.static_chunk_size = static_chunk_size
+        self.use_dynamic_chunk = use_dynamic_chunk
+        self.use_dynamic_left_chunk = use_dynamic_left_chunk
+        self.gradient_checkpointing = gradient_checkpointing
+        activation = COSYVOICE_ACTIVATION_CLASSES[activation_type]()
+        # self-attention module definition
+        encoder_selfattn_layer_args = (
+            attention_heads,
+            output_size,
+            attention_dropout_rate,
+            key_bias,
+        )
+        # feed-forward module definition
+        positionwise_layer_args = (
+            output_size,
+            linear_units,
+            dropout_rate,
+            activation,
+        )
+        # convolution module definition
+        convolution_layer_args = (output_size, cnn_module_kernel, activation,
+                                  cnn_module_norm, causal)
+        self.pre_lookahead_layer = PreLookaheadLayer(channels=512, pre_lookahead_len=3)
+        self.encoders = torch.nn.ModuleList([
+            ConformerEncoderLayer(
+                output_size,
+                COSYVOICE_ATTENTION_CLASSES[selfattention_layer_type](
+                    *encoder_selfattn_layer_args),
+                PositionwiseFeedForward(*positionwise_layer_args),
+                PositionwiseFeedForward(
+                    *positionwise_layer_args) if macaron_style else None,
+                ConvolutionModule(
+                    *convolution_layer_args) if use_cnn_module else None,
+                dropout_rate,
+                normalize_before,
+            ) for _ in range(num_blocks)
+        ])
+        self.up_layer = Upsample1D(channels=512, out_channels=512, stride=2)
+        self.up_embed = COSYVOICE_SUBSAMPLE_CLASSES[input_layer](
+            input_size,
+            output_size,
+            dropout_rate,
+            COSYVOICE_EMB_CLASSES[pos_enc_layer_type](output_size,
+                                                      positional_dropout_rate),
+        )
+        self.up_encoders = torch.nn.ModuleList([
+            ConformerEncoderLayer(
+                output_size,
+                COSYVOICE_ATTENTION_CLASSES[selfattention_layer_type](
+                    *encoder_selfattn_layer_args),
+                PositionwiseFeedForward(*positionwise_layer_args),
+                PositionwiseFeedForward(
+                    *positionwise_layer_args) if macaron_style else None,
+                ConvolutionModule(
+                    *convolution_layer_args) if use_cnn_module else None,
+                dropout_rate,
+                normalize_before,
+            ) for _ in range(4)
+        ])
+
+    def output_size(self) -> int:
+        return self._output_size
+
+    def forward(
+        self,
+        xs: torch.Tensor,
+        xs_lens: torch.Tensor,
+        decoding_chunk_size: int = 0,
+        num_decoding_left_chunks: int = -1,
+    ) -> Tuple[torch.Tensor, torch.Tensor]:
+        """Embed positions in tensor.
+
+        Args:
+            xs: padded input tensor (B, T, D)
+            xs_lens: input length (B)
+            decoding_chunk_size: decoding chunk size for dynamic chunk
+                0: default for training, use random dynamic chunk.
+                <0: for decoding, use full chunk.
+                >0: for decoding, use fixed chunk size as set.
+            num_decoding_left_chunks: number of left chunks, this is for decoding,
+            the chunk size is decoding_chunk_size.
+                >=0: use num_decoding_left_chunks
+                <0: use all left chunks
+        Returns:
+            encoder output tensor xs, and subsampled masks
+            xs: padded output tensor (B, T' ~= T/subsample_rate, D)
+            masks: torch.Tensor batch padding mask after subsample
+                (B, 1, T' ~= T/subsample_rate)
+        NOTE(xcsong):
+            We pass the `__call__` method of the modules instead of `forward` to the
+            checkpointing API because `__call__` attaches all the hooks of the module.
+            https://discuss.pytorch.org/t/any-different-between-model-input-and-model-forward-input/3690/2
+        """
+        T = xs.size(1)
+        masks = ~make_pad_mask(xs_lens, T).unsqueeze(1)  # (B, 1, T)
+        if self.global_cmvn is not None:
+            xs = self.global_cmvn(xs)
+        xs, pos_emb, masks = self.embed(xs, masks)
+        mask_pad = masks  # (B, 1, T/subsample_rate)
+        chunk_masks = add_optional_chunk_mask(xs, masks,
+                                              self.use_dynamic_chunk,
+                                              self.use_dynamic_left_chunk,
+                                              decoding_chunk_size,
+                                              self.static_chunk_size,
+                                              num_decoding_left_chunks)
+        # lookahead + conformer encoder
+        xs = self.pre_lookahead_layer(xs)
+        xs = self.forward_layers(xs, chunk_masks, pos_emb, mask_pad)
+
+        # upsample + conformer encoder
+        xs = xs.transpose(1, 2).contiguous()
+        xs, xs_lens = self.up_layer(xs, xs_lens)
+        xs = xs.transpose(1, 2).contiguous()
+        T = xs.size(1)
+        masks = ~make_pad_mask(xs_lens, T).unsqueeze(1)  # (B, 1, T)
+        xs, pos_emb, masks = self.up_embed(xs, masks)
+        mask_pad = masks  # (B, 1, T/subsample_rate)
+        chunk_masks = add_optional_chunk_mask(xs, masks,
+                                              self.use_dynamic_chunk,
+                                              self.use_dynamic_left_chunk,
+                                              decoding_chunk_size,
+                                              self.static_chunk_size * self.up_layer.stride,
+                                              num_decoding_left_chunks)
+        xs = self.forward_up_layers(xs, chunk_masks, pos_emb, mask_pad)
+
+        if self.normalize_before:
+            xs = self.after_norm(xs)
+        # Here we assume the mask is not changed in encoder layers, so just
+        # return the masks before encoder layers, and the masks will be used
+        # for cross attention with decoder later
+        return xs, masks
+
+    def forward_layers(self, xs: torch.Tensor, chunk_masks: torch.Tensor,
+                       pos_emb: torch.Tensor,
+                       mask_pad: torch.Tensor) -> torch.Tensor:
+        for layer in self.encoders:
+            xs, chunk_masks, _, _ = layer(xs, chunk_masks, pos_emb, mask_pad)
+        return xs
+
+    def forward_up_layers(self, xs: torch.Tensor, chunk_masks: torch.Tensor,
+                          pos_emb: torch.Tensor,
+                          mask_pad: torch.Tensor) -> torch.Tensor:
+        for layer in self.up_encoders:
+            xs, chunk_masks, _, _ = layer(xs, chunk_masks, pos_emb, mask_pad)
+        return xs

engines/chatterbox_official_23lang/models/s3gen/utils/class_utils.py

@@ -0,0 +1,71 @@
+# Copyright [2023-11-28] <sxc19@mails.tsinghua.edu.cn, Xingchen Song>
+#            2024 Alibaba Inc (authors: Xiang Lyu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import torch
+
+from ..transformer.activation import Swish
+from ..transformer.subsampling import (
+    LinearNoSubsampling,
+    EmbedinigNoSubsampling,
+    Conv1dSubsampling2,
+    Conv2dSubsampling4,
+    Conv2dSubsampling6,
+    Conv2dSubsampling8,
+)
+from ..transformer.embedding import (
+    PositionalEncoding,
+    RelPositionalEncoding,
+    WhisperPositionalEncoding,
+    LearnablePositionalEncoding,
+    NoPositionalEncoding)
+from ..transformer.attention import (MultiHeadedAttention,
+    RelPositionMultiHeadedAttention)
+from ..transformer.embedding import EspnetRelPositionalEncoding
+from ..transformer.subsampling import LegacyLinearNoSubsampling
+
+
+COSYVOICE_ACTIVATION_CLASSES = {
+    "hardtanh": torch.nn.Hardtanh,
+    "tanh": torch.nn.Tanh,
+    "relu": torch.nn.ReLU,
+    "selu": torch.nn.SELU,
+    "swish": getattr(torch.nn, "SiLU", Swish),
+    "gelu": torch.nn.GELU,
+}
+
+COSYVOICE_SUBSAMPLE_CLASSES = {
+    "linear": LinearNoSubsampling,
+    "linear_legacy": LegacyLinearNoSubsampling,
+    "embed": EmbedinigNoSubsampling,
+    "conv1d2": Conv1dSubsampling2,
+    "conv2d": Conv2dSubsampling4,
+    "conv2d6": Conv2dSubsampling6,
+    "conv2d8": Conv2dSubsampling8,
+    'paraformer_dummy': torch.nn.Identity
+}
+
+COSYVOICE_EMB_CLASSES = {
+    "embed": PositionalEncoding,
+    "abs_pos": PositionalEncoding,
+    "rel_pos": RelPositionalEncoding,
+    "rel_pos_espnet": EspnetRelPositionalEncoding,
+    "no_pos": NoPositionalEncoding,
+    "abs_pos_whisper": WhisperPositionalEncoding,
+    "embed_learnable_pe": LearnablePositionalEncoding,
+}
+
+COSYVOICE_ATTENTION_CLASSES = {
+    "selfattn": MultiHeadedAttention,
+    "rel_selfattn": RelPositionMultiHeadedAttention,
+}

engines/chatterbox_official_23lang/models/s3gen/utils/mask.py

@@ -0,0 +1,193 @@
+# Copyright (c) 2019 Shigeki Karita
+#               2020 Mobvoi Inc (Binbin Zhang)
+#               2024 Alibaba Inc (authors: Xiang Lyu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import torch
+
+'''
+def subsequent_mask(
+        size: int,
+        device: torch.device = torch.device("cpu"),
+) -> torch.Tensor:
+    """Create mask for subsequent steps (size, size).
+
+    This mask is used only in decoder which works in an auto-regressive mode.
+    This means the current step could only do attention with its left steps.
+
+    In encoder, fully attention is used when streaming is not necessary and
+    the sequence is not long. In this  case, no attention mask is needed.
+
+    When streaming is need, chunk-based attention is used in encoder. See
+    subsequent_chunk_mask for the chunk-based attention mask.
+
+    Args:
+        size (int): size of mask
+        str device (str): "cpu" or "cuda" or torch.Tensor.device
+        dtype (torch.device): result dtype
+
+    Returns:
+        torch.Tensor: mask
+
+    Examples:
+        >>> subsequent_mask(3)
+        [[1, 0, 0],
+         [1, 1, 0],
+         [1, 1, 1]]
+    """
+    ret = torch.ones(size, size, device=device, dtype=torch.bool)
+    return torch.tril(ret)
+'''
+
+
+def subsequent_chunk_mask(
+        size: int,
+        chunk_size: int,
+        num_left_chunks: int = -1,
+        device: torch.device = torch.device("cpu"),
+) -> torch.Tensor:
+    """Create mask for subsequent steps (size, size) with chunk size,
+       this is for streaming encoder
+
+    Args:
+        size (int): size of mask
+        chunk_size (int): size of chunk
+        num_left_chunks (int): number of left chunks
+            <0: use full chunk
+            >=0: use num_left_chunks
+        device (torch.device): "cpu" or "cuda" or torch.Tensor.device
+
+    Returns:
+        torch.Tensor: mask
+
+    Examples:
+        >>> subsequent_chunk_mask(4, 2)
+        [[1, 1, 0, 0],
+         [1, 1, 0, 0],
+         [1, 1, 1, 1],
+         [1, 1, 1, 1]]
+    """
+    # NOTE this modified implementation meets onnx export requirements, but it doesn't support num_left_chunks
+    # actually this is not needed after we have inference cache implemented, will remove it later
+    pos_idx = torch.arange(size, device=device)
+    block_value = (torch.div(pos_idx, chunk_size, rounding_mode='trunc') + 1) * chunk_size
+    ret = pos_idx.unsqueeze(0) < block_value.unsqueeze(1)
+    return ret
+
+
+def add_optional_chunk_mask(xs: torch.Tensor,
+                            masks: torch.Tensor,
+                            use_dynamic_chunk: bool,
+                            use_dynamic_left_chunk: bool,
+                            decoding_chunk_size: int,
+                            static_chunk_size: int,
+                            num_decoding_left_chunks: int,
+                            enable_full_context: bool = True):
+    """ Apply optional mask for encoder.
+
+    Args:
+        xs (torch.Tensor): padded input, (B, L, D), L for max length
+        mask (torch.Tensor): mask for xs, (B, 1, L)
+        use_dynamic_chunk (bool): whether to use dynamic chunk or not
+        use_dynamic_left_chunk (bool): whether to use dynamic left chunk for
+            training.
+        decoding_chunk_size (int): decoding chunk size for dynamic chunk, it's
+            0: default for training, use random dynamic chunk.
+            <0: for decoding, use full chunk.
+            >0: for decoding, use fixed chunk size as set.
+        static_chunk_size (int): chunk size for static chunk training/decoding
+            if it's greater than 0, if use_dynamic_chunk is true,
+            this parameter will be ignored
+        num_decoding_left_chunks: number of left chunks, this is for decoding,
+            the chunk size is decoding_chunk_size.
+            >=0: use num_decoding_left_chunks
+            <0: use all left chunks
+        enable_full_context (bool):
+            True: chunk size is either [1, 25] or full context(max_len)
+            False: chunk size ~ U[1, 25]
+
+    Returns:
+        torch.Tensor: chunk mask of the input xs.
+    """
+    # Whether to use chunk mask or not
+    if use_dynamic_chunk:
+        max_len = xs.size(1)
+        if decoding_chunk_size < 0:
+            chunk_size = max_len
+            num_left_chunks = -1
+        elif decoding_chunk_size > 0:
+            chunk_size = decoding_chunk_size
+            num_left_chunks = num_decoding_left_chunks
+        else:
+            # chunk size is either [1, 25] or full context(max_len).
+            # Since we use 4 times subsampling and allow up to 1s(100 frames)
+            # delay, the maximum frame is 100 / 4 = 25.
+            chunk_size = torch.randint(1, max_len, (1, )).item()
+            num_left_chunks = -1
+            if chunk_size > max_len // 2 and enable_full_context:
+                chunk_size = max_len
+            else:
+                chunk_size = chunk_size % 25 + 1
+                if use_dynamic_left_chunk:
+                    max_left_chunks = (max_len - 1) // chunk_size
+                    num_left_chunks = torch.randint(0, max_left_chunks,
+                                                    (1, )).item()
+        chunk_masks = subsequent_chunk_mask(xs.size(1), chunk_size,
+                                            num_left_chunks,
+                                            xs.device)  # (L, L)
+        chunk_masks = chunk_masks.unsqueeze(0)  # (1, L, L)
+        chunk_masks = masks & chunk_masks  # (B, L, L)
+    elif static_chunk_size > 0:
+        num_left_chunks = num_decoding_left_chunks
+        chunk_masks = subsequent_chunk_mask(xs.size(1), static_chunk_size,
+                                            num_left_chunks,
+                                            xs.device)  # (L, L)
+        chunk_masks = chunk_masks.unsqueeze(0)  # (1, L, L)
+        chunk_masks = masks & chunk_masks  # (B, L, L)
+    else:
+        chunk_masks = masks
+    assert chunk_masks.dtype == torch.bool
+    if (chunk_masks.sum(dim=-1) == 0).sum().item() != 0:
+        logging.warning('get chunk_masks all false at some timestep, force set to true, make sure they are masked in futuer computation!')
+        chunk_masks[chunk_masks.sum(dim=-1)==0] = True
+    return chunk_masks
+
+
+def make_pad_mask(lengths: torch.Tensor, max_len: int = 0) -> torch.Tensor:
+    """Make mask tensor containing indices of padded part.
+
+    See description of make_non_pad_mask.
+
+    Args:
+        lengths (torch.Tensor): Batch of lengths (B,).
+    Returns:
+        torch.Tensor: Mask tensor containing indices of padded part.
+
+    Examples:
+        >>> lengths = [5, 3, 2]
+        >>> make_pad_mask(lengths)
+        masks = [[0, 0, 0, 0 ,0],
+                 [0, 0, 0, 1, 1],
+                 [0, 0, 1, 1, 1]]
+    """
+    batch_size = lengths.size(0)
+    max_len = max_len if max_len > 0 else lengths.max().item()
+    seq_range = torch.arange(0,
+                             max_len,
+                             dtype=torch.int64,
+                             device=lengths.device)
+    seq_range_expand = seq_range.unsqueeze(0).expand(batch_size, max_len)
+    seq_length_expand = lengths.unsqueeze(-1)
+    mask = seq_range_expand >= seq_length_expand
+    return mask

engines/chatterbox_official_23lang/models/s3gen/utils/mel.py

@@ -0,0 +1,85 @@
+"""mel-spectrogram extraction in Matcha-TTS"""
+import logging
+from librosa.filters import mel as librosa_mel_fn
+import torch
+import numpy as np
+
+logger = logging.getLogger(__name__)
+
+
+# NOTE: they decalred these global vars
+mel_basis = {}
+hann_window = {}
+
+
+def dynamic_range_compression_torch(x, C=1, clip_val=1e-5):
+    return torch.log(torch.clamp(x, min=clip_val) * C)
+
+
+def spectral_normalize_torch(magnitudes):
+    output = dynamic_range_compression_torch(magnitudes)
+    return output
+
+"""
+feat_extractor: !name:matcha.utils.audio.mel_spectrogram
+    n_fft: 1920
+    num_mels: 80
+    sampling_rate: 24000
+    hop_size: 480
+    win_size: 1920
+    fmin: 0
+    fmax: 8000
+    center: False
+
+"""
+
+def mel_spectrogram(y, n_fft=1920, num_mels=80, sampling_rate=24000, hop_size=480, win_size=1920,
+                    fmin=0, fmax=8000, center=False):
+    """Copied from https://github.com/shivammehta25/Matcha-TTS/blob/main/matcha/utils/audio.py
+    Set default values according to Cosyvoice's config.
+    """
+
+    if isinstance(y, np.ndarray):
+        y = torch.tensor(y).float()
+
+    if len(y.shape) == 1:
+        y = y[None, ]
+
+    # Debug: Check for audio clipping (values outside [-1.0, 1.0] range)
+    min_val = torch.min(y)
+    max_val = torch.max(y)
+    if min_val < -1.0 or max_val > 1.0:
+        logger.warning(f"Audio values outside normalized range: min={min_val.item():.4f}, max={max_val.item():.4f}")
+
+    global mel_basis, hann_window  # pylint: disable=global-statement,global-variable-not-assigned
+    if f"{str(fmax)}_{str(y.device)}" not in mel_basis:
+        mel = librosa_mel_fn(sr=sampling_rate, n_fft=n_fft, n_mels=num_mels, fmin=fmin, fmax=fmax)
+        mel_basis[str(fmax) + "_" + str(y.device)] = torch.from_numpy(mel).float().to(y.device)
+        hann_window[str(y.device)] = torch.hann_window(win_size).to(y.device)
+
+    y = torch.nn.functional.pad(
+        y.unsqueeze(1), (int((n_fft - hop_size) / 2), int((n_fft - hop_size) / 2)), mode="reflect"
+    )
+    y = y.squeeze(1)
+
+    spec = torch.view_as_real(
+        torch.stft(
+            y,
+            n_fft,
+            hop_length=hop_size,
+            win_length=win_size,
+            window=hann_window[str(y.device)],
+            center=center,
+            pad_mode="reflect",
+            normalized=False,
+            onesided=True,
+            return_complex=True,
+        )
+    )
+
+    spec = torch.sqrt(spec.pow(2).sum(-1) + (1e-9))
+
+    spec = torch.matmul(mel_basis[str(fmax) + "_" + str(y.device)], spec)
+    spec = spectral_normalize_torch(spec)
+
+    return spec

engines/chatterbox_official_23lang/models/s3gen/xvector.py

@@ -0,0 +1,428 @@
+#!/usr/bin/env python3
+# -*- encoding: utf-8 -*-
+# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.
+#  MIT License  (https://opensource.org/licenses/MIT)
+# Modified from 3D-Speaker (https://github.com/alibaba-damo-academy/3D-Speaker)
+
+
+from collections import OrderedDict
+import torch
+import torch.nn.functional as F
+import torch.utils.checkpoint as cp
+import torchaudio.compliance.kaldi as Kaldi
+
+
+def pad_list(xs, pad_value):
+    """Perform padding for the list of tensors.
+
+    Args:
+        xs (List): List of Tensors [(T_1, `*`), (T_2, `*`), ..., (T_B, `*`)].
+        pad_value (float): Value for padding.
+
+    Returns:
+        Tensor: Padded tensor (B, Tmax, `*`).
+
+    Examples:
+        >>> x = [torch.ones(4), torch.ones(2), torch.ones(1)]
+        >>> x
+        [tensor([1., 1., 1., 1.]), tensor([1., 1.]), tensor([1.])]
+        >>> pad_list(x, 0)
+        tensor([[1., 1., 1., 1.],
+                [1., 1., 0., 0.],
+                [1., 0., 0., 0.]])
+
+    """
+    n_batch = len(xs)
+    max_len = max(x.size(0) for x in xs)
+    pad = xs[0].new(n_batch, max_len, *xs[0].size()[1:]).fill_(pad_value)
+
+    for i in range(n_batch):
+        pad[i, : xs[i].size(0)] = xs[i]
+
+    return pad
+
+
+def extract_feature(audio):
+    features = []
+    feature_times = []
+    feature_lengths = []
+    for au in audio:
+        feature = Kaldi.fbank(au.unsqueeze(0), num_mel_bins=80)
+        feature = feature - feature.mean(dim=0, keepdim=True)
+        features.append(feature)
+        feature_times.append(au.shape[0])
+        feature_lengths.append(feature.shape[0])
+    # padding for batch inference
+    features_padded = pad_list(features, pad_value=0)
+    # features = torch.cat(features)
+    return features_padded, feature_lengths, feature_times
+
+
+class BasicResBlock(torch.nn.Module):
+    expansion = 1
+
+    def __init__(self, in_planes, planes, stride=1):
+        super(BasicResBlock, self).__init__()
+        self.conv1 = torch.nn.Conv2d(
+            in_planes, planes, kernel_size=3, stride=(stride, 1), padding=1, bias=False
+        )
+        self.bn1 = torch.nn.BatchNorm2d(planes)
+        self.conv2 = torch.nn.Conv2d(planes, planes, kernel_size=3, stride=1, padding=1, bias=False)
+        self.bn2 = torch.nn.BatchNorm2d(planes)
+
+        self.shortcut = torch.nn.Sequential()
+        if stride != 1 or in_planes != self.expansion * planes:
+            self.shortcut = torch.nn.Sequential(
+                torch.nn.Conv2d(
+                    in_planes,
+                    self.expansion * planes,
+                    kernel_size=1,
+                    stride=(stride, 1),
+                    bias=False,
+                ),
+                torch.nn.BatchNorm2d(self.expansion * planes),
+            )
+
+    def forward(self, x):
+        out = F.relu(self.bn1(self.conv1(x)))
+        out = self.bn2(self.conv2(out))
+        out += self.shortcut(x)
+        out = F.relu(out)
+        return out
+
+
+class FCM(torch.nn.Module):
+    def __init__(self, block=BasicResBlock, num_blocks=[2, 2], m_channels=32, feat_dim=80):
+        super(FCM, self).__init__()
+        self.in_planes = m_channels
+        self.conv1 = torch.nn.Conv2d(1, m_channels, kernel_size=3, stride=1, padding=1, bias=False)
+        self.bn1 = torch.nn.BatchNorm2d(m_channels)
+
+        self.layer1 = self._make_layer(block, m_channels, num_blocks[0], stride=2)
+        self.layer2 = self._make_layer(block, m_channels, num_blocks[0], stride=2)
+
+        self.conv2 = torch.nn.Conv2d(
+            m_channels, m_channels, kernel_size=3, stride=(2, 1), padding=1, bias=False
+        )
+        self.bn2 = torch.nn.BatchNorm2d(m_channels)
+        self.out_channels = m_channels * (feat_dim // 8)
+
+    def _make_layer(self, block, planes, num_blocks, stride):
+        strides = [stride] + [1] * (num_blocks - 1)
+        layers = []
+        for stride in strides:
+            layers.append(block(self.in_planes, planes, stride))
+            self.in_planes = planes * block.expansion
+        return torch.nn.Sequential(*layers)
+
+    def forward(self, x):
+        x = x.unsqueeze(1)
+        out = F.relu(self.bn1(self.conv1(x)))
+        out = self.layer1(out)
+        out = self.layer2(out)
+        out = F.relu(self.bn2(self.conv2(out)))
+
+        shape = out.shape
+        out = out.reshape(shape[0], shape[1] * shape[2], shape[3])
+        return out
+
+
+def get_nonlinear(config_str, channels):
+    nonlinear = torch.nn.Sequential()
+    for name in config_str.split("-"):
+        if name == "relu":
+            nonlinear.add_module("relu", torch.nn.ReLU(inplace=True))
+        elif name == "prelu":
+            nonlinear.add_module("prelu", torch.nn.PReLU(channels))
+        elif name == "batchnorm":
+            nonlinear.add_module("batchnorm", torch.nn.BatchNorm1d(channels))
+        elif name == "batchnorm_":
+            nonlinear.add_module("batchnorm", torch.nn.BatchNorm1d(channels, affine=False))
+        else:
+            raise ValueError("Unexpected module ({}).".format(name))
+    return nonlinear
+
+
+def statistics_pooling(x, dim=-1, keepdim=False, unbiased=True, eps=1e-2):
+    mean = x.mean(dim=dim)
+    std = x.std(dim=dim, unbiased=unbiased)
+    stats = torch.cat([mean, std], dim=-1)
+    if keepdim:
+        stats = stats.unsqueeze(dim=dim)
+    return stats
+
+
+class StatsPool(torch.nn.Module):
+    def forward(self, x):
+        return statistics_pooling(x)
+
+
+class TDNNLayer(torch.nn.Module):
+    def __init__(
+        self,
+        in_channels,
+        out_channels,
+        kernel_size,
+        stride=1,
+        padding=0,
+        dilation=1,
+        bias=False,
+        config_str="batchnorm-relu",
+    ):
+        super(TDNNLayer, self).__init__()
+        if padding < 0:
+            assert (
+                kernel_size % 2 == 1
+            ), "Expect equal paddings, but got even kernel size ({})".format(kernel_size)
+            padding = (kernel_size - 1) // 2 * dilation
+        self.linear = torch.nn.Conv1d(
+            in_channels,
+            out_channels,
+            kernel_size,
+            stride=stride,
+            padding=padding,
+            dilation=dilation,
+            bias=bias,
+        )
+        self.nonlinear = get_nonlinear(config_str, out_channels)
+
+    def forward(self, x):
+        x = self.linear(x)
+        x = self.nonlinear(x)
+        return x
+
+
+class CAMLayer(torch.nn.Module):
+    def __init__(
+        self, bn_channels, out_channels, kernel_size, stride, padding, dilation, bias, reduction=2
+    ):
+        super(CAMLayer, self).__init__()
+        self.linear_local = torch.nn.Conv1d(
+            bn_channels,
+            out_channels,
+            kernel_size,
+            stride=stride,
+            padding=padding,
+            dilation=dilation,
+            bias=bias,
+        )
+        self.linear1 = torch.nn.Conv1d(bn_channels, bn_channels // reduction, 1)
+        self.relu = torch.nn.ReLU(inplace=True)
+        self.linear2 = torch.nn.Conv1d(bn_channels // reduction, out_channels, 1)
+        self.sigmoid = torch.nn.Sigmoid()
+
+    def forward(self, x):
+        y = self.linear_local(x)
+        context = x.mean(-1, keepdim=True) + self.seg_pooling(x)
+        context = self.relu(self.linear1(context))
+        m = self.sigmoid(self.linear2(context))
+        return y * m
+
+    def seg_pooling(self, x, seg_len=100, stype="avg"):
+        if stype == "avg":
+            seg = F.avg_pool1d(x, kernel_size=seg_len, stride=seg_len, ceil_mode=True)
+        elif stype == "max":
+            seg = F.max_pool1d(x, kernel_size=seg_len, stride=seg_len, ceil_mode=True)
+        else:
+            raise ValueError("Wrong segment pooling type.")
+        shape = seg.shape
+        seg = seg.unsqueeze(-1).expand(*shape, seg_len).reshape(*shape[:-1], -1)
+        seg = seg[..., : x.shape[-1]]
+        return seg
+
+
+class CAMDenseTDNNLayer(torch.nn.Module):
+    def __init__(
+        self,
+        in_channels,
+        out_channels,
+        bn_channels,
+        kernel_size,
+        stride=1,
+        dilation=1,
+        bias=False,
+        config_str="batchnorm-relu",
+        memory_efficient=False,
+    ):
+        super(CAMDenseTDNNLayer, self).__init__()
+        assert kernel_size % 2 == 1, "Expect equal paddings, but got even kernel size ({})".format(
+            kernel_size
+        )
+        padding = (kernel_size - 1) // 2 * dilation
+        self.memory_efficient = memory_efficient
+        self.nonlinear1 = get_nonlinear(config_str, in_channels)
+        self.linear1 = torch.nn.Conv1d(in_channels, bn_channels, 1, bias=False)
+        self.nonlinear2 = get_nonlinear(config_str, bn_channels)
+        self.cam_layer = CAMLayer(
+            bn_channels,
+            out_channels,
+            kernel_size,
+            stride=stride,
+            padding=padding,
+            dilation=dilation,
+            bias=bias,
+        )
+
+    def bn_function(self, x):
+        return self.linear1(self.nonlinear1(x))
+
+    def forward(self, x):
+        if self.training and self.memory_efficient:
+            x = cp.checkpoint(self.bn_function, x)
+        else:
+            x = self.bn_function(x)
+        x = self.cam_layer(self.nonlinear2(x))
+        return x
+
+
+class CAMDenseTDNNBlock(torch.nn.ModuleList):
+    def __init__(
+        self,
+        num_layers,
+        in_channels,
+        out_channels,
+        bn_channels,
+        kernel_size,
+        stride=1,
+        dilation=1,
+        bias=False,
+        config_str="batchnorm-relu",
+        memory_efficient=False,
+    ):
+        super(CAMDenseTDNNBlock, self).__init__()
+        for i in range(num_layers):
+            layer = CAMDenseTDNNLayer(
+                in_channels=in_channels + i * out_channels,
+                out_channels=out_channels,
+                bn_channels=bn_channels,
+                kernel_size=kernel_size,
+                stride=stride,
+                dilation=dilation,
+                bias=bias,
+                config_str=config_str,
+                memory_efficient=memory_efficient,
+            )
+            self.add_module("tdnnd%d" % (i + 1), layer)
+
+    def forward(self, x):
+        for layer in self:
+            x = torch.cat([x, layer(x)], dim=1)
+        return x
+
+
+class TransitLayer(torch.nn.Module):
+    def __init__(self, in_channels, out_channels, bias=True, config_str="batchnorm-relu"):
+        super(TransitLayer, self).__init__()
+        self.nonlinear = get_nonlinear(config_str, in_channels)
+        self.linear = torch.nn.Conv1d(in_channels, out_channels, 1, bias=bias)
+
+    def forward(self, x):
+        x = self.nonlinear(x)
+        x = self.linear(x)
+        return x
+
+
+class DenseLayer(torch.nn.Module):
+    def __init__(self, in_channels, out_channels, bias=False, config_str="batchnorm-relu"):
+        super(DenseLayer, self).__init__()
+        self.linear = torch.nn.Conv1d(in_channels, out_channels, 1, bias=bias)
+        self.nonlinear = get_nonlinear(config_str, out_channels)
+
+    def forward(self, x):
+        if len(x.shape) == 2:
+            x = self.linear(x.unsqueeze(dim=-1)).squeeze(dim=-1)
+        else:
+            x = self.linear(x)
+        x = self.nonlinear(x)
+        return x
+
+# @tables.register("model_classes", "CAMPPlus")
+class CAMPPlus(torch.nn.Module):
+    def __init__(
+        self,
+        feat_dim=80,
+        embedding_size=192,
+        growth_rate=32,
+        bn_size=4,
+        init_channels=128,
+        config_str="batchnorm-relu",
+        memory_efficient=True,
+        output_level="segment",
+        **kwargs,
+    ):
+        super().__init__()
+
+        self.head = FCM(feat_dim=feat_dim)
+        channels = self.head.out_channels
+        self.output_level = output_level
+
+        self.xvector = torch.nn.Sequential(
+            OrderedDict(
+                [
+                    (
+                        "tdnn",
+                        TDNNLayer(
+                            channels,
+                            init_channels,
+                            5,
+                            stride=2,
+                            dilation=1,
+                            padding=-1,
+                            config_str=config_str,
+                        ),
+                    ),
+                ]
+            )
+        )
+        channels = init_channels
+        for i, (num_layers, kernel_size, dilation) in enumerate(
+            zip((12, 24, 16), (3, 3, 3), (1, 2, 2))
+        ):
+            block = CAMDenseTDNNBlock(
+                num_layers=num_layers,
+                in_channels=channels,
+                out_channels=growth_rate,
+                bn_channels=bn_size * growth_rate,
+                kernel_size=kernel_size,
+                dilation=dilation,
+                config_str=config_str,
+                memory_efficient=memory_efficient,
+            )
+            self.xvector.add_module("block%d" % (i + 1), block)
+            channels = channels + num_layers * growth_rate
+            self.xvector.add_module(
+                "transit%d" % (i + 1),
+                TransitLayer(channels, channels // 2, bias=False, config_str=config_str),
+            )
+            channels //= 2
+
+        self.xvector.add_module("out_nonlinear", get_nonlinear(config_str, channels))
+
+        if self.output_level == "segment":
+            self.xvector.add_module("stats", StatsPool())
+            self.xvector.add_module(
+                "dense", DenseLayer(channels * 2, embedding_size, config_str="batchnorm_")
+            )
+        else:
+            assert (
+                self.output_level == "frame"
+            ), "`output_level` should be set to 'segment' or 'frame'. "
+
+        for m in self.modules():
+            if isinstance(m, (torch.nn.Conv1d, torch.nn.Linear)):
+                torch.nn.init.kaiming_normal_(m.weight.data)
+                if m.bias is not None:
+                    torch.nn.init.zeros_(m.bias)
+
+    def forward(self, x):
+        x = x.permute(0, 2, 1)  # (B,T,F) => (B,F,T)
+        x = self.head(x)
+        x = self.xvector(x)
+        if self.output_level == "frame":
+            x = x.transpose(1, 2)
+        return x
+
+    def inference(self, audio_list):
+        speech, speech_lengths, speech_times = extract_feature(audio_list)
+        results = self.forward(speech.to(torch.float32))
+        return results

engines/chatterbox_official_23lang/models/s3tokenizer/__init__.py

@@ -0,0 +1,30 @@
+from .s3tokenizer import (
+    S3_SR,
+    S3_HOP,
+    S3_TOKEN_HOP,
+    S3_TOKEN_RATE,
+    SPEECH_VOCAB_SIZE,
+    S3Tokenizer,
+)
+
+
+SOS = SPEECH_VOCAB_SIZE
+EOS = SPEECH_VOCAB_SIZE + 1
+
+
+
+def drop_invalid_tokens(x):
+    """Drop SoS and EoS"""
+    assert len(x.shape) == 1 or (len(x.shape) == 2 and x.shape[0] == 1), "only batch size of one allowed for now"
+    if SOS in x:
+        s = (x == SOS).nonzero(as_tuple=True)[0].squeeze(0) + 1
+    else:
+        s = 0
+
+    if EOS in x:
+        e = (x == EOS).nonzero(as_tuple=True)[0].squeeze(0)
+    else:
+        e = None
+
+    x = x[s: e]
+    return x