diodiogod-TTS-Audio-Suite/PROJECT_INDEX.md

# TTS Audio Suite - Project Index

*Comprehensive file index for the universal multi-engine TTS extension for ComfyUI*

## Architecture Overview

This extension features a **unified modular architecture** supporting multiple TTS engines:
- **Unified Node Interface**: Single set of nodes (TTS Text, TTS SRT, Voice Changer) that work with any engine via clean delegation
- **ComfyUI Model Management Integration**: All TTS models now integrate with ComfyUI's native model management system, enabling "Clear VRAM" functionality and automatic memory management
- **Universal Model Loading System**: Standardized model loading interface across all engines (ChatterBox, F5-TTS, Higgs Audio, VibeVoice, RVC, Audio Separation) with fallback support
- **Engine Processors**: Internal processing engines for each TTS system (ChatterBox, F5-TTS, Higgs Audio, VibeVoice) handling engine-specific orchestration
- **Engine Adapters**: Modular adapters for ChatterBox, F5-TTS, Higgs Audio 2, VibeVoice, and RVC voice conversion
- **Centralized Download System**: Unified downloader eliminates HuggingFace cache duplication with direct downloads to organized TTS/ folder structure, with full extra_model_paths.yaml support for shared model directories
- **Thread-Safe Architecture**: Stateless ChatterBox wrapper eliminates shared state corruption (Note: parallel processing is slower than sequential)
- **Universal Streaming Infrastructure**: Unified streaming system with configurable workers (batch_size parameter) - sequential mode (batch_size=0) recommended for optimal performance
- **Multilingual Support**: German and Norwegian models for ChatterBox, plus ChatterBox Official 23-Lang supporting 23 languages including Arabic, Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew, Hindi, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese, Russian, Spanish, Swedish, Swahili, and Turkish
- **Smart Language Grouping**: SRT processing by language groups to minimize model switching
- **Character Voice Management**: Centralized character voice system with flexible input types
- **Comprehensive Audio Processing**: Interactive waveform analyzer, vocal separation, and audio mixing
- **Professional Voice Conversion**: RVC implementation with UVR5 vocal separation and advanced pitch control

## Documentation Files

**README.md** - Main project documentation with installation guide, features overview, and usage instructions for the TTS Audio Suite

**CLAUDE.md** - Development guidelines and project context for Claude Code when working with this codebase, including critical thinking requirements and response style guidelines

**CHANGELOG.md** - Complete version history with detailed feature additions, bug fixes, and architectural improvements across all versions

### User Documentation

**docs/CHARACTER_SWITCHING_GUIDE.md** - Guide for multi-character TTS using [CharacterName] tags, voice organization, and character voice management system

**docs/CHATTERBOX_V2_SPECIAL_TOKENS.md** - Complete guide for ChatterBox Official 23-Lang v2 special emotion and sound tokens (<giggle>, <whisper>, <sigh>, etc.) with 30+ expressive tokens for enhanced TTS generation

**docs/🌊_Audio_Wave_Analyzer-Complete_User_Guide.md** - Complete user guide for Audio Wave Analyzer with interactive waveform visualization and timing extraction

**docs/VERSION_3.1_RELEASE_GUIDE.md** - Release documentation for character switching system and overlapping subtitles support

### Developer Documentation

**docs/Dev reports/F5TTS_INTEGRATION_SPECIFICATION.md** - Technical specification for F5-TTS integration architecture

**docs/Dev reports/F5TTS_STREAMING_INTEGRATION_PLAN.md** - Universal streaming architecture plan to eliminate spaghetti code and enable F5-TTS streaming

**docs/Dev reports/STREAMING_ARCHITECTURE.md** - Streaming parallel processing architecture documentation with feasibility analysis for multiple engines

**docs/Dev reports/STATELESS_WRAPPER_IMPLEMENTATION_PLAN.md** - Stateless ChatterBox wrapper implementation solving shared model state corruption in parallel processing

**docs/Dev reports/POST_V4.2.3_DEVELOPMENT_REVIEW.md** - Comprehensive analysis of 45 commits implementing universal streaming architecture and performance insights

**docs/Dev reports/PAUSE_TAGS_IMPLEMENTATION_REPORT.md** - Implementation details for pause tag system with timing control syntax

**docs/Dev reports/REFACTORING_PLAN.md** - Architectural refactoring plan and migration strategy documentation

## Core Architecture Files

**__init__.py** - Main ComfyUI extension entry point defining WEB_DIRECTORY and delegating to nodes.py for node registration

**nodes.py** - Central module loader with sophisticated node discovery, conditional imports, and startup diagnostics for all TTS/VC/Audio features

## Unified Engine Architecture

### Engine Implementations

**engines/chatterbox/** - Complete ChatterBox TTS and VC engine implementation
- **tts.py** - ChatterBox TTS engine with multilingual support, perth watermarking, and ComfyUI model management integration
- **vc.py** - ChatterBox Voice Conversion with multilingual model loading
- **language_models.py** - Language model registry for English, German, and Norwegian models
- **audio_timing.py** - Audio timing utilities and time-stretching functionality
- **streaming_model_manager.py** - Pre-loading and management of multiple language models for streaming workers
- **batch_processor.py** - Batch processor for character group processing with true parallel inference
- **character_grouper.py** - Character grouping utilities for batch processing optimization
- **stateless_wrapper.py** - Thread-safe stateless wrapper eliminating shared state corruption in parallel generation
- **models/** - Complete ChatterBox model architecture (S3Gen, T3, tokenizers, voice encoder)

**engines/chatterbox_official_23lang/** - ChatterBox Official 23-Lang multilingual engine implementation
- **tts.py** - Main ChatterBox Official 23-Lang TTS engine with 23-language support
- **vc.py** - Voice conversion engine supporting all 23 languages with refinement passes
- **language_models.py** - Language model configuration and multilingual support definitions
- **stateless_wrapper.py** - Thread-safe wrapper for multilingual parallel processing
- **batch_processor.py**, **adaptive_processor.py**, **gpu_aware_processor.py** - Advanced processing systems
- **models/** - Complete multilingual model architecture (S3Gen, T3, tokenizers, voice encoder) optimized for 23-language generation

**engines/f5tts/** - F5-TTS engine implementation and editing capabilities
- **f5tts.py** - Main F5-TTS wrapper with ComfyUI integration
- **f5tts_edit_engine.py** - Speech editing engine for targeted word replacement
- **audio_compositing.py** - Audio compositing with crossfade and segment processing

**engines/higgs_audio/** - Higgs Audio 2 engine implementation with voice cloning
- **higgs_audio.py** - Main Higgs Audio 2 wrapper with voice cloning and chunking support
- **higgs_audio_downloader.py** - Model auto-download system for generation and tokenizer models
- **boson_multimodal/** - Complete boson_multimodal implementation from HiggsAudio team

**engines/vibevoice_engine/** - VibeVoice engine implementation with multi-speaker and long-form capabilities
- **vibevoice_engine.py** - Main VibeVoice wrapper with 90-minute generation and multi-speaker support
- **vibevoice_downloader.py** - Model auto-download system for Microsoft VibeVoice models (1.5B and 7B)

**engines/rvc/** - RVC (Real-time Voice Conversion) engine implementation
- **__init__.py** - RVC engine initialization and ComfyUI integration
- **hubert_downloader.py** - HuBERT model auto-download from Hugging Face with TTS/ folder organization
- **impl/** - Complete RVC implementation with vocal separation and voice conversion
  - **rvc_audio.py** - Audio processing utilities for RVC pipeline
  - **rvc_downloader.py** - Model downloading system for RVC, UVR, and Karafan models
  - **rvc_utils.py** - Utility functions for device detection and file management
  - **uvr5_cli.py** - UVR5 vocal separation command-line interface
  - **lib/** - Core RVC libraries and model implementations
    - **separators.py** - Audio separation algorithms (MDX, VR, UVR5)
    - **model_utils.py** - Model loading and hash utilities
    - **audio.py** - Audio I/O and format conversion utilities
    - **utils.py** - General utilities and merge functions
    - **karafan/** - Karafan separation model implementation
    - **infer_pack/** - RVC inference and model loading components
    - **uvr5_pack/** - UVR5 vocal separation algorithms

### Engine Adapters

**engines/adapters/chatterbox_adapter.py** - ChatterBox engine adapter providing standardized interface for unified nodes

**engines/adapters/f5tts_adapter.py** - F5-TTS engine adapter with parameter mapping and cache integration

**engines/adapters/higgs_audio_adapter.py** - Higgs Audio 2 engine adapter with voice cloning and parameter validation

**engines/adapters/vibevoice_adapter.py** - VibeVoice engine adapter with multi-speaker format conversion and parameter mapping

**engines/adapters/chatterbox_streaming_adapter.py** - ChatterBox streaming adapter bridging existing implementation to universal streaming system

**engines/adapters/f5tts_streaming_adapter.py** - F5-TTS streaming adapter enabling parallel processing with language model switching

## Node Implementations

### Engine Configuration Nodes

**nodes/engines/chatterbox_engine_node.py** - ChatterBox engine configuration node for language, device, and generation parameters

**nodes/engines/f5tts_engine_node.py** - F5-TTS engine configuration node with model selection and generation settings

**nodes/engines/higgs_audio_engine_node.py** - Higgs Audio 2 engine configuration node with voice cloning parameters and generation settings

**nodes/engines/vibevoice_engine_node.py** - VibeVoice engine configuration node with multi-speaker modes and long-form generation settings

**nodes/engines/rvc_engine_node.py** - RVC voice conversion node with pitch control and quality settings

**nodes/engines/index_tts_engine_node.py** - IndexTTS-2 engine configuration node with character voice selection and generation parameters

**nodes/engines/index_tts_emotion_options_node.py** - 🌈 IndexTTS-2 Emotion Vectors node with interactive radar chart for 8-emotion control

### Unified Interface Nodes

**nodes/unified/tts_text_node.py** - Universal TTS text generation node working with any configured engine

**nodes/unified/tts_srt_node.py** - Universal SRT subtitle processing node - clean delegation layer that routes to engine-specific processors

**nodes/unified/voice_changer_node.py** - Universal voice conversion node with multilingual model support and flexible audio inputs

### Shared Components

**nodes/shared/character_voices_node.py** - Character voice management system providing NARRATOR_VOICE outputs for any TTS node

### Text Processing Nodes

**nodes/text/phoneme_text_normalizer_node.py** - 📝 Phoneme Text Normalizer with multilingual text preprocessing for improved TTS pronunciation. Features 4 processing methods: Pass-through, Unicode Decomposition (ą→a̧), IPA Phonemization (espeak backend), and Character Mapping (ASCII fallback). Supports auto-language detection and cross-platform phonemizer support.

### Engine-Specific Nodes

*Note: These are internal processors/engines used by the Unified nodes, not direct ComfyUI interface nodes. They handle engine-specific orchestration while the Unified nodes provide the user interface.*

**nodes/chatterbox/** - ChatterBox engine implementation nodes (called by Unified nodes)
- **chatterbox_tts_node.py** - ChatterBox TTS engine node with streaming batch processing and character switching
- **chatterbox_srt_node.py** - ChatterBox SRT engine node with streaming parallel processing and timing modes  
- **chatterbox_vc_node.py** - ChatterBox voice conversion engine with iterative refinement

**nodes/chatterbox_official_23lang/** - ChatterBox Official 23-Lang multilingual processors
- **chatterbox_official_23lang_processor.py** - Main TTS processor supporting all 23 languages with character switching
- **chatterbox_official_23lang_srt_processor.py** - SRT subtitle processor with multilingual timing and audio assembly
- **chatterbox_official_23lang_vc_processor.py** - Voice conversion processor supporting all 23 languages with refinement passes

**nodes/f5tts/** - F5-TTS specific nodes
- **f5tts_node.py** - Direct F5-TTS generation node
- **f5tts_srt_node.py** - F5-TTS SRT processing with language grouping
- **f5tts_edit_node.py** - F5-TTS speech editor for word replacement
- **f5tts_edit_options_node.py** - Advanced F5-TTS editing configuration

**nodes/higgs_audio/** - Higgs Audio 2 internal processors
- **higgs_audio_srt_processor.py** - Higgs Audio SRT orchestrator with multi-speaker support, character switching, and timing modes (internal processor used by Unified SRT node)

**nodes/vibevoice/** - VibeVoice internal processors  
- **vibevoice_processor.py** - VibeVoice TTS orchestrator with multi-speaker support and long-form generation handling (internal processor used by Unified TTS node)

### Audio Processing System

**nodes/audio/analyzer_node.py** - Interactive Audio Wave Analyzer with web interface and timing extraction

**nodes/audio/analyzer_options_node.py** - Configuration options for audio analysis (silence detection, peak analysis)

**nodes/audio/recorder_node.py** - Voice recording node with microphone input and silence detection

**nodes/audio/vocal_removal_node.py** - Professional vocal/instrumental separation using UVR5, MDX, VR, and Karafan models

**nodes/audio/merge_audio_node.py** - Advanced audio mixing with multiple algorithms and pitch control

**nodes/audio/rvc_pitch_options_node.py** - Advanced pitch extraction settings for RVC voice conversion

### Video Analysis System

**nodes/video/mouth_movement_analyzer_node.py** - 🗣️ Silent Speech Analyzer for detecting mouth movement in silent video frames to extract precise timing for TTS SRT synchronization

**nodes/video/viseme_options_node.py** - Advanced viseme detection configuration with vowel/consonant detection, word prediction, and temporal analysis settings

### RVC Model Management

**nodes/models/load_rvc_model_node.py** - RVC model loader with FAISS index support and automatic downloading

## Base Classes and Foundation

**nodes/base/base_node.py** - Universal base class for all nodes with device resolution, model management, and cleanup

**nodes/base/f5tts_base_node.py** - F5-TTS specific base class extending universal foundation with F5-TTS requirements

## Utility Systems

### Model Management

**utils/models/manager.py** - Intelligent model discovery and caching with multilingual support for both TTS and VC models, integrated with ComfyUI model management and extra_model_paths.yaml for shared storage support

**utils/models/smart_loader.py** - Universal smart model loader preventing duplicate model loading across all engines and modes

**utils/models/f5tts_manager.py** - F5-TTS specific model management extending base manager functionality, now with ComfyUI integration

**utils/models/language_mapper.py** - Language-to-model mapping system with fallback support

**utils/models/fallback_utils.py** - Model fallback and error recovery utilities

**utils/models/comfyui_model_wrapper/** - Modular ComfyUI model wrapper system with engine-specific handlers (base_wrapper.py, model_manager.py, cache_utils.py, engine_handlers.py) enabling TTS models to integrate with ComfyUI's native model management

**utils/models/unified_model_interface.py** - Universal model loading interface providing standardized factory pattern for all engines (ChatterBox, F5-TTS, Higgs Audio, RVC, Audio Separation) with ComfyUI integration

**utils/models/extra_paths.py** - ComfyUI extra_model_paths.yaml integration system enabling shared model directories across multiple ComfyUI installations with intelligent path resolution and fallback support

### Audio Processing

**utils/audio/processing.py** - Comprehensive audio utilities for tensor manipulation, normalization, and format conversion

**utils/ffmpeg_utils.py** - Centralized FFmpeg dependency handling with graceful fallbacks for audio conversion and timing analysis

**utils/audio/analysis.py** - Advanced audio analysis including waveform processing, silence detection, and timing extraction

**utils/audio/audio_hash.py** - Centralized content-based hashing for consistent cache keys across all processing modes

**utils/audio/cache.py** - Unified caching system for TTS engines with engine-specific cache management

**utils/audio/chunk_combiner.py** - Modular chunk combination utility with smart auto-selection based on text analysis (sentence boundaries, commas, forced splits) and detailed timing information

**utils/audio/chunk_timing.py** - Chunk combination timing information helper providing standardized timing info integration across all TTS engines

### Text Processing

**utils/text/character_parser.py** - Universal character switching system with [CharacterName] tag parsing and language-aware processing

**utils/text/chunking.py** - Enhanced text chunking with sentence boundary detection and character limits

**utils/text/pause_processor.py** - Pause tag parsing supporting [pause:xx] syntax for precise timing control

**utils/text/segment_parameters.py** - Per-segment parameter system enabling inline control of seed, temperature, cfg, speed, and other engine-specific parameters via pipe-separated syntax like [Alice|seed:42|temperature:0.5]. Supports aliases (cfg_weight→cfg, temp→temperature) and case-insensitive input with universal parameter filtering and validation across all TTS engines

**utils/text/phonemizer_utils.py** - F5-TTS multilingual phonemization system with IPA conversion, cross-platform backend support (espeak-phonemizer-windows/phonemizer), smart language detection, and model-specific exceptions for optimal quality

### Voice Management

**utils/voice/discovery.py** - Enhanced voice file discovery with multi-path fallback system (models/voices/, models/TTS/voices/, extra_model_paths.yaml directories, voices_examples/), character mapping, and alias loading with priority support

**utils/voice/multilingual_engine.py** - Central orchestrator for multilingual TTS with language switching optimization

### System Integration

**utils/system/import_manager.py** - Smart dependency resolution managing bundled vs system installations

**utils/system/subprocess.py** - Safe subprocess execution with error handling and process isolation

### Compatibility System

**utils/compatibility/transformers_patches.py** - Centralized transformers compatibility patches (monkey patches) managing version compatibility across different transformers library versions including FlashAttentionKwargs, BaseStreamer, DynamicCache properties, and VibeVoice generation method signatures

**utils/compatibility/numba_compat.py** - Centralized Numba/Librosa compatibility system with fast startup testing and intelligent JIT fallback management for Python 3.13+ compatibility

### Downloads

**utils/downloads/unified_downloader.py** - Centralized download system for all engines eliminating HuggingFace cache duplication with direct HTTP downloads to organized TTS/ structure, featuring full extra_model_paths.yaml support for shared model directories

**utils/downloads/model_downloader.py** - RVC-specific model auto-download system with direct downloads (legacy, integrated with unified system)

### Timing and SRT

**utils/timing/engine.py** - Advanced SRT timing engine with smart audio synchronization

**utils/timing/assembly.py** - Audio segment assembly supporting multiple timing modes

**utils/timing/parser.py** - SRT subtitle parsing with timestamp extraction and validation

**utils/timing/reporting.py** - Timing analysis and report generation

### Streaming System

**utils/streaming/__init__.py** - Universal streaming system initialization

**utils/streaming/streaming_types.py** - Universal data structures (StreamingSegment, StreamingResult) eliminating format conversions

**utils/streaming/streaming_interface.py** - Abstract streaming interface for engine adapters

**utils/streaming/streaming_coordinator.py** - Universal coordinator replacing all format-specific routers

**utils/streaming/work_queue_processor.py** - Engine-agnostic parallel processing system

## Web Interface Components

### TTS Tag Editor (Modularized Architecture)

**web/string_multiline_tag_editor.js**, **widget-event-handlers.js**, **widget-ui-builder.js**, **widget-parameter-section.js**, **widget-preset-system.js**, **editor-state.js**, **tag-utilities.js**, **syntax-highlighter.js**, **font-controls.js** - Modular TTS tag editor with event handling, UI sections, parameters, presets, state management, tag utilities, syntax highlighting, and font controls for character/parameter tag editing with undo/redo, formatting, and persistence

### Audio Analysis

**web/audio_analyzer_interface.js** - Main ComfyUI integration and node communication for Audio Wave Analyzer

**web/audio_analyzer_core.js** - Core JavaScript functionality for waveform visualization

**web/audio_analyzer_ui.js** - User interface components and control elements

**web/audio_analyzer_visualization.js** - Waveform rendering and canvas-based visualization

**web/audio_analyzer_regions.js** - Timing region management and interaction system

**web/audio_analyzer_controls.js** - Audio playback controls and user interaction handling

**web/audio_analyzer_widgets.js** - Custom ComfyUI widget implementations

**web/audio_analyzer_drawing.js** - Canvas drawing utilities and rendering functions

**web/audio_analyzer_events.js** - Event handling for mouse and keyboard interactions

**web/audio_analyzer_layout.js** - Layout management and UI positioning

**web/audio_analyzer_node_integration.js** - Node integration bridge for ComfyUI communication

**web/chatterbox_voice_capture.js** - Voice recording interface for microphone input

**web/index_tts_emotion_radar.js** - ComfyUI integration for 🌈 IndexTTS-2 Emotion Vectors radar chart

**web/emotion_radar_canvas_widget.js** - Interactive radar chart widget with dynamic color blending and click/drag controls

**web/chatterbox_srt_showcontrol.js** - SRT subtitle display and timing controls

**web/audio_analyzer.css** - Styling for audio analyzer interface components

## Development Tools

**scripts/bump_version_enhanced.py** - Advanced automated version management with changelog generation and git integration

**scripts/version_utils.py** - Version management utilities and helper functions

## Configuration Files

**requirements.txt** - Core dependencies including ChatterboxTTS, audio processing libraries, and optional F5-TTS dependencies

**pyproject.toml** - Project metadata with ComfyUI registry configuration and dependency specifications

## Example Workflows

**example_workflows/📺 Chatterbox SRT.json** - Complete SRT processing workflow with ChatterBox engine

**example_workflows/Chatterbox integration.json** - General ChatterBox TTS and Voice Conversion demonstration

**example_workflows/👄 F5-TTS Speech Editor Workflow.json** - F5-TTS speech editing with Audio Wave Analyzer

**example_workflows/🎤 📺 F5-TTS SRT and Normal Generation.json** - F5-TTS SRT processing and standard generation

## Voice Examples

**voices_examples/** - Character voice reference files with both audio samples and text descriptions
- **male/** - Male voice examples with reference audio and text
- **female/** - Female voice examples with reference audio and text
- **higgs_audio/** - Higgs Audio 2 voice presets (belinda, en_woman, en_man, chadwick, vex, etc.) with config.json
- **#character_alias_map.txt** - Character alias mapping configuration
- Individual character voice files (Clint Eastwood, David Attenborough, Morgan Freeman, Sophie Anderson)

## Project Management

**IgnoredForGitHubDocs/** - Development documentation excluded from main repository
- **FOLDER_STRUCTURE_REFACTORING_PLAN.md** - Architectural refactoring planning
- **CHATTERBOX_MODEL_BUG_REPORT.md** - Bug tracking and resolution documentation

## Architecture Summary

The TTS Audio Suite follows a **unified modular architecture** where:

1. **Engine Nodes** (`nodes/engines/`) provide user configuration interfaces
2. **Unified Nodes** (`nodes/unified/`) serve as clean delegation layers routing to appropriate engine processors  
3. **Engine Processors** (`nodes/chatterbox/`, `nodes/f5tts/`, `nodes/higgs_audio/`) handle engine-specific orchestration and workflow logic
4. **Engine Implementations** (`engines/`) handle the actual TTS/VC processing and model inference
5. **Adapters** (`engines/adapters/`) bridge low-level engines to higher-level processors
6. **Utilities** (`utils/`) provide shared functionality across all components
7. **Web Interface** (`web/`) enables interactive features like audio analysis

This layered design ensures consistent user experience while allowing engine-specific optimizations. The unified nodes act as thin delegation layers, eliminating code duplication and maintaining architectural consistency across all TTS engines.