diodiogod
517d11ed4c
Add DramaBox and ChatterBox V3 support
...
- Integrate DramaBox with unified TTS and SRT generation, duration targeting, prompt templates, memory strategies, FP8, compile, and silence diagnostics
- Add ChatterBox 23-Lang V3 checkpoint support and artifact handling
- Extend parameter switching, generated-audio caching, tag editor controls, model downloads, and engine registration
- Add user documentation, engine metadata, generated comparison tables, and FL-MCP launcher reliability updates
2026-07-25 11:38:52 -03:00
diodiogod
88feb3a99a
Add Qwen3-TTS engine with Phase 1 implementation
...
Implements complete Qwen3-TTS integration following NEW_ENGINE_IMPLEMENTATION_GUIDE
with unified architecture, character switching, language support, and Voice Designer.
## Core Engine Implementation
**Engine Wrapper** (`engines/qwen3_tts/qwen3_tts.py`):
- Wraps official Qwen3-TTS implementation with .to() method for VRAM management
- Device checking before generation with automatic model reload
- Support for 3 model types: CustomVoice (presets), VoiceDesign (text-to-voice), Base (voice cloning)
- Progress bar integration via transformers streamer interface
- 24kHz mono audio output in ComfyUI-compatible format
**Bundled Implementation** (`engines/qwen3_tts/impl/`):
- Official Qwen3-TTS library bundled for stability
- Patched modeling_qwen3_tts.py (line 2069) for progress callback support
- Sox dependency stubbed out (25Hz models only, we use 12Hz)
- Supports 6 models: 0.6B/1.7B × CustomVoice/VoiceDesign/Base
**Adapter** (`engines/adapters/qwen3_tts_adapter.py`):
- Intelligent model selection based on context (voice_preset/node_type)
- Automatic switching: preset→CustomVoice, None→Base, Designer→VoiceDesign
- Full caching integration with audio_cache.generate_cache_key()
- Voice reference processing for Base model (audio + transcript)
- Character-to-speaker mapping for CustomVoice presets (deterministic MD5 hashing)
## Node Implementation
**Engine Configuration Node** (`nodes/engines/qwen3_tts_engine_node.py`):
- Unified interface with voice_preset dropdown (9 preset speakers + None for zero-shot)
- Model size selection: 0.6B (low VRAM) / 1.7B (high quality)
- Language dropdown: Auto + 10 languages (Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian)
- Conditional instruction field (locked when unsupported: Base model or 0.6B CustomVoice)
- Generation parameters: top_k, top_p, temperature, repetition_penalty, max_new_tokens
- Advanced: dtype, attn_implementation, x_vector_only_mode
**Voice Designer Node** (`nodes/qwen3_tts/qwen3_tts_voice_designer_node.py`):
- UNIQUE FEATURE: Create voices from natural language descriptions
- Smart disk check: loads existing characters without model loading/generation
- Runtime caching for same-session regeneration optimization
- Saves to models/voices/ with standard format (.wav + .reference.txt + .txt metadata)
- Outputs unified NARRATOR_VOICE format matching Character Voices node
- Duplicate detection prevents regenerating identical voices
**TTS Processor** (`nodes/qwen3_tts/qwen3_tts_processor.py`):
- Full character switching support via character parser
- CustomVoice: deterministic character→preset mapping (preserves narrator's chosen preset)
- Base: uses models/voices/ audio references (standard voice cloning)
- VoiceDesign: uses saved voice descriptions from Designer node
- Pause tag processing with silence generation
- Language switching support via [language:Character] tags
- Segment parameter support ([seed:X], [temperature:0.8])
- Interrupt handling for cancellation
- Chunk combining with ChunkCombiner + ChunkTimingHelper
## JavaScript Widget
**Widget Locking** (`web/qwen3_tts_widgets.js`):
- Conditional UI: locks instruction field when unsupported
- Locked when: voice_preset="None" OR (model_size="0.6B" + preset selected)
- Console warnings explain limitations
- Preserves field value when locked (doesn't clear)
## System Integration
**Language Support** (`utils/models/language_mapper.py`):
- Added QWEN3_TTS_LANGUAGE_MAP: maps ISO codes to full names (e.g., "po"→"Portuguese")
- Added map_language_code_to_qwen3_tts() helper function
- Added "po" to global LANGUAGE_ALIASES (maps to pt-pt)
- Qwen3-TTS expects FULL language names, not ISO codes
**Unified Model Interface** (`utils/models/unified_model_interface.py`):
- Registered qwen3_tts factory in initialize_all_factories()
- Consolidated loading messages (single-line model loading output)
- Context-based model selection (VoiceDesign/CustomVoice/Base)
**Segment Parameters** (`utils/text/segment_parameters.py`):
- Added "qwen3_tts" to PARAMETER_ENGINES for inline tag support
**Voice Discovery** (`utils/voice/discovery.py`):
- Enhanced get_character_mapping() to detect VoiceDesign saved voices
- Checks for voice metadata format (description vs audio reference)
- Returns voice type for processor to handle correctly
**TTS Text Node** (`nodes/unified/tts_text_node.py`):
- Added qwen3_tts processor routing
- CustomVoice console messages (shows preset speaker instead of narrator warning)
- Recognizes unified NARRATOR_VOICE format from Voice Designer
- Language code mapping display (shows "po" → "Portuguese" conversion)
**Node Registration** (`nodes.py`):
- Registered Qwen3TTSEngineNode and Qwen3TTSVoiceDesignerNode
- Added error handling with fallback messages
## Features Implemented (Phase 1 Complete)
✅ Basic engine integration with unified architecture
✅ Character switching (CustomVoice presets + Base model cloning)
✅ Language switching (10 languages + Auto detection)
✅ Pause tags ([pause:1.5])
✅ Segment parameters ([seed:42], [temperature:0.8])
✅ Audio caching (prevents duplicate generation)
✅ VRAM management (.to() method + device checking)
✅ Interrupt handling (user cancellation)
✅ Progress bars (console + ComfyUI)
✅ Voice Designer (text-to-voice creation)
✅ Smart disk caching (skips regeneration for saved voices)
✅ Unified NARRATOR_VOICE output format
## Technical Details
- **Sample Rate**: 24kHz mono
- **Models**: 6 variants (0.6B/1.7B × CustomVoice/VoiceDesign/Base)
- **Tokenizer**: Qwen3-TTS-Tokenizer-12Hz (12.5 fps, 16 quantizers)
- **Languages**: 10 (Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian)
- **CustomVoice Presets**: 9 (Vivian, Serena, Uncle_Fu, Dylan, Eric, Ryan, Aiden, Ono_Anna, Sohee)
- **Audio Format**: torch.Tensor [1, 1, samples] at 24kHz
- **Cache Integration**: Uses audio_cache with stable key generation
- **Character Mapping**: MD5-based deterministic assignment (cache-friendly)
## Testing Completed
- Engine node loads and configures correctly
- Preset voice generation (CustomVoice with 9 speakers)
- Zero-shot cloning (Base model with voice references)
- Voice Designer text-to-voice generation
- Character switching with automatic preset mapping
- Pause tags create correct silence durations
- Language selection and switching
- Caching prevents duplicate generation
- VRAM management (Clear VRAM + reload)
- Generation info reports with timing
- Interrupt cancellation
- Widget locking behavior
- Disk check optimization (skips model loading for existing voices)
## Phase 2+ (Future)
- SRT processor integration (Phase 4)
- Streaming generation (requires special deployment)
- Fine-tuning nodes (advanced feature)
2026-01-28 00:02:02 -03:00
diodiogod
e4a48045a1
Add per-segment parameter switching system (VibeVoice Custom Character Switching)
...
- New utils/text/segment_parameters.py with parameter validation, aliases, and engine compatibility
- Extended CharacterSegment to include per-segment parameters
- Extended language resolver to parse pipe-separated parameters: [Alice|seed:42|temp:0.5]
- Support for: seed, temperature, cfg, speed, top_p, top_k, inference_steps, exaggeration, emotion_alpha
- Case-insensitive aliases and order-independent syntax
- Integrated into VibeVoice Custom Character Switching mode (text and SRT)
- Fixed grouping logic to break groups when parameters change
- Added PARAMETER_SWITCHING_GUIDE.md documentation
- Updated PROJECT_INDEX.md
NOTE: Currently implemented for VibeVoice Custom Character Switching only.
Native Multi-Speaker mode and other engines will be added in future updates.
2025-10-29 00:36:44 -03:00
diodiogod