- Remove Speed column (unnecessary for overview)
- Remove 🎯 emoji from title (too many emojis)
- Add ChatterBox to featured engines (now shows 7 engines)
- Created comprehensive YAML source of truth for all TTS engines
- Implemented Python generator for 3 documentation tables:
* ENGINE_COMPARISON.md - Main engine overview with capabilities
* LANGUAGE_SUPPORT.md - Detailed language support matrix
* FEATURE_COMPARISON.md - Feature capabilities comparison
- YAML captures all engine metadata without information loss
- Tables auto-generated via scripts/generate_engine_tables.py
- Added flag emojis to language support table for easy scanning
- Speed performance fixes: F5-TTS marked as "Very Fast"
Fixed 'dict' object has no attribute 'cpu' error when using Voice Designer
voices with TTS Text node.
## Root Cause
Voice Designer and Character Voices nodes output unified NARRATOR_VOICE format:
{
"audio": {"waveform": tensor, "sample_rate": 24000},
"audio_path": "...",
"reference_text": "...",
...
}
But TTS Text node was wrapping this AGAIN when creating voice_mapping:
{
"waveform": audio_tensor, # audio_tensor was already a dict!
"sample_rate": 24000,
"text": "..."
}
This created triple-nested structure that broke audio hashing.
## Fixes Applied
**TTS Text Node** (nodes/unified/tts_text_node.py):
- Extract actual waveform tensor from audio dict before creating voice_mapping
- Applied to all 6 places where audio_tensor is used:
- Narrator voice mapping (line 1151)
- Character ICL fallback (line 1183)
- Character x_vector fallback (line 1217)
- Character no-text fallback (line 1220)
- Character no-audio fallback (line 1237)
- Pattern: Check if audio_tensor is dict, extract waveform/sample_rate if so
**Qwen3-TTS Adapter** (engines/adapters/qwen3_tts_adapter.py):
- Extract inner audio dict before passing to hash function
- Changed from passing full voice_ref to passing voice_ref['audio']
**Audio Hash Utility** (utils/audio/audio_hash.py):
- Added nested structure handling with fallback
- Detects Character Voices format {"audio": {...}} vs direct format
- Enhanced debug logging to diagnose structure issues
## Testing
- Voice Designer → TTS Text: Works without hash error
- Character Voices → TTS Text: Works without hash error
- Audio hashing generates stable keys for caching
- Both VoiceDesign and traditional voices function correctly
New Feature: <restore:N@M> Syntax
- <restore> - Restore using original pre-edit audio (1 VC pass)
- <restore:N> - N VC passes using original audio as reference
- <restore:N@M> - N VC passes using audio from iteration M as reference
Implementation:
- Store audio snapshots after each edit iteration
- Parse iteration reference from @ syntax in restore tags
- Use specified iteration audio as VC reference instead of original
- Allows preserving partial effects (e.g., keep whisper, remove laughter)
Use Case Example:
<style:whisper:2> <Laughter:3> <restore:1@2>
- Applies whisper (iterations 1-2)
- Applies laughter (iterations 3-5)
- Restores using iteration 2 (whispered audio) with 1 VC pass
- Result: Keeps whisper effect but removes laughter voice degradation
Documentation:
- Updated INLINE_EDIT_TAGS_USER_GUIDE.md with restore tag section
- Added examples for basic and iteration-reference restoration
- Explained use cases and when to use each syntax
- Updated processing order and reference sections
- Complete guide for Step Audio EditX inline tag system
- Covers all tag types: paralinguistic, emotion, style, speed
- ChatterBox 23-Lang v2 native vs post-processing tag distinction
- Language support warnings (Chinese, English, Japanese, Korean only)
- Voice resemblance degradation warnings for 3+ iterations
- Performance tips and best practices
- Inline vs manual node workflow comparison
- Pro tips for stronger effects with contextual text
- Planned voice restoration feature note
- FAQ and troubleshooting sections
Added edit tag extraction and batch processing for Higgs Audio TTS and SRT processors following the established pattern from VibeVoice and Step Audio EditX.
Key changes:
- Extract edit tags at top level before pause processing to keep nested functions returning tensors
- Apply edit post-processing after all audio generation completes
- For SRT: batch process all subtitle edits at end instead of per-subtitle
- Fix CUDA graph crash: detect original engine for stateless wrappers and refuse unloading when CUDA graphs enabled
- Store original_engine in ModelInfo to track Higgs even when masked as stateless_tts
- Update Higgs handler to check correct attribute paths for _cuda_graphs_enabled flag
Files modified:
- nodes/higgs_audio/higgs_audio_tts_processor.py: Add edit tag extraction and post-processing
- nodes/higgs_audio/higgs_audio_srt_processor.py: Batch edit processing for all subtitles
- utils/models/comfyui_model_wrapper/base_wrapper.py: Track original_engine for masked engines
- utils/models/comfyui_model_wrapper/model_manager.py: Store original engine when masking
- utils/models/comfyui_model_wrapper/engine_handlers/higgs_audio_handler.py: Refuse unload when CUDA graphs enabled
- docs/Dev reports/SRT_Edit_Tag_Batch_Processing_Pattern.md: Mark Higgs as implemented
Fixed inefficient per-subtitle edit processing that caused multiple engine loads.
Now all edit processing happens in a single batch after all subtitles are generated.
Changes:
- VibeVoice SRT: Collects all segments with edit_tags, applies edits once at end
- Step Audio EditX SRT: Same batching pattern, skips edits in _process_character_switching when in SRT mode
- Step processor: Added check to only apply edits when NOT in SRT mode
- Subtitle processing functions: Return segment dicts instead of applying edits directly
Benefits:
- Edit engine loads ONCE instead of once per subtitle
- Significantly faster SRT generation with edit tags
- Less VRAM thrashing from repeated model loading/unloading
- Maintains same quality output
Added dev report documenting the pattern for future engine implementations.
Major changes:
1. Post-processor now uses Audio Editor node instead of calling engine directly
- Inherits progress bars, caching, tested code
- Much simpler and more maintainable
2. Fixed unified model interface usage across Step Audio EditX
- Adapter now loads via unified_model_interface.load_model()
- Audio Editor node now loads via unified_model_interface.load_model()
- Both use model_type="tts" for consistent cache keys
- Prevents duplicate model loading (was loading twice before)
3. Fixed pause tag + edit tag interaction
- Pause tags split FIRST, then edit tags extracted per segment
- Allows: "Hello <Laughter> [pause:1] World <Sigh>"
- Laughter applies to "Hello", Sigh applies to "World"
4. Added lesson to fails document
- All engines must use unified interface in ALL code paths
- Cache key must match across adapters/nodes/processors
Bug fixes:
- Missing Dict import in post-processor
- Import path for Audio Editor node (use importlib)
- Progress bars now work during editing
Files modified:
- utils/audio/edit_post_processor.py - refactored to use Audio Editor node
- nodes/step_audio_editx/step_audio_editx_processor.py - pause handling
- engines/adapters/step_audio_editx_adapter.py - unified interface
- nodes/step_audio_editx_special/step_audio_editx_audio_editor_node.py - unified interface
- docs/Dev reports/fails_to_avoid_TTS_Engine_Implementation.md - added lesson
Added support for inline edit tags in TTS text that automatically apply
Step Audio EditX edits to generated audio from ANY engine.
Syntax examples:
- <Laughter:2> - paralinguistic with iterations
- <emotion:happy:1> - emotion edit
- <style:whisper:3> - style edit
- <speed:faster> - speed adjustment
- <Laughter:2|style:whisper> - multiple effects
Key changes:
- Extended step_audio_editx_special_tags.py with EditTag dataclass and iteration parsing
- Created edit_post_processor.py for centralized post-TTS edit application
- Integrated into Step Audio EditX processor with proper pause tag handling
- Pause tags split FIRST, then edit tags extracted per pause-segment
- Reuses already-loaded engine to avoid duplicate loading
Flow:
1. Parse text, split by pause tags first
2. Extract edit tags from each pause-delimited segment
3. Generate TTS with clean text (tags stripped)
4. Post-processor applies edits to segments with tags
5. Assemble final audio
Currently working for Step Audio EditX TTS and SRT nodes.
Replace complex weight restoration workaround with a simple one-line config fix.
The root cause: Step-Audio-EditX model config was missing 'tie_word_embeddings'
setting, so transformers 4.54+ defaulted to True and tied lm_head to embed_tokens
even though the checkpoint has different weights for each.
Solution: Add 'tie_word_embeddings': false to config.json to prevent tying.
This is much simpler and cleaner than the previous safetensors weight restoration
approach. One line in config.json fixes the issue for all transformers versions.
- Add step_audio_editx to PARAMETER_ENGINES for seed and temperature
- Fix engine node tooltips: parameters ARE used in clone mode (not hardcoded)
- Add reference implementation paths to implementation plan
- Update fails doc with latest lessons learned
Make both engines follow ChatterBox 23-Lang modular pattern:
**Step Audio EditX:**
- Update combine_audio_segments() to use ChunkCombiner with return_info=True
- Return (audio, chunk_info) tuple when requested
- Fix single segment shape handling (ensure 2D [channels, samples])
**IndexTTS:**
- Add combine_audio_segments() method using ChunkCombiner
- Update process_text() to accept return_info parameter
- Build chunk_info from text segments and return timing data
**Unified Node:**
- Call both engines with return_info=True
- Use ChunkTimingHelper.enhance_generation_info() for detailed reports
- Clean text without tags before calculating length
**Documentation:**
- Add generation_info consistency requirement to fails document
- Document correct pattern: ChunkCombiner + ChunkTimingHelper
- Note wrong pattern: inline torch.cat without timing
Now both engines provide detailed timing reports like F5-TTS and ChatterBox.
Models should only unload when explicitly requested (Clear VRAM button) or when
switching engines - NOT automatically after generation completes.
__del__ was causing models to unload immediately after wrapper instance goes
out of scope, preventing model reuse between generations.
Pattern: F5-TTS and ChatterBox don't have __del__, only Index2 and StepAudio had it.
Removed to match correct behavior.
- Configure character parser with available characters and language defaults like IndexTTS
- Add characters from text tags to available_characters to prevent fallback to "narrator"
- Add fallback voice mapping entries for characters without voice files
- Update fails document with character switching implementation lessons
Implemented Step Audio EditX integration with the following components:
Engine Implementation:
- Bundled Step Audio EditX reference code in step_audio_editx_impl/
- Created engine wrapper (step_audio_editx.py) with clone/edit methods
- Created adapter (step_audio_editx_adapter.py) for unified interface
- Created processor (step_audio_editx_processor.py) for TTS Suite integration
Model Management:
- Created downloader (step_audio_editx_downloader.py) for:
- Main Step-Audio-EditX model (3B LLM + CosyVoice vocoder)
- Step-Audio-Tokenizer files (linguistic_tokenizer.npy, speech_tokenizer_v1.onnx)
- FunASR-Paraformer model with correct config for bundled funasr_detach
- Fixed FunASR config to use bundled class names (ParaformerStreaming, WavFrontendOnline, SpecAugLFR)
- Added auto-download for all required models on first use
- Updated unified_model_interface.py to auto-download Step-Audio-EditX components
Node Configuration:
- Created Step Audio EditX engine config node
- Registered in nodes.py and unified TTS text node
Known Issues:
- LLM generation hangs during inference despite:
- torch.no_grad() wrapper
- torch.cuda.synchronize()
- Eager attention mode (Step1ForCausalLM doesn't support SDPA)
- Reduced max_new_tokens
- Removed logits processor
- Model loads successfully, all components initialized correctly
- Issue appears to be with transformers.generate() call itself
- Requires further investigation of Step1ForCausalLM compatibility
Files Modified:
- nodes.py: Added Step Audio EditX registration
- requirements.txt: Added FunASR dependencies
- utils/models/unified_model_interface.py: Added Step Audio EditX factory
- Various Step Audio EditX implementation files
Model Downloads:
- Step-Audio-EditX: 7GB (3B LLM + vocoder)
- FunASR-Paraformer: 881MB (from custom HF repo with fixed config)
- Step-Audio-Tokenizer: Bundled with main model download
Fix Italian ChatterBox case-sensitive language prefix detection
- Make [it] prefix detection case-insensitive to prevent double-prefixing
- Users can now use [it], [IT], [It] or any capitalization without issues
- Avoids duplication that caused 'IT' to be spoken in output
- New utils/text/segment_parameters.py with parameter validation, aliases, and engine compatibility
- Extended CharacterSegment to include per-segment parameters
- Extended language resolver to parse pipe-separated parameters: [Alice|seed:42|temp:0.5]
- Support for: seed, temperature, cfg, speed, top_p, top_k, inference_steps, exaggeration, emotion_alpha
- Case-insensitive aliases and order-independent syntax
- Integrated into VibeVoice Custom Character Switching mode (text and SRT)
- Fixed grouping logic to break groups when parameters change
- Added PARAMETER_SWITCHING_GUIDE.md documentation
- Updated PROJECT_INDEX.md
NOTE: Currently implemented for VibeVoice Custom Character Switching only.
Native Multi-Speaker mode and other engines will be added in future updates.
Add Step 9.5 to NEW_ENGINE_IMPLEMENTATION_GUIDE.md:
- Explain ComfyUI interrupt mechanism and why it's critical
- Provide import pattern and code examples
- Show where to add interrupt checks in SRT and TTS processors
- Add test checklist item for interrupt handling
- Reference real implementation examples
- Add interrupt checking to critical integration points list
This ensures future engine implementations include proper cancellation support from day one.
- Add prominent warning that special tokens have limited effectiveness
- Link to ResembleAI issue #186 showing lack of official documentation
- Note that implementation is ready for future improvements
- Be honest about testing results: tokens may produce minimal effects
- Don't recommend v1 over v2 (v2 has other improvements beyond tags)
Implemented full support for ChatterBox v2 with enhanced tokenization featuring
30+ special tokens for emotions, sounds, and vocal effects.
Key Features:
- Model version selection (v1/v2) in engine configuration node
- Auto-download of v2 model files (t3_mtl23ls_v2.safetensors, grapheme mappings)
- Special tags system using <emotion> syntax to avoid character switching conflicts
- Automatic conversion of <giggle>, <sigh>, <whisper> etc. to v2 format
- Cache invalidation between v1/v2 to prevent conflicts
- Full integration with character switching and pause tags
Technical Changes:
- Added model_version parameter throughout TTS and SRT processing chain
- Created chatterbox_v2_special_tags.py for tag conversion
- Updated cache keys to include model_version
- Version-aware model loading and validation
- Tag conversion happens after character/pause parsing, before engine
User can now write:
[Alice] Hello! <giggle> Nice to meet you. [pause:0.5] <whisper> Secret!
Documentation added in docs/CHATTERBOX_V2_SPECIAL_TOKENS.md
Update bundled IndexTTS-2 to latest upstream version
Technical implementation:
- Sync with https://github.com/index-tts/index-tts commits 8336824..64cb31a
- Add normalize_emo_vec() function from commit 8aa8064 for voice identity protection
- Add _load_and_cut_audio() helper from commit 0828dcb for 15s audio truncation
- Add persistent cache clearing from commit 64cb31a with torch.cuda.empty_cache()
- Fix BigVGAN CUDA kernel import path from commits ee23371, e409c4a
- Enhance emotion vector logging for better user feedback
- Add IndexTTS-2 to Multi-Engine TTS feature list
- Add IndexTTS-2 With Emotion Control section to What's New
- Add IndexTTS-2 Model Downloads section with correct model structure
- Include proper repository links (IndexTeam/IndexTTS-2, facebook/w2v-bert-2.0)
- Document unified emotion control architecture and features
- Add comprehensive IndexTTS-2 Emotion Control Guide
- Initial IndexTTS-2 TTS engine implementation
- Basic unified interface integration
- Auto-download system for IndexTTS-2 models
- Character voice switching support via QwenEmotion
- Token-based chunking system
- ComfyUI model management integration
- Fixed debug output and basic inference working
- Multi-language text preprocessing foundation
- Basic error handling framework
Note: Initial implementation - requires further testing and refinement
Major Features:
• IndexTTS-2 engine with zero-shot voice cloning and emotion control
• Multi-modal emotion support (audio refs, text descriptions, manual vectors)
• Character switching with emotion references: [Alice:angry_bob]
• Native IndexTTS-2 token-based chunking integration
• Comprehensive parameter exposure with 25+ engine-specific options
Core Components:
• engines/processors/index_tts_processor.py - Main processor with emotion support
• nodes/engines/index_tts_engine_node.py - Engine configuration node
• Updated unified TTS Text node with IndexTTS-2 integration
• Node registration in main nodes.py
Dependencies:
• Added 8 safe IndexTTS-2 dependencies to install.py and requirements.txt
• Dependency testing confirmed only descript-audiotools conflicts (already handled)
• All new deps use flexible versioning (>=) to avoid lock conflicts
Character Parser Fix:
• Fixed circular import by moving CharacterSegment to separate types.py
• Maintains backward compatibility while supporting emotion references
• Modular structure preserved (4 modules under 500 lines each)
Technical Details:
• Emotion disentanglement separates speaker identity from emotional expression
• Supports [character:emotion] syntax for per-segment emotion control
• IndexTTS-2 native chunking overrides character-based chunking when enabled
• Full caching integration and pause tag support included
• Error handling with graceful fallbacks for missing voices/emotions
MAJOR: IndexTTS-2 TTS engine integration
- Core IndexTTS engine with unified model interface integration
- Auto-downloader for HuggingFace models + GitHub source code
- Engine adapter with emotion support [Alice:angry_bob] syntax
- Cache integration for IndexTTS-2 parameters
- Factory registration for proper model management
MAJOR: Character parser modularization (930 → 300 lines each)
- Split into base_parser.py, language_resolver.py, segment_processor.py, validation.py
- Added emotion support with backward compatibility
- New split_by_character_with_emotions() method for advanced engines
- Flexible tag parsing: [de:Alice:angry_bob] support
- Language disambiguation to avoid Alice vs language code conflicts
- Full backward compatibility maintained
FEATURES:
- IndexTTS-2 emotion disentanglement (speaker vs emotion separation)
- Duration control and precise voice cloning
- Multi-modal emotion control (audio refs, text, vectors)
- Character switching with emotion references
- All official IndexTTS-2 parameters exposed
DOCS: Implementation plan and architectural TODO items added
Fix pkg_resources deprecation warning in installer
- Replace deprecated pkg_resources with importlib.metadata
- Add fallback support for Python < 3.8 compatibility
- Suppress warnings when using legacy fallback
Update VibeVoice 7B to official Microsoft repository
Technical details:
- Change repo from WestZhang/VibeVoice-Large-pt to microsoft/VibeVoice-Large
- Update documentation and downloader references
- Maintain backward compatibility with existing installations
Technical details:
- Add VibeVoice model loading and caching to ModelManager class
- Modify VibeVoice adapter to use ModelManager instead of creating own engine instances
- Add load_vibevoice_model() and unload_vibevoice_models() methods to ModelManager
- Import VibeVoice components into ModelManager with availability checking
- Update adapter to cache model and processor instances via ModelManager
- Follow ChatterBox pattern for persistent model caching across node executions
Fix F5-TTS model name case sensitivity issue
- Fix uppercase V1 being rejected in model validation
- Add regex normalization for version numbers (V1 -> v1)
- Ensure backward compatibility with saved workflows
Fix Higgs Audio model management with CUDA Graph compatibility toggle
Technical implementation:
- Add enable_cuda_graphs parameter to resolve CUDA Graph vs model unloading conflict
- Implement DynamicCache support for safe memory management mode
- Fix DynamicCache compatibility issues (reset, seq_length, copying)
- Add conditional cache creation and cleanup logic
- Update tooltip with accurate performance trade-offs (12 vs 55+ tok/s)
diodiogod