Improve VibeVoice and KugelAudio model support
Technical details:
- Add kugel-2 support as a VibeVoice KugelAudio variant
- Normalize KugelAudio v2 config fields for transformers 4.x compatibility
- Fix UnifiedDownloader alt_repo handling for VibeVoice Hindi tokenizer downloads
- Improve VibeVoice pause-tag tracing in custom character switching
- Update VibeVoice model registry and generated docs
Implementation details:
- add a standalone punctuation/truecase text node for raw ASR transcripts
- create a separate auxiliary-model YAML source of truth and docs generator
- register punctuation helper models outside the engine registry
- add lazy local download/runtime support for punctuators-backed ONNX models
- document auxiliary model sources and layouts for future helper-model expansion
- add README model download summary rows to the YAML source of truth
- generate and inject the README auto-download table via generate_engine_tables.py
- include Granite ASR in the generated model download summary
Add Granite ASR engine and expand unified ASR support
Implementation details:
- Add Granite ASR engine node, adapter, runtime, and unified downloader integration
- Reuse the Qwen forced aligner for optional custom Granite timestamps and SRT output
- Expose engine-side ASR translation target and prompt templates for Granite and Qwen
- Fix SRT overflow splitting so subtitle text is no longer silently dropped
- Update README and engine metadata for the expanded ASR lineup
- Fix inline edit tags (e.g. <Laughter>) incorrectly applying to entire
combined audio when pause tags are present. Now parse pause tags first,
extract edit tags per text fragment, apply edit post-processing per
fragment before combining with silence.
- Fix force_speaker_kv + short fragments: replace hard error with smart
auto-disable for fragments under 20 chars. Long fragments still benefit
from force_speaker_kv; short ones generate cleanly without noisy tails.
- Fix VRAM unload/reload crash ("Inference tensors do not track version
counter"): clone pca_state dict tensors before .to() to escape
inference-mode version tracking after ComfyUI "Unload Models".
- Add Echo-TTS to tts_audio_suite_engines.yaml source of truth
(English-only, ~7GB total, diffusion-based, CC-BY-NC-SA).
Integrate Egyptian Arabic (oddadmix) model and fix vocab mismatch
Technical details:
- Added 'oddadmix/chatterbox-egyptian-v0' to chatterbox-23l engine
- Implemented automatic download of 'resemble-ai/chatterbox' config.json to resolve vocabulary mismatch
- Updated engine documentation source of truth (YAML) with consistent Arabic flags
- Fix: Insert Zero Width Non-Joiner between flag emojis in tables to prevent ligatures
- Update: Set IndexTTS-2 model size to ~4.7GB in YAML and Tables
- Add dedicated KugelAudio implementation with relative imports
- Update engine/downloader/processor to support KugelAudio with fallback
- Regenerate engine comparison tables and update README
- Fix: Use relative imports for robustness
- Fix: Prevent 'Speaker 1:' prefix regression in processor
Restrict Transformers to Qwen3-TTS-compatible range
Implementation details:
- Cap transformers<=4.57.3 in requirements and installer
- Add startup warning when Transformers 5.x is detected
- Add investigation report for Qwen3-TTS + Transformers 5.x
Add Qwen3-TTS torch.compile optimizations (based on dffdeeq/Qwen3-TTS-streaming)
Implementation:
- Integrate streaming optimizations from dffdeeq/Qwen3-TTS-streaming fork
- Add torch.compile support with mode selection (default/reduce-overhead/max-autotune)
- Add manual CUDA graph capture option for additional optimization
- Update to support PyTorch 2.10+ and triton-windows 3.6+
- Remove obsolete inductor patches (not needed for PyTorch 2.10+)
- Add comprehensive optimization documentation with benchmark results
- Enable TensorFloat32 for Ampere+ GPUs automatically
Credit: Streaming implementation based on work by dffdeeq
https://github.com/dffdeeq/Qwen3-TTS-streaming
- Remove Speed column (unnecessary for overview)
- Remove 🎯 emoji from title (too many emojis)
- Add ChatterBox to featured engines (now shows 7 engines)
- Created comprehensive YAML source of truth for all TTS engines
- Implemented Python generator for 3 documentation tables:
* ENGINE_COMPARISON.md - Main engine overview with capabilities
* LANGUAGE_SUPPORT.md - Detailed language support matrix
* FEATURE_COMPARISON.md - Feature capabilities comparison
- YAML captures all engine metadata without information loss
- Tables auto-generated via scripts/generate_engine_tables.py
- Added flag emojis to language support table for easy scanning
- Speed performance fixes: F5-TTS marked as "Very Fast"
Fixed 'dict' object has no attribute 'cpu' error when using Voice Designer
voices with TTS Text node.
## Root Cause
Voice Designer and Character Voices nodes output unified NARRATOR_VOICE format:
{
"audio": {"waveform": tensor, "sample_rate": 24000},
"audio_path": "...",
"reference_text": "...",
...
}
But TTS Text node was wrapping this AGAIN when creating voice_mapping:
{
"waveform": audio_tensor, # audio_tensor was already a dict!
"sample_rate": 24000,
"text": "..."
}
This created triple-nested structure that broke audio hashing.
## Fixes Applied
**TTS Text Node** (nodes/unified/tts_text_node.py):
- Extract actual waveform tensor from audio dict before creating voice_mapping
- Applied to all 6 places where audio_tensor is used:
- Narrator voice mapping (line 1151)
- Character ICL fallback (line 1183)
- Character x_vector fallback (line 1217)
- Character no-text fallback (line 1220)
- Character no-audio fallback (line 1237)
- Pattern: Check if audio_tensor is dict, extract waveform/sample_rate if so
**Qwen3-TTS Adapter** (engines/adapters/qwen3_tts_adapter.py):
- Extract inner audio dict before passing to hash function
- Changed from passing full voice_ref to passing voice_ref['audio']
**Audio Hash Utility** (utils/audio/audio_hash.py):
- Added nested structure handling with fallback
- Detects Character Voices format {"audio": {...}} vs direct format
- Enhanced debug logging to diagnose structure issues
## Testing
- Voice Designer → TTS Text: Works without hash error
- Character Voices → TTS Text: Works without hash error
- Audio hashing generates stable keys for caching
- Both VoiceDesign and traditional voices function correctly
New Feature: <restore:N@M> Syntax
- <restore> - Restore using original pre-edit audio (1 VC pass)
- <restore:N> - N VC passes using original audio as reference
- <restore:N@M> - N VC passes using audio from iteration M as reference
Implementation:
- Store audio snapshots after each edit iteration
- Parse iteration reference from @ syntax in restore tags
- Use specified iteration audio as VC reference instead of original
- Allows preserving partial effects (e.g., keep whisper, remove laughter)
Use Case Example:
<style:whisper:2> <Laughter:3> <restore:1@2>
- Applies whisper (iterations 1-2)
- Applies laughter (iterations 3-5)
- Restores using iteration 2 (whispered audio) with 1 VC pass
- Result: Keeps whisper effect but removes laughter voice degradation
Documentation:
- Updated INLINE_EDIT_TAGS_USER_GUIDE.md with restore tag section
- Added examples for basic and iteration-reference restoration
- Explained use cases and when to use each syntax
- Updated processing order and reference sections
- Complete guide for Step Audio EditX inline tag system
- Covers all tag types: paralinguistic, emotion, style, speed
- ChatterBox 23-Lang v2 native vs post-processing tag distinction
- Language support warnings (Chinese, English, Japanese, Korean only)
- Voice resemblance degradation warnings for 3+ iterations
- Performance tips and best practices
- Inline vs manual node workflow comparison
- Pro tips for stronger effects with contextual text
- Planned voice restoration feature note
- FAQ and troubleshooting sections
Added edit tag extraction and batch processing for Higgs Audio TTS and SRT processors following the established pattern from VibeVoice and Step Audio EditX.
Key changes:
- Extract edit tags at top level before pause processing to keep nested functions returning tensors
- Apply edit post-processing after all audio generation completes
- For SRT: batch process all subtitle edits at end instead of per-subtitle
- Fix CUDA graph crash: detect original engine for stateless wrappers and refuse unloading when CUDA graphs enabled
- Store original_engine in ModelInfo to track Higgs even when masked as stateless_tts
- Update Higgs handler to check correct attribute paths for _cuda_graphs_enabled flag
Files modified:
- nodes/higgs_audio/higgs_audio_tts_processor.py: Add edit tag extraction and post-processing
- nodes/higgs_audio/higgs_audio_srt_processor.py: Batch edit processing for all subtitles
- utils/models/comfyui_model_wrapper/base_wrapper.py: Track original_engine for masked engines
- utils/models/comfyui_model_wrapper/model_manager.py: Store original engine when masking
- utils/models/comfyui_model_wrapper/engine_handlers/higgs_audio_handler.py: Refuse unload when CUDA graphs enabled
- docs/Dev reports/SRT_Edit_Tag_Batch_Processing_Pattern.md: Mark Higgs as implemented
Fixed inefficient per-subtitle edit processing that caused multiple engine loads.
Now all edit processing happens in a single batch after all subtitles are generated.
Changes:
- VibeVoice SRT: Collects all segments with edit_tags, applies edits once at end
- Step Audio EditX SRT: Same batching pattern, skips edits in _process_character_switching when in SRT mode
- Step processor: Added check to only apply edits when NOT in SRT mode
- Subtitle processing functions: Return segment dicts instead of applying edits directly
Benefits:
- Edit engine loads ONCE instead of once per subtitle
- Significantly faster SRT generation with edit tags
- Less VRAM thrashing from repeated model loading/unloading
- Maintains same quality output
Added dev report documenting the pattern for future engine implementations.
Major changes:
1. Post-processor now uses Audio Editor node instead of calling engine directly
- Inherits progress bars, caching, tested code
- Much simpler and more maintainable
2. Fixed unified model interface usage across Step Audio EditX
- Adapter now loads via unified_model_interface.load_model()
- Audio Editor node now loads via unified_model_interface.load_model()
- Both use model_type="tts" for consistent cache keys
- Prevents duplicate model loading (was loading twice before)
3. Fixed pause tag + edit tag interaction
- Pause tags split FIRST, then edit tags extracted per segment
- Allows: "Hello <Laughter> [pause:1] World <Sigh>"
- Laughter applies to "Hello", Sigh applies to "World"
4. Added lesson to fails document
- All engines must use unified interface in ALL code paths
- Cache key must match across adapters/nodes/processors
Bug fixes:
- Missing Dict import in post-processor
- Import path for Audio Editor node (use importlib)
- Progress bars now work during editing
Files modified:
- utils/audio/edit_post_processor.py - refactored to use Audio Editor node
- nodes/step_audio_editx/step_audio_editx_processor.py - pause handling
- engines/adapters/step_audio_editx_adapter.py - unified interface
- nodes/step_audio_editx_special/step_audio_editx_audio_editor_node.py - unified interface
- docs/Dev reports/fails_to_avoid_TTS_Engine_Implementation.md - added lesson
Added support for inline edit tags in TTS text that automatically apply
Step Audio EditX edits to generated audio from ANY engine.
Syntax examples:
- <Laughter:2> - paralinguistic with iterations
- <emotion:happy:1> - emotion edit
- <style:whisper:3> - style edit
- <speed:faster> - speed adjustment
- <Laughter:2|style:whisper> - multiple effects
Key changes:
- Extended step_audio_editx_special_tags.py with EditTag dataclass and iteration parsing
- Created edit_post_processor.py for centralized post-TTS edit application
- Integrated into Step Audio EditX processor with proper pause tag handling
- Pause tags split FIRST, then edit tags extracted per pause-segment
- Reuses already-loaded engine to avoid duplicate loading
Flow:
1. Parse text, split by pause tags first
2. Extract edit tags from each pause-delimited segment
3. Generate TTS with clean text (tags stripped)
4. Post-processor applies edits to segments with tags
5. Assemble final audio
Currently working for Step Audio EditX TTS and SRT nodes.
Replace complex weight restoration workaround with a simple one-line config fix.
The root cause: Step-Audio-EditX model config was missing 'tie_word_embeddings'
setting, so transformers 4.54+ defaulted to True and tied lm_head to embed_tokens
even though the checkpoint has different weights for each.
Solution: Add 'tie_word_embeddings': false to config.json to prevent tying.
This is much simpler and cleaner than the previous safetensors weight restoration
approach. One line in config.json fixes the issue for all transformers versions.
- Add step_audio_editx to PARAMETER_ENGINES for seed and temperature
- Fix engine node tooltips: parameters ARE used in clone mode (not hardcoded)
- Add reference implementation paths to implementation plan
- Update fails doc with latest lessons learned
Make both engines follow ChatterBox 23-Lang modular pattern:
**Step Audio EditX:**
- Update combine_audio_segments() to use ChunkCombiner with return_info=True
- Return (audio, chunk_info) tuple when requested
- Fix single segment shape handling (ensure 2D [channels, samples])
**IndexTTS:**
- Add combine_audio_segments() method using ChunkCombiner
- Update process_text() to accept return_info parameter
- Build chunk_info from text segments and return timing data
**Unified Node:**
- Call both engines with return_info=True
- Use ChunkTimingHelper.enhance_generation_info() for detailed reports
- Clean text without tags before calculating length
**Documentation:**
- Add generation_info consistency requirement to fails document
- Document correct pattern: ChunkCombiner + ChunkTimingHelper
- Note wrong pattern: inline torch.cat without timing
Now both engines provide detailed timing reports like F5-TTS and ChatterBox.
Models should only unload when explicitly requested (Clear VRAM button) or when
switching engines - NOT automatically after generation completes.
__del__ was causing models to unload immediately after wrapper instance goes
out of scope, preventing model reuse between generations.
Pattern: F5-TTS and ChatterBox don't have __del__, only Index2 and StepAudio had it.
Removed to match correct behavior.
- Configure character parser with available characters and language defaults like IndexTTS
- Add characters from text tags to available_characters to prevent fallback to "narrator"
- Add fallback voice mapping entries for characters without voice files
- Update fails document with character switching implementation lessons
Implemented Step Audio EditX integration with the following components:
Engine Implementation:
- Bundled Step Audio EditX reference code in step_audio_editx_impl/
- Created engine wrapper (step_audio_editx.py) with clone/edit methods
- Created adapter (step_audio_editx_adapter.py) for unified interface
- Created processor (step_audio_editx_processor.py) for TTS Suite integration
Model Management:
- Created downloader (step_audio_editx_downloader.py) for:
- Main Step-Audio-EditX model (3B LLM + CosyVoice vocoder)
- Step-Audio-Tokenizer files (linguistic_tokenizer.npy, speech_tokenizer_v1.onnx)
- FunASR-Paraformer model with correct config for bundled funasr_detach
- Fixed FunASR config to use bundled class names (ParaformerStreaming, WavFrontendOnline, SpecAugLFR)
- Added auto-download for all required models on first use
- Updated unified_model_interface.py to auto-download Step-Audio-EditX components
Node Configuration:
- Created Step Audio EditX engine config node
- Registered in nodes.py and unified TTS text node
Known Issues:
- LLM generation hangs during inference despite:
- torch.no_grad() wrapper
- torch.cuda.synchronize()
- Eager attention mode (Step1ForCausalLM doesn't support SDPA)
- Reduced max_new_tokens
- Removed logits processor
- Model loads successfully, all components initialized correctly
- Issue appears to be with transformers.generate() call itself
- Requires further investigation of Step1ForCausalLM compatibility
Files Modified:
- nodes.py: Added Step Audio EditX registration
- requirements.txt: Added FunASR dependencies
- utils/models/unified_model_interface.py: Added Step Audio EditX factory
- Various Step Audio EditX implementation files
Model Downloads:
- Step-Audio-EditX: 7GB (3B LLM + vocoder)
- FunASR-Paraformer: 881MB (from custom HF repo with fixed config)
- Step-Audio-Tokenizer: Bundled with main model download
Fix Italian ChatterBox case-sensitive language prefix detection
- Make [it] prefix detection case-insensitive to prevent double-prefixing
- Users can now use [it], [IT], [It] or any capitalization without issues
- Avoids duplication that caused 'IT' to be spoken in output
diodiogod