Implemented per-segment parameter overrides for seed and speed:
- Register CosyVoice in segment_parameters.py for seed and speed
- Apply segment.parameters using modular apply_segment_parameters()
- Extract segment_seed and segment_speed with fallback to node defaults
- Pass parameters through all generation paths (pause tags, chunking, direct)
- Add speed parameter to _generate_audio_segment() method
- Print parameter application message matching other engines
Supports inline tags like [Alice|seed:42|speed:1.5] with automatic
warnings for unsupported parameters via modular validation system.
- Prepend CosyVoice native language tags (<|en|>, <|zh|>, <|ja|>, <|ko|>) based on segment language
- Integrate with unified language system using resolve_language_alias()
- Support both explicit [lang:] tags and character language defaults
- Validate manual language tags to avoid duplicates
- Only prepend if no valid CosyVoice language tag already present
Swap tag conversion order to prevent conflicts:
- Single tags converted first: <laughter> → [laughter]
- Wrapper tags converted last: <laughing>text</laughing> → <laughter>text</laughter>
This prevents the single tag pattern from breaking wrapper conversions.
Fixed issues:
1. Special tags were being stripped by text_frontend normalization
2. No differentiation between single tags and wrapper tags
Solutions:
- Auto-detect special tags in text and set text_frontend=False to preserve them
- Added wrapper tag conversion: <laughing>text</laughing> → <laughter>text</laughter>
- Improved single tag pattern to avoid matching wrapper opening tags
- Added text display with separators like Step Audio EditX
Tag support:
- Single tags: <cough>, <breath>, <laughter>, etc. → [cough], [breath], [laughter]
- Wrapper tags: <laughing>text</laughing> → <laughter>text</laughter>
- Emphasis: <strong>text</strong> (already correct format, unchanged)
- Language: <|en|>, <|zh|>, <|ja|>, <|ko|> (detected and preserved)
Files modified:
- engines/cosyvoice/cosyvoice.py: Tag detection, text_frontend control, text display
- utils/text/cosyvoice_special_tags.py: Improved tag conversion logic
Example output:
📝 CosyVoice3 - Generating zero_shot:
============================================================
On Tuesdays, the pigeons held parliament. [cough] They debated...
============================================================
⚙️ text_frontend=False (special tags detected)
- Implement auto-mode detection (zero_shot/instruct/cross_lingual)
- Remove mode dropdown and reference_text from engine node
- Fix sample rate from 22050 Hz to 24000 Hz across all files
- Add paralinguistic tag conversion (<tag> to [tag]) via cosyvoice_special_tags.py
- Clean up requirements.txt (remove 13 unnecessary deps, keep only inflect)
- Update engine node tooltip with clear mode mutual exclusivity warning
All dependencies now properly handled by install.py except inflect (needed for text normalization)
- Filter engine node dropdown to only show valid Step Audio EditX models (those with CosyVoice vocoder)
- Prevents internal components like FunASR-Paraformer from appearing in model selection
- Add proper 'local:' prefix handling in adapter to resolve model paths correctly
- Respects extra_model_paths.yaml configuration for yaml-configured paths
- Fixes error when selecting 'local:Step-Audio-EditX' from dropdown
New Feature: <restore:N@M> Syntax
- <restore> - Restore using original pre-edit audio (1 VC pass)
- <restore:N> - N VC passes using original audio as reference
- <restore:N@M> - N VC passes using audio from iteration M as reference
Implementation:
- Store audio snapshots after each edit iteration
- Parse iteration reference from @ syntax in restore tags
- Use specified iteration audio as VC reference instead of original
- Allows preserving partial effects (e.g., keep whisper, remove laughter)
Use Case Example:
<style:whisper:2> <Laughter:3> <restore:1@2>
- Applies whisper (iterations 1-2)
- Applies laughter (iterations 3-5)
- Restores using iteration 2 (whispered audio) with 1 VC pass
- Result: Keeps whisper effect but removes laughter voice degradation
Documentation:
- Updated INLINE_EDIT_TAGS_USER_GUIDE.md with restore tag section
- Added examples for basic and iteration-reference restoration
- Explained use cases and when to use each syntax
- Updated processing order and reference sections
Position Tracking Fixes:
- Adjust tag positions after multiple spaces are collapsed to single space
- Track cumulative position shifts from space collapsing operations
- Add space-before logic when reinserting paralinguistic tags
- Ensure proper spacing on both sides of reinserted tags (support <Laughter> hahaha)
Display Improvements:
- Show clean transcript instead of raw tag syntax in edit post-processor
- Add suppress_progress parameter to Step Audio EditX Audio Editor node
- Suppress redundant "Edit iteration 1/1" messages when called from edit_post_processor
- Edit post-processor already shows clear iteration progress (→ Iteration X/Y)
Fixes issues with:
- Tags inserted at wrong positions when separate tags create multiple spaces
- Tags glued to words without spacing (support<Laughter> instead of support <Laughter>)
- Confusing duplicate iteration progress messages
- Refactor pause processing to return separate segments instead of combined audio
- Apply edit post-processing to individual segments after all TTS generation completes
- Fix position tracking for pipe-separated tags (e.g., <style:whisper:2|Laughter:5>)
- Adjust tag positions after leading whitespace stripping to maintain accuracy
- Fix space-after logic to only check tags from current match, not all previous tags
- Use clean text for TTS generation (strip edit tags before sending to model)
- Improve display to show clean transcript instead of raw tag syntax
- Add position offset tracking when inserting multiple paralinguistic tags
Resolves issues with:
- <restore> tag being sent to TTS model
- Paralinguistic tags inserted at wrong positions (e.g., h<Laughter> ahaha)
- Edit tags not being extracted properly in pause-segmented text
- Batch processing architecture for edit post-processing
Implements voice restoration feature using ChatterBox Official 23-Lang VC
to restore original voice resemblance after Step Audio EditX edits
(especially after multiple iterations of paralinguistic/emotion/style effects).
Changes:
- Add 'restore' tag parsing to step_audio_editx_special_tags.py
* Support <restore> and <restore:N> syntax for multi-pass VC
* Add restore to EDIT_TYPE_VALUES and sorting priority (runs last)
- Add voice restoration to edit_post_processor.py
* New _restore_voice_via_vc() function reusing UnifiedVoiceChangerNode
* Store PRE-EDIT audio as voice reference (no engine changes needed)
* Process restore tags after all other edits complete
* Fix importlib path for VC node loading
Benefits:
- Works across all 14 engines (7 text + 7 SRT) automatically
- No engine processor changes required
- Reuses existing VC cache system
- Uses pre-edit audio as target (clean, simple)
- Supports multi-pass refinement (1-5 iterations)
- Fully backward compatible
Processing order: emotion → style → speed → paralinguistic → restore
- Character parser now skips ChatterBox v2 special tokens ([laughter], [giggle], etc)
- These tokens are TTS control tags, not character names
- Warns if user has a character with same name as v2 token
- Fix warning system to trigger for all conflicting tags (not just when both formats present)
- Remove debug print from model manager ensure_device
Resolved OOM errors during model switching by implementing proper VRAM clearing:
- Added real VRAM measurement using torch.cuda.memory_allocated() in model_manager
- Removed fake VRAM reporting from VibeVoice handler
- Removed class-level caching from VibeVoice engine (unified manager handles all caching)
- Centralized device movement through ensure_device() in model manager
- Models now move to CPU instead of deletion for fast reload from cache
- Added CUDA synchronization after clearing to ensure memory is actually freed
- Fixed recursion by adding _from_ensure_device flag to model_load()
- model_load() now calls ensure_device() first to clear other models before loading
Also fixed inline edit tag spacing issues:
- Tags now properly add space after paralinguistic tags when followed by alphanumeric text
- Fixed both parser and post-processor to handle spacing correctly
ModelLoadConfig parameter fixes:
- Use additional_params dict for torch_dtype and quantization (not direct kwargs)
- Fixed in both adapter and Audio Editor node
- Added model_path resolution to default ComfyUI models directory
Factory fixes:
- Return StepAudioEditXEngine wrapper instead of StepAudioTTS
- Wrapper provides get_sample_rate() and other methods needed by nodes
- Maintains compatibility with adapter and Audio Editor node
Adapter fixes:
- Handle both tuple and tensor returns from engine.clone()
- Inner engine returns (audio, sample_rate), wrapper returns just audio
Paralinguistic tag spacing:
- Auto-insert space before paralinguistic tags when glued to words
- 'whispering<Laughter>' becomes 'whispering [Laughter]'
- Required for Step Audio EditX paralinguistic insertion to work correctly
Added support for inline edit tags in TTS text that automatically apply
Step Audio EditX edits to generated audio from ANY engine.
Syntax examples:
- <Laughter:2> - paralinguistic with iterations
- <emotion:happy:1> - emotion edit
- <style:whisper:3> - style edit
- <speed:faster> - speed adjustment
- <Laughter:2|style:whisper> - multiple effects
Key changes:
- Extended step_audio_editx_special_tags.py with EditTag dataclass and iteration parsing
- Created edit_post_processor.py for centralized post-TTS edit application
- Integrated into Step Audio EditX processor with proper pause tag handling
- Pause tags split FIRST, then edit tags extracted per pause-segment
- Reuses already-loaded engine to avoid duplicate loading
Flow:
1. Parse text, split by pause tags first
2. Extract edit tags from each pause-delimited segment
3. Generate TTS with clean text (tags stripped)
4. Post-processor applies edits to segments with tags
5. Assemble final audio
Currently working for Step Audio EditX TTS and SRT nodes.
- New dedicated Audio Editor node with 6 edit types: emotion (14), style (32), speed (4), paralinguistic (10), denoise, vad
- Smart paralinguistic text handling: user writes <Laughter> tags, auto-converts to [Laughter] for engine
- Derives clean transcript automatically by stripping tags from input text
- Iteration caching: try 3 passes, then 2, then 1 without re-running (like Voice Changer)
- Token generation progress bar during each edit iteration
- Normalizes newlines/whitespace in text to prevent model parsing issues
- Graceful error when wrong engine type connected (guides user to use Step Audio EditX Engine)
- New special tags registry at utils/text/step_audio_editx_special_tags.py
- Add step_audio_editx to PARAMETER_ENGINES for seed and temperature
- Fix engine node tooltips: parameters ARE used in clone mode (not hardcoded)
- Add reference implementation paths to implementation plan
- Update fails doc with latest lessons learned
Fix Italian ChatterBox case-sensitive language prefix detection
- Make [it] prefix detection case-insensitive to prevent double-prefixing
- Users can now use [it], [IT], [It] or any capitalization without issues
- Avoids duplication that caused 'IT' to be spoken in output
FIXES: Parameter-only tags like [seed:4|exag:1.0] were incorrectly using aliased narrator voice (David) instead of main narrator voice (Tony)
CAUSE: When parsing parameter-only tags, the character parser:
1. Defaulted character to 'narrator'
2. Detected segment_parameters exist
3. But only skipped alias resolution for language-only tags like [de:] or [it:]
4. Result: 'narrator' was normalized through alias system to 'david_attenborough cc3'
SOLUTION: Also skip alias resolution for parameter-only tags by detecting:
- raw_character == default_character (narrator)
- segment_parameters exist (has parameters)
- explicit_language is None (no language specification)
This ensures parameter-only tags use the main narrator voice (Tony from opt_narrator) not the aliased narrator (David)
FIXES: When switching from v2 to v1 model, audio cache was not being properly invalidated, causing v2-generated audio to be returned for v1 requests
KEY CHANGES:
1. Extract model_version from config in SRT processor (defaults to v2)
2. Pass model_version to all generate_speech() calls in parameter_switching and simple subtitle paths
3. This ensures cache keys differ between v1 and v2 models, preventing cache misses
ALSO INCLUDES:
- Refactored lang-23 SRT processor to use two-pass architecture matching regular ChatterBox
- First pass: Analyze and categorize all subtitles
- Second pass: Route to appropriate handler (MultilingualEngine for complex, direct for simple)
- Fixed character voice selection to use proper alias mapping via voice_refs
- Use original_character in regular ChatterBox to preserve character identity
This ensures parameter-only tags like [seed:4] correctly use narrator voice (Tony) instead of aliased narrator (David)
- Consolidate character voice info into block generation messages
- Fix tokenizer path structure to match actual storage location (vibevoice/ instead of VibeVoice/tokenizer/)
- Use consistent path for tokenizer downloads so they're found on subsequent runs
- Comment out non-error device_map debug messages
- Remove duplicate SINGLE SEGMENT text printing
- Fix logged_characters to only log once per character (not per priority check)
- New utils/text/segment_parameters.py with parameter validation, aliases, and engine compatibility
- Extended CharacterSegment to include per-segment parameters
- Extended language resolver to parse pipe-separated parameters: [Alice|seed:42|temp:0.5]
- Support for: seed, temperature, cfg, speed, top_p, top_k, inference_steps, exaggeration, emotion_alpha
- Case-insensitive aliases and order-independent syntax
- Integrated into VibeVoice Custom Character Switching mode (text and SRT)
- Fixed grouping logic to break groups when parameters change
- Added PARAMETER_SWITCHING_GUIDE.md documentation
- Updated PROJECT_INDEX.md
NOTE: Currently implemented for VibeVoice Custom Character Switching only.
Native Multi-Speaker mode and other engines will be added in future updates.
Implemented full support for ChatterBox v2 with enhanced tokenization featuring
30+ special tokens for emotions, sounds, and vocal effects.
Key Features:
- Model version selection (v1/v2) in engine configuration node
- Auto-download of v2 model files (t3_mtl23ls_v2.safetensors, grapheme mappings)
- Special tags system using <emotion> syntax to avoid character switching conflicts
- Automatic conversion of <giggle>, <sigh>, <whisper> etc. to v2 format
- Cache invalidation between v1/v2 to prevent conflicts
- Full integration with character switching and pause tags
Technical Changes:
- Added model_version parameter throughout TTS and SRT processing chain
- Created chatterbox_v2_special_tags.py for tag conversion
- Updated cache keys to include model_version
- Version-aware model loading and validation
- Tag conversion happens after character/pause parsing, before engine
User can now write:
[Alice] Hello! <giggle> Nice to meet you. [pause:0.5] <whisper> Secret!
Documentation added in docs/CHATTERBOX_V2_SPECIAL_TOKENS.md
For compatibility with wildminder's VibeVoice implementation, added support
for numeric character tags [1] [2] [3] [4] in native multi-speaker mode.
- Numeric characters [1]-[4] now map directly to Speaker 1-4 voice inputs
- Works alongside existing character tag to speaker mapping system
- Mixed usage supported: [1] [2] [Alice] maps to Speakers 1, 2, 3
- Character parser unchanged - handles all existing tags normally
- VibeVoice-specific enhancement in native multi-speaker mode only
This provides users a convenient shorthand format while maintaining
full backward compatibility with existing character tag systems.
The old auto-phonemization system is now disabled in favor of the
dedicated 📝 Phoneme Text Normalizer node which gives users full control.
Changes:
- Modified should_use_phonemization() to always return False
- Added deprecation notice explaining the replacement
- Users now have explicit control via the Phoneme Text Normalizer node
This prevents unexpected automatic phonemization and respects the
auto_phonemization=False setting properly.
Fix ChatterBox Official 23-Lang language resolution and narrator input
Technical details:
- Fix resolve_language_alias AttributeError across multiple engines
- Fix ChatterBox Official 23-Lang not using opt_narrator audio input
- Fix phonemizer_utils language alias resolution method call
- Fix character_parser validation module language resolution
- Add centralized emotion resolution to character parser
- Fix QwenEmotion variable scope error in IndexTTS-2 inference
- Refactor emotion resolution to avoid code duplication
- Enable both connected engine emotions and character-specific emotion overrides
- Support [Alice:Bob] syntax for voice identity with emotion reference
- Clean integration with existing character alias system
Add delegation method to CharacterParser that calls the language resolver's
set_engine_aware_default_language method. This fixes the AttributeError in
F5-TTS engine when trying to set engine-aware default language.
Major Features:
• IndexTTS-2 engine with zero-shot voice cloning and emotion control
• Multi-modal emotion support (audio refs, text descriptions, manual vectors)
• Character switching with emotion references: [Alice:angry_bob]
• Native IndexTTS-2 token-based chunking integration
• Comprehensive parameter exposure with 25+ engine-specific options
Core Components:
• engines/processors/index_tts_processor.py - Main processor with emotion support
• nodes/engines/index_tts_engine_node.py - Engine configuration node
• Updated unified TTS Text node with IndexTTS-2 integration
• Node registration in main nodes.py
Dependencies:
• Added 8 safe IndexTTS-2 dependencies to install.py and requirements.txt
• Dependency testing confirmed only descript-audiotools conflicts (already handled)
• All new deps use flexible versioning (>=) to avoid lock conflicts
Character Parser Fix:
• Fixed circular import by moving CharacterSegment to separate types.py
• Maintains backward compatibility while supporting emotion references
• Modular structure preserved (4 modules under 500 lines each)
Technical Details:
• Emotion disentanglement separates speaker identity from emotional expression
• Supports [character:emotion] syntax for per-segment emotion control
• IndexTTS-2 native chunking overrides character-based chunking when enabled
• Full caching integration and pause tag support included
• Error handling with graceful fallbacks for missing voices/emotions
MAJOR: IndexTTS-2 TTS engine integration
- Core IndexTTS engine with unified model interface integration
- Auto-downloader for HuggingFace models + GitHub source code
- Engine adapter with emotion support [Alice:angry_bob] syntax
- Cache integration for IndexTTS-2 parameters
- Factory registration for proper model management
MAJOR: Character parser modularization (930 → 300 lines each)
- Split into base_parser.py, language_resolver.py, segment_processor.py, validation.py
- Added emotion support with backward compatibility
- New split_by_character_with_emotions() method for advanced engines
- Flexible tag parsing: [de:Alice:angry_bob] support
- Language disambiguation to avoid Alice vs language code conflicts
- Full backward compatibility maintained
FEATURES:
- IndexTTS-2 emotion disentanglement (speaker vs emotion separation)
- Duration control and precise voice cloning
- Multi-modal emotion control (audio refs, text, vectors)
- Character switching with emotion references
- All official IndexTTS-2 parameters exposed
DOCS: Implementation plan and architectural TODO items added
- Add 'it' to models_to_skip list alongside PT-BR
- Update debug message to reflect both Portuguese and Italian models
- Update PROJECT_INDEX.md with phonemizer utilities documentation
Italian F5-TTS models work better with native text rather than IPA phonemization,
similar to Portuguese-Brazil models.
Fix F5-TTS auto_phonemization toggle not working
Technical details:
- Fix parameter missing from unified TTS text node call
- Add auto_phonemization parameter to F5TTS API chain
- Fix F5TTS wrapper not passing parameter to inference
- Add parameter to infer_process and infer_batch_process
- Fix cache invalidation with proper parameter flow
- Remove debug messages when phonemization disabled
Fix ChatterBox Official 23-Lang safetensors support
Technical implementation:
- Add smart loading logic preferring official .pt files
- Support user safetensors with strict=False compatibility mode
- Implement download fallback from .pt to safetensors format
- Add proper missing file detection and re-download logic
- Fix file format validation in download checker
Fix F5-TTS multilingual phonemization system
Technical implementation:
- Add auto_phonemization UI toggle to F5-TTS Engine node
- Implement smart phonemization detection with user control
- Disable phonemization for F5-PT-BR (confirmed to worsen quality)
- Add experimental warnings for multilingual model testing
- Ensure phonemization flows through SRT and regular TTS workflows
- Add cache separation for different phonemization settings
- Add debug language_id parameter display for TTS generation calls
- Clean up verbose console messages (keep only model path and errors)
- Add missing 12 language aliases to character parser for full 23-language support:
Arabic, Danish, Greek, Finnish, Hebrew, Malay, Polish, Swedish, Swahili, Turkish
- Fix character parser language normalization for ChatterBox Official 23-Lang
- Add full SRT subtitle support via unified SRT node integration
- Fix cache collision between ChatterBox and ChatterBox Official 23-Lang engines
- All 23 languages now work with both tagged and untagged text
- Language switching and voice references fully functional
- SRT timing modes, batch processing, and multilingual subtitles supported
- Updated character parser regex to exclude standalone [it]/[italian] tags while preserving [it:]/[italian:] syntax
- Added apply_italian_prefix_if_needed() method for automatic [it] prefix application when Italian language is detected
- Extended character parser with split_by_character_with_language_and_explicit_flag() to preserve explicit language flags
- Updated ChatterBox TTS and SRT nodes to use new parser method with automatic Italian prefix handling
- Fixed timing report text to show processed segments instead of raw input with character tags
- Added console debug prints to show final text sent to TTS models for verification
Italian ChatterBox model now correctly receives [it] prefix for Italian text when:
- User explicitly uses [it:Alice] or [italian:Bob] character syntax
- User uses [it:] or [italian:] language switching syntax
- Character is mapped to Italian in alias system
- Does NOT apply prefix when just using Italian model globally (preserves bilingual capability)
Add manual Speaker N: format and priority system to VibeVoice
Technical implementation:
- Add support for manual 'Speaker 1: Speaker 2:' format in Native Multi-Speaker mode
- Implement priority system where speaker inputs override character aliases
- Add proper Speaker 1 mapping to main narrator voice (Tony)
- Fix voice resolution for manual Speaker format with debug logging
- Update character parser to handle Speaker N: format detection
- Enhance tooltips to clearly explain both format options
Major fixes for VibeVoice engine functionality:
Audio Quality Improvements:
- Replace linear interpolation resampling with librosa.resample (matches official)
- Fix normalization to use -25 dB FS RMS instead of peak normalization (matches official)
- Implement proper stereo-to-mono conversion with channel averaging (matches official)
Multi-Character Voice Switching:
- Add proper character mapping using get_character_mapping() like F5-TTS does
- Build complete voice_mapping with fallbacks for all parsed characters
- Fix character parser to auto-discover characters when available_characters is None
- Remove unnecessary fallback logic from processor (handled at wrapper level)
Pause Tag Support:
- Add modular pause tag processing like F5-TTS using generate_vibevoice_with_pause_tags()
- Integrate PauseTagProcessor for [pause:X] support in VibeVoice
- Follow established modular pattern from F5-TTS and ChatterBox implementations
Cache System Improvements:
- Fix floating point precision issues in cache key generation (round to 3 decimals)
- Add proper parameter propagation to cached engine instances
- Ensure cache keys include all generation parameters properly
All changes tested and working - multi-character switching now works correctly
and pause tags are processed properly using the modular system.
- Language-only tags like [fr:] should use selected narrator voice (dropdown/opt_narrator)
- Previously, "narrator" from [fr:] was resolved via character alias to "david_attenborough cc3"
- Added skip_narrator_alias parameter to normalize_character_name method
- Detect language-only tags and skip alias resolution for narrator character
- Preserves narrator voice priority: opt_narrator > dropdown > character map
MAJOR FIXES:
- Fix character alias system overriding user voice selection for language-only tags
- Fix multilingual engine loading wrong tokenizer (English) for all languages
- Fix subtitle processing using wrong model after multilingual segment processing
- Preserve original character info to detect language-only tags vs explicit character tags
TECHNICAL DETAILS:
- Add original_character field to CharacterSegment to track pre-alias resolution names
- Fix multilingual engine to load correct language model first (not default English)
- Add model restoration logic after multilingual processing in traditional SRT
- Improve voice selection messages to show when using main voice vs character voice
- Fix cache coverage method to handle new 5-tuple segment data format
This resolves the regression where German/Norwegian segments sounded English
despite logs showing correct models loading. The issue was the multilingual engine
loading English tokenizer first, then caching it for all subsequent languages.
Root cause: WorkItem dataclass missing critical fields causing streaming workers to use hardcoded defaults instead of user parameters.
Key fixes:
- Add missing fields to WorkItem: seed, enable_audio_cache, crash_protection_template, device
- Update worker inputs to use WorkItem fields instead of hardcoded values
- Fix WorkItem creation to populate all fields from original inputs
- Remove local: prefix handling in language_mapper for consistency
- Add comprehensive language normalization in character_parser
- Comment out verbose cache debug logs while keeping success messages
Now both traditional (batch_size=0) and streaming (batch_size=3) modes generate identical cache keys with same parameters, ensuring proper cache hits and avoiding duplicate processing.
Streaming parallelization system now fully functional with cache consistency.
- Language mapper now passes through local models (e.g., 'local:German') directly
- Character parser handles ChatterBox language codes properly vs F5-TTS model names
- This should fix narrator segments using correct main language selection
diodiogod