Add MOSS LoRA loading support and compatibility validation
Implementation details:
- Add managed MOSS LoRA discovery and Hugging Face install flow under models/TTS/moss_tts/loras
- Wire PEFT adapter loading through the MOSS engine, adapter, cache, generation info, and unified model interface
- Validate adapter base-model compatibility before loading to prevent huge PEFT mismatch errors
- Raise real ComfyUI popup errors for MOSS LoRA/base model mismatches instead of generic engine creation failures
- Patch bundled MOSS Delay and TTSD classes for PEFT generation compatibility
- Upgrade installer requirement handling so outdated packages like peft are upgraded automatically
- Document the MOSS LoRA folder contract and update MOSS metadata
Fix Echo-TTS character voice file-path fallback and compressed audio loading
Technical details:
- Preserve Character Voices and dropdown audio_path fallback in Echo-TTS voice mapping
- Allow Echo-TTS narrator and character fallback to use file paths when decoded tensors are unavailable
- Expand safe audio loading fallback order to support OGG and other compressed formats when torchaudio fails
Fix Python 3.13 numba and librosa compatibility handling
Technical details:
- Stop forcing NUMBA_DISABLE_JIT at startup on Python 3.13
- Use safe audio loading and resampling in IndexTTS-2 and ChatterBox Official 23-Lang
- Update audio hashing and fallback paths for newer numba stacks
Fix CosyVoice device mismatch in voice restoration
Technical implementation:
- Add CosyVoiceHandler.partially_load() for proper device reload
- Update base_wrapper.model_load() to use handler's partially_load()
- Move all CosyVoice components (llm, flow, hift, campplus) with nested modules
- Simplify CosyVoice engine wrapper device management (delegate to handler)
- Add CosyVoice variant setting for <restore> tags (RL/standard options)
- Update settings UI and API endpoint for variant selection
Fixes device tensor mismatch errors when TTS and VC restoration run sequentially
and prevents unnecessary model unload/reload when variants match.
Fixed 'dict' object has no attribute 'cpu' error when using Voice Designer
voices with TTS Text node.
## Root Cause
Voice Designer and Character Voices nodes output unified NARRATOR_VOICE format:
{
"audio": {"waveform": tensor, "sample_rate": 24000},
"audio_path": "...",
"reference_text": "...",
...
}
But TTS Text node was wrapping this AGAIN when creating voice_mapping:
{
"waveform": audio_tensor, # audio_tensor was already a dict!
"sample_rate": 24000,
"text": "..."
}
This created triple-nested structure that broke audio hashing.
## Fixes Applied
**TTS Text Node** (nodes/unified/tts_text_node.py):
- Extract actual waveform tensor from audio dict before creating voice_mapping
- Applied to all 6 places where audio_tensor is used:
- Narrator voice mapping (line 1151)
- Character ICL fallback (line 1183)
- Character x_vector fallback (line 1217)
- Character no-text fallback (line 1220)
- Character no-audio fallback (line 1237)
- Pattern: Check if audio_tensor is dict, extract waveform/sample_rate if so
**Qwen3-TTS Adapter** (engines/adapters/qwen3_tts_adapter.py):
- Extract inner audio dict before passing to hash function
- Changed from passing full voice_ref to passing voice_ref['audio']
**Audio Hash Utility** (utils/audio/audio_hash.py):
- Added nested structure handling with fallback
- Detects Character Voices format {"audio": {...}} vs direct format
- Enhanced debug logging to diagnose structure issues
## Testing
- Voice Designer → TTS Text: Works without hash error
- Character Voices → TTS Text: Works without hash error
- Audio hashing generates stable keys for caching
- Both VoiceDesign and traditional voices function correctly
Implements support for Fun-CosyVoice3-0.5B (standard) and Fun-CosyVoice3-0.5B-RL variants with lazy downloading:
- Only downloads variant-specific LLM file (~2GB) when needed
- Shared files (flow.pt, hift.pt, etc.) downloaded once and reused
- Both variants use same folder: Fun-CosyVoice3-0.5B
- Proper cache invalidation when switching between variants
Technical changes:
- Patched bundled CosyVoice library to accept llm_filename parameter
- Modified downloader to use SHARED_FILES + variant-specific LLM
- Audio cache now includes model_variant in hash calculation
- Model loading cache uses llm_filename in additional_params
- Added variant detection in path resolution and verification
Correcting test assertions for create_silence (returns 1D for mono)
Correcting test assertions for normalize_audio_tensor (keeps 1D as 1D)
Fixed production bug in pad_audio_to_duration - silence tensor dimensions didn't match input audio
Fixed two issues with paralinguistic tag insertion in edit post-processing:
1. Spacing: Changed logic to add space before/after tags when adjacent character
isn't already a space (previously only checked alphanumeric, causing tags to
stick to punctuation like "Hello!<Breathing>")
2. Position offset: Removed incorrect position offset tracking when inserting
multiple tags. Since tags are sorted descending, each insertion happens from
right to left and doesn't affect earlier positions.
This fixes text corruption when multiple paralinguistic tags are present
(e.g., "<Br <Breathing> eathing>") and ensures proper spacing around tags.
Fixed issue where EditPostProcessor was loading a second Step Audio EditX model (7GB+)
in VRAM during edit tag processing instead of reusing the pre-loaded engine from TTS generation.
Changes:
- Added pre_loaded_engine parameter passing through EditPostProcessor to Audio Editor
- Modified Audio Editor to wrap raw StepAudioTTS in StepAudioEditXEngine for edit_single() method
- Fixed paralinguistic code path to pass tts_engine_data instead of None
- Added debug logging to track pre-loaded engine usage
This eliminates unnecessary VRAM usage by ensuring only ONE Step Audio EditX model
is loaded during the complete TTS+editing workflow.
Eliminated duplicate model loading when EditPostProcessor applies inline edit tags:
1. **Pass pre-loaded engine directly**: EditPostProcessor now passes the TTS
processor's pre-loaded engine to the Audio Editor node via tts_engine_data dict
2. **Audio Editor engine reuse**: Added pre_loaded_engine parameter to
_get_or_create_engine() which returns pre-loaded engine directly if provided
3. **Fix ModelLoadConfig kwargs**: Fixed Audio Editor to pass torch_dtype and
quantization in additional_params dict instead of as direct kwargs (was
causing "unexpected keyword argument" error)
4. **Update EditPostProcessor**: Modified process_segments() and
_apply_edit_via_node() to properly pass pre_loaded_engine through the chain
This prevents loading a second 7GB+ Step Audio EditX model in VRAM during
edit post-processing. The Audio Editor now reuses the same engine instance
that was already loaded for TTS generation.
Fixes duplicate VRAM usage and "Loading Step Audio EditX engine" messages
during edit tag processing in SRT and regular TTS workflows.
Fixed CPU offloading for non-quantized (bfloat16/float16/float32) models:
- StepAudioTTS components (llm, cosy_model) don't have .to() method directly
- Handler now moves model.llm and model.cosy_model.cosy_impl individually
- Updates CosyVoice.device tracker when moving cosy_impl to CPU
- Properly handles both raw StepAudioTTS and wrapped models
Fixes unload behavior for non-quantized models to actually free VRAM
instead of silently failing with "no memory freed (model may already be on CPU)"
New Feature: <restore:N@M> Syntax
- <restore> - Restore using original pre-edit audio (1 VC pass)
- <restore:N> - N VC passes using original audio as reference
- <restore:N@M> - N VC passes using audio from iteration M as reference
Implementation:
- Store audio snapshots after each edit iteration
- Parse iteration reference from @ syntax in restore tags
- Use specified iteration audio as VC reference instead of original
- Allows preserving partial effects (e.g., keep whisper, remove laughter)
Use Case Example:
<style:whisper:2> <Laughter:3> <restore:1@2>
- Applies whisper (iterations 1-2)
- Applies laughter (iterations 3-5)
- Restores using iteration 2 (whispered audio) with 1 VC pass
- Result: Keeps whisper effect but removes laughter voice degradation
Documentation:
- Updated INLINE_EDIT_TAGS_USER_GUIDE.md with restore tag section
- Added examples for basic and iteration-reference restoration
- Explained use cases and when to use each syntax
- Updated processing order and reference sections
Position Tracking Fixes:
- Adjust tag positions after multiple spaces are collapsed to single space
- Track cumulative position shifts from space collapsing operations
- Add space-before logic when reinserting paralinguistic tags
- Ensure proper spacing on both sides of reinserted tags (support <Laughter> hahaha)
Display Improvements:
- Show clean transcript instead of raw tag syntax in edit post-processor
- Add suppress_progress parameter to Step Audio EditX Audio Editor node
- Suppress redundant "Edit iteration 1/1" messages when called from edit_post_processor
- Edit post-processor already shows clear iteration progress (→ Iteration X/Y)
Fixes issues with:
- Tags inserted at wrong positions when separate tags create multiple spaces
- Tags glued to words without spacing (support<Laughter> instead of support <Laughter>)
- Confusing duplicate iteration progress messages
- Refactor pause processing to return separate segments instead of combined audio
- Apply edit post-processing to individual segments after all TTS generation completes
- Fix position tracking for pipe-separated tags (e.g., <style:whisper:2|Laughter:5>)
- Adjust tag positions after leading whitespace stripping to maintain accuracy
- Fix space-after logic to only check tags from current match, not all previous tags
- Use clean text for TTS generation (strip edit tags before sending to model)
- Improve display to show clean transcript instead of raw tag syntax
- Add position offset tracking when inserting multiple paralinguistic tags
Resolves issues with:
- <restore> tag being sent to TTS model
- Paralinguistic tags inserted at wrong positions (e.g., h<Laughter> ahaha)
- Edit tags not being extracted properly in pause-segmented text
- Batch processing architecture for edit post-processing
Implements voice restoration feature using ChatterBox Official 23-Lang VC
to restore original voice resemblance after Step Audio EditX edits
(especially after multiple iterations of paralinguistic/emotion/style effects).
Changes:
- Add 'restore' tag parsing to step_audio_editx_special_tags.py
* Support <restore> and <restore:N> syntax for multi-pass VC
* Add restore to EDIT_TYPE_VALUES and sorting priority (runs last)
- Add voice restoration to edit_post_processor.py
* New _restore_voice_via_vc() function reusing UnifiedVoiceChangerNode
* Store PRE-EDIT audio as voice reference (no engine changes needed)
* Process restore tags after all other edits complete
* Fix importlib path for VC node loading
Benefits:
- Works across all 14 engines (7 text + 7 SRT) automatically
- No engine processor changes required
- Reuses existing VC cache system
- Uses pre-edit audio as target (clean, simple)
- Supports multi-pass refinement (1-5 iterations)
- Fully backward compatible
Processing order: emotion → style → speed → paralinguistic → restore
Resolved OOM errors during model switching by implementing proper VRAM clearing:
- Added real VRAM measurement using torch.cuda.memory_allocated() in model_manager
- Removed fake VRAM reporting from VibeVoice handler
- Removed class-level caching from VibeVoice engine (unified manager handles all caching)
- Centralized device movement through ensure_device() in model manager
- Models now move to CPU instead of deletion for fast reload from cache
- Added CUDA synchronization after clearing to ensure memory is actually freed
- Fixed recursion by adding _from_ensure_device flag to model_load()
- model_load() now calls ensure_device() first to clear other models before loading
Also fixed inline edit tag spacing issues:
- Tags now properly add space after paralinguistic tags when followed by alphanumeric text
- Fixed both parser and post-processor to handle spacing correctly
Major changes:
1. Post-processor now uses Audio Editor node instead of calling engine directly
- Inherits progress bars, caching, tested code
- Much simpler and more maintainable
2. Fixed unified model interface usage across Step Audio EditX
- Adapter now loads via unified_model_interface.load_model()
- Audio Editor node now loads via unified_model_interface.load_model()
- Both use model_type="tts" for consistent cache keys
- Prevents duplicate model loading (was loading twice before)
3. Fixed pause tag + edit tag interaction
- Pause tags split FIRST, then edit tags extracted per segment
- Allows: "Hello <Laughter> [pause:1] World <Sigh>"
- Laughter applies to "Hello", Sigh applies to "World"
4. Added lesson to fails document
- All engines must use unified interface in ALL code paths
- Cache key must match across adapters/nodes/processors
Bug fixes:
- Missing Dict import in post-processor
- Import path for Audio Editor node (use importlib)
- Progress bars now work during editing
Files modified:
- utils/audio/edit_post_processor.py - refactored to use Audio Editor node
- nodes/step_audio_editx/step_audio_editx_processor.py - pause handling
- engines/adapters/step_audio_editx_adapter.py - unified interface
- nodes/step_audio_editx_special/step_audio_editx_audio_editor_node.py - unified interface
- docs/Dev reports/fails_to_avoid_TTS_Engine_Implementation.md - added lesson
Added support for inline edit tags in TTS text that automatically apply
Step Audio EditX edits to generated audio from ANY engine.
Syntax examples:
- <Laughter:2> - paralinguistic with iterations
- <emotion:happy:1> - emotion edit
- <style:whisper:3> - style edit
- <speed:faster> - speed adjustment
- <Laughter:2|style:whisper> - multiple effects
Key changes:
- Extended step_audio_editx_special_tags.py with EditTag dataclass and iteration parsing
- Created edit_post_processor.py for centralized post-TTS edit application
- Integrated into Step Audio EditX processor with proper pause tag handling
- Pause tags split FIRST, then edit tags extracted per pause-segment
- Reuses already-loaded engine to avoid duplicate loading
Flow:
1. Parse text, split by pause tags first
2. Extract edit tags from each pause-delimited segment
3. Generate TTS with clean text (tags stripped)
4. Post-processor applies edits to segments with tags
5. Assemble final audio
Currently working for Step Audio EditX TTS and SRT nodes.
Add cache support using unified audio cache system:
- Create StepAudioEditXCacheKeyGenerator in cache.py (includes all relevant parameters)
- Register generator in AudioCache.__init__()
- Update _calculate_duration() to handle 24000 Hz sample rate
- Implement cache lookup in _generate_direct() before generation
- Cache results after successful generation with proper duration
Cache key includes: text, audio_component (stable hash), prompt_text, temperature,
do_sample, max_new_tokens, seed, model_path, device, torch_dtype, quantization, character
Now hits cache for unchanged segments across multiple generations, just like other engines.
Replace direct torchaudio.load() calls with safe_load_audio() utility in
AudioAnalyzer.load_audio() to handle PyTorch 2.9 TorchCodec DLL errors and
int16 normalization automatically.
Fixes 'Could not load libtorchcodec' error when using 🌊 Audio Wave Analyzer
node with Character Voices or direct audio input on PyTorch 2.9.0+cu128.
Move int16 to float [-1, 1] normalization from Character Voices node into
the safe_load_audio() utility function so ALL engines benefit from the fix.
Changes:
- safe_load_audio() now normalizes int16 values to [-1, 1] range automatically
- Handles PyTorch 2.9 torchaudio.load() behavior change and scipy fallback
- Remove duplicate normalization from Character Voices node
- F5-TTS now uses safe_load_audio() instead of torchaudio.load() directly
This fixes TorchCodec DLL errors on ALL engines that load audio files with PyTorch 2.9,
not just Character Voices. F5-TTS, ChatterBox, and future engines all benefit.
Critical fixes:
1. Added missing parameters to cache key: do_sample, num_beams, stream_return,
more_segment_before, use_torch_compile, use_accel
2. Fixed processor to pass do_sample, stream_return, more_segment_before to adapter
3. Added console logging for optimization and streaming parameters
This ensures:
- Cache properly invalidates when any parameter changes
- All new parameters flow through processor → adapter → engine
- User sees all configuration changes in console output
- Backward compatibility maintained (new params optional)
Removed torchcodec from requirements.txt and install.py to eliminate the FFmpeg
system dependency that was causing Windows installation failures.
Implementation:
- Removed torchcodec>=0.2.0 from requirements.txt
- Removed torchcodec>=0.2.0 from install.py core_packages list
- Removed torchaudio 2.9+ migration warnings (no longer applicable)
- Added AudioProcessingUtils.safe_load_audio() utility for robust audio loading
- Updated all audio loading calls to use fallback-aware utility function
Audio Loading Changes:
- character_voices_node.py: Uses safe_load_audio() with fallback support
- tts_text_node.py: Uses safe_load_audio() for narrator and character voices (2x)
- tts_srt_node.py: Uses safe_load_audio() for narrator voice
- higgs_audio_tts_processor.py: Uses safe_load_audio() (2x)
- higgs_audio_srt_processor.py: Uses safe_load_audio() (2x)
- index_tts_processor.py: Uses safe_load_audio() (2x)
- f5tts_streaming_adapter.py: Uses safe_load_audio()
- higgs_audio_adapter.py: Uses safe_load_audio()
- subprocess.py: Uses safe_load_audio()
Compatibility:
- torchaudio.load() works fine with fallback backends (soundfile, scipy)
- No quality loss - only negligible audio loading speed difference (~milliseconds)
- All existing audio formats supported
- Better error handling with graceful fallbacks
- _extract_audio_tensor now returns both original and mono versions
- Analysis (silence detection, energy, peaks) performed on MONO version only
- Visualization and graphs generated from MONO version
- Output audio (processed and segmented) uses ORIGINAL stereo format
- Timing regions found from mono are applied directly to stereo samples
- Stereo audio now passes through analyzer without quality loss
- Segmented audio preserves stereo channel information
Implemented full support for ChatterBox v2 with enhanced tokenization featuring
30+ special tokens for emotions, sounds, and vocal effects.
Key Features:
- Model version selection (v1/v2) in engine configuration node
- Auto-download of v2 model files (t3_mtl23ls_v2.safetensors, grapheme mappings)
- Special tags system using <emotion> syntax to avoid character switching conflicts
- Automatic conversion of <giggle>, <sigh>, <whisper> etc. to v2 format
- Cache invalidation between v1/v2 to prevent conflicts
- Full integration with character switching and pause tags
Technical Changes:
- Added model_version parameter throughout TTS and SRT processing chain
- Created chatterbox_v2_special_tags.py for tag conversion
- Updated cache keys to include model_version
- Version-aware model loading and validation
- Tag conversion happens after character/pause parsing, before engine
User can now write:
[Alice] Hello! <giggle> Nice to meet you. [pause:0.5] <whisper> Secret!
Documentation added in docs/CHATTERBOX_V2_SPECIAL_TOKENS.md
diodiogod