Fix CosyVoice device mismatch in voice restoration
Technical implementation:
- Add CosyVoiceHandler.partially_load() for proper device reload
- Update base_wrapper.model_load() to use handler's partially_load()
- Move all CosyVoice components (llm, flow, hift, campplus) with nested modules
- Simplify CosyVoice engine wrapper device management (delegate to handler)
- Add CosyVoice variant setting for <restore> tags (RL/standard options)
- Update settings UI and API endpoint for variant selection
Fixes device tensor mismatch errors when TTS and VC restoration run sequentially
and prevents unnecessary model unload/reload when variants match.
Fixed two issues with paralinguistic tag insertion in edit post-processing:
1. Spacing: Changed logic to add space before/after tags when adjacent character
isn't already a space (previously only checked alphanumeric, causing tags to
stick to punctuation like "Hello!<Breathing>")
2. Position offset: Removed incorrect position offset tracking when inserting
multiple tags. Since tags are sorted descending, each insertion happens from
right to left and doesn't affect earlier positions.
This fixes text corruption when multiple paralinguistic tags are present
(e.g., "<Br <Breathing> eathing>") and ensures proper spacing around tags.
Fixed issue where EditPostProcessor was loading a second Step Audio EditX model (7GB+)
in VRAM during edit tag processing instead of reusing the pre-loaded engine from TTS generation.
Changes:
- Added pre_loaded_engine parameter passing through EditPostProcessor to Audio Editor
- Modified Audio Editor to wrap raw StepAudioTTS in StepAudioEditXEngine for edit_single() method
- Fixed paralinguistic code path to pass tts_engine_data instead of None
- Added debug logging to track pre-loaded engine usage
This eliminates unnecessary VRAM usage by ensuring only ONE Step Audio EditX model
is loaded during the complete TTS+editing workflow.
Eliminated duplicate model loading when EditPostProcessor applies inline edit tags:
1. **Pass pre-loaded engine directly**: EditPostProcessor now passes the TTS
processor's pre-loaded engine to the Audio Editor node via tts_engine_data dict
2. **Audio Editor engine reuse**: Added pre_loaded_engine parameter to
_get_or_create_engine() which returns pre-loaded engine directly if provided
3. **Fix ModelLoadConfig kwargs**: Fixed Audio Editor to pass torch_dtype and
quantization in additional_params dict instead of as direct kwargs (was
causing "unexpected keyword argument" error)
4. **Update EditPostProcessor**: Modified process_segments() and
_apply_edit_via_node() to properly pass pre_loaded_engine through the chain
This prevents loading a second 7GB+ Step Audio EditX model in VRAM during
edit post-processing. The Audio Editor now reuses the same engine instance
that was already loaded for TTS generation.
Fixes duplicate VRAM usage and "Loading Step Audio EditX engine" messages
during edit tag processing in SRT and regular TTS workflows.
Fixed CPU offloading for non-quantized (bfloat16/float16/float32) models:
- StepAudioTTS components (llm, cosy_model) don't have .to() method directly
- Handler now moves model.llm and model.cosy_model.cosy_impl individually
- Updates CosyVoice.device tracker when moving cosy_impl to CPU
- Properly handles both raw StepAudioTTS and wrapped models
Fixes unload behavior for non-quantized models to actually free VRAM
instead of silently failing with "no memory freed (model may already be on CPU)"
New Feature: <restore:N@M> Syntax
- <restore> - Restore using original pre-edit audio (1 VC pass)
- <restore:N> - N VC passes using original audio as reference
- <restore:N@M> - N VC passes using audio from iteration M as reference
Implementation:
- Store audio snapshots after each edit iteration
- Parse iteration reference from @ syntax in restore tags
- Use specified iteration audio as VC reference instead of original
- Allows preserving partial effects (e.g., keep whisper, remove laughter)
Use Case Example:
<style:whisper:2> <Laughter:3> <restore:1@2>
- Applies whisper (iterations 1-2)
- Applies laughter (iterations 3-5)
- Restores using iteration 2 (whispered audio) with 1 VC pass
- Result: Keeps whisper effect but removes laughter voice degradation
Documentation:
- Updated INLINE_EDIT_TAGS_USER_GUIDE.md with restore tag section
- Added examples for basic and iteration-reference restoration
- Explained use cases and when to use each syntax
- Updated processing order and reference sections
Position Tracking Fixes:
- Adjust tag positions after multiple spaces are collapsed to single space
- Track cumulative position shifts from space collapsing operations
- Add space-before logic when reinserting paralinguistic tags
- Ensure proper spacing on both sides of reinserted tags (support <Laughter> hahaha)
Display Improvements:
- Show clean transcript instead of raw tag syntax in edit post-processor
- Add suppress_progress parameter to Step Audio EditX Audio Editor node
- Suppress redundant "Edit iteration 1/1" messages when called from edit_post_processor
- Edit post-processor already shows clear iteration progress (→ Iteration X/Y)
Fixes issues with:
- Tags inserted at wrong positions when separate tags create multiple spaces
- Tags glued to words without spacing (support<Laughter> instead of support <Laughter>)
- Confusing duplicate iteration progress messages
- Refactor pause processing to return separate segments instead of combined audio
- Apply edit post-processing to individual segments after all TTS generation completes
- Fix position tracking for pipe-separated tags (e.g., <style:whisper:2|Laughter:5>)
- Adjust tag positions after leading whitespace stripping to maintain accuracy
- Fix space-after logic to only check tags from current match, not all previous tags
- Use clean text for TTS generation (strip edit tags before sending to model)
- Improve display to show clean transcript instead of raw tag syntax
- Add position offset tracking when inserting multiple paralinguistic tags
Resolves issues with:
- <restore> tag being sent to TTS model
- Paralinguistic tags inserted at wrong positions (e.g., h<Laughter> ahaha)
- Edit tags not being extracted properly in pause-segmented text
- Batch processing architecture for edit post-processing
Implements voice restoration feature using ChatterBox Official 23-Lang VC
to restore original voice resemblance after Step Audio EditX edits
(especially after multiple iterations of paralinguistic/emotion/style effects).
Changes:
- Add 'restore' tag parsing to step_audio_editx_special_tags.py
* Support <restore> and <restore:N> syntax for multi-pass VC
* Add restore to EDIT_TYPE_VALUES and sorting priority (runs last)
- Add voice restoration to edit_post_processor.py
* New _restore_voice_via_vc() function reusing UnifiedVoiceChangerNode
* Store PRE-EDIT audio as voice reference (no engine changes needed)
* Process restore tags after all other edits complete
* Fix importlib path for VC node loading
Benefits:
- Works across all 14 engines (7 text + 7 SRT) automatically
- No engine processor changes required
- Reuses existing VC cache system
- Uses pre-edit audio as target (clean, simple)
- Supports multi-pass refinement (1-5 iterations)
- Fully backward compatible
Processing order: emotion → style → speed → paralinguistic → restore
Resolved OOM errors during model switching by implementing proper VRAM clearing:
- Added real VRAM measurement using torch.cuda.memory_allocated() in model_manager
- Removed fake VRAM reporting from VibeVoice handler
- Removed class-level caching from VibeVoice engine (unified manager handles all caching)
- Centralized device movement through ensure_device() in model manager
- Models now move to CPU instead of deletion for fast reload from cache
- Added CUDA synchronization after clearing to ensure memory is actually freed
- Fixed recursion by adding _from_ensure_device flag to model_load()
- model_load() now calls ensure_device() first to clear other models before loading
Also fixed inline edit tag spacing issues:
- Tags now properly add space after paralinguistic tags when followed by alphanumeric text
- Fixed both parser and post-processor to handle spacing correctly
Major changes:
1. Post-processor now uses Audio Editor node instead of calling engine directly
- Inherits progress bars, caching, tested code
- Much simpler and more maintainable
2. Fixed unified model interface usage across Step Audio EditX
- Adapter now loads via unified_model_interface.load_model()
- Audio Editor node now loads via unified_model_interface.load_model()
- Both use model_type="tts" for consistent cache keys
- Prevents duplicate model loading (was loading twice before)
3. Fixed pause tag + edit tag interaction
- Pause tags split FIRST, then edit tags extracted per segment
- Allows: "Hello <Laughter> [pause:1] World <Sigh>"
- Laughter applies to "Hello", Sigh applies to "World"
4. Added lesson to fails document
- All engines must use unified interface in ALL code paths
- Cache key must match across adapters/nodes/processors
Bug fixes:
- Missing Dict import in post-processor
- Import path for Audio Editor node (use importlib)
- Progress bars now work during editing
Files modified:
- utils/audio/edit_post_processor.py - refactored to use Audio Editor node
- nodes/step_audio_editx/step_audio_editx_processor.py - pause handling
- engines/adapters/step_audio_editx_adapter.py - unified interface
- nodes/step_audio_editx_special/step_audio_editx_audio_editor_node.py - unified interface
- docs/Dev reports/fails_to_avoid_TTS_Engine_Implementation.md - added lesson
Added support for inline edit tags in TTS text that automatically apply
Step Audio EditX edits to generated audio from ANY engine.
Syntax examples:
- <Laughter:2> - paralinguistic with iterations
- <emotion:happy:1> - emotion edit
- <style:whisper:3> - style edit
- <speed:faster> - speed adjustment
- <Laughter:2|style:whisper> - multiple effects
Key changes:
- Extended step_audio_editx_special_tags.py with EditTag dataclass and iteration parsing
- Created edit_post_processor.py for centralized post-TTS edit application
- Integrated into Step Audio EditX processor with proper pause tag handling
- Pause tags split FIRST, then edit tags extracted per pause-segment
- Reuses already-loaded engine to avoid duplicate loading
Flow:
1. Parse text, split by pause tags first
2. Extract edit tags from each pause-delimited segment
3. Generate TTS with clean text (tags stripped)
4. Post-processor applies edits to segments with tags
5. Assemble final audio
Currently working for Step Audio EditX TTS and SRT nodes.
drphero