nodes/diodiogod-TTS-Audio-Suite
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
codex/audio-cpp-integration
diodiogod-TTS-Audio-Suite/docs/Upstream_Engine_Tracking/INDEXTTS2_UPDATE_LOG.md
T
diodiogod 2bc4c2a18d Add IndexTTS 2.5 support
2026-08-11 23:08:35 -03:00

15 KiB
Raw Permalink Blame History

IndexTTS-2 Updates Log

This document tracks updates applied to our bundled IndexTTS-2 code from the upstream repository.

Update Process

  1. Check for upstream changes: cd IgnoredForGitHubDocs/For_reference/index_tts && git pull
  2. Review changes: git diff OLD_COMMIT..NEW_COMMIT -- indextts/infer_v2.py
  3. Apply relevant changes to engines/index_tts/indextts/infer_v2.py
  4. Test functionality with our IndexTTS-2 nodes
  5. Document changes in this log

2026-08-11: IndexTTS-2.5 Version Integration

Official sources: index-tts/index-tts commit b5ea881bec284b72f0b1cc04e0a724ff0c6b93e9; model snapshot ba2480d9f7f629eb18f6acaebb357679d9ba88a4

Changes applied

  • Added IndexTTS-2.5 as a selectable version of the existing index_tts engine.
  • Bundled the official 25 Hz semantic codec, multilingual tokenizer, Japanese G2P, and NeMo normalization bridge.
  • Preserved suite dual-source audio plus vector/text emotion blending.
  • Added Chinese, English, Japanese, Spanish, and Arabic conditioning.
  • Added the official 2.5-only duration_factor, documented honestly as nearest-neighbor internal semantic-feature scaling rather than natural prosody or exact-duration planning.
  • Deliberately excluded IndexTTS-2.5 from TTS SRT's native-duration option; the suite-owned exact-seconds extrapolation was removed after source and listening review.
  • Kept legacy IndexTTS-2 checkpoints, FP16 loading, MaskGCT, workflows, and node identity intact.
  • Pinned the audited Hugging Face model revision and retained the main Transformers 5 environment.
  • Added model-aware Text/SRT processor and audio-cache identities so switching 2.0/2.5 or a 2.5 generation parameter cannot reuse stale output.
  • Documented the suite's manual finding that 2.5 is not a universal cloning-quality upgrade: 2.0 may retain speaker resemblance better under strong different-speaker emotion transfer.

Validation status

  • Python compilation
  • Bundled backend import under TTS_SUITE_TEST_VENV_PYTHON
  • Full checkpoint download and live ComfyUI generation
  • Manual audio-quality review of the official duration factor and 2.0/2.5 speaker resemblance
  • Live 2.5 → 2.0 model switching after processor-cache invalidation fix

2025-09-18: Major Update - Cache & Emotion Improvements

Reference commit range: 8336824..64cb31a (September 11 → September 18, 2025) Upstream commits reviewed: 5 commits affecting infer_v2.py

🔧 Changes Applied:

1. Emotion Vector Normalization Function (commit: 8aa8064)

  • Added: normalize_emo_vec() helper function
  • Purpose: Prevents voice identity loss from overly strong emotion settings
  • Implementation:
    • Applies emotion bias factors to reduce problematic emotions
    • Caps total emotion sum to 0.8 maximum
    • Protects against extreme emotion vector values
  • User benefit: Better emotion control without losing speaker characteristics

2. Audio Length Limiting (commit: 0828dcb)

  • Added: _load_and_cut_audio() helper function
  • Purpose: Prevents memory/VRAM issues from overly long reference audio
  • Implementation:
    • Automatically truncates audio to 15 seconds maximum
    • Supports different sample rates (16kHz, 22kHz)
    • Provides verbose logging of truncation
  • User benefit: More stable generation with long reference audio files

3. Persistent Cache Buildup Fix (commit: 64cb31a)

  • Added: Proper cache clearing with torch.cuda.empty_cache()
  • Purpose: Solves memory accumulation issues during multiple generations
  • Implementation:
    • Clears old cache variables before loading new audio
    • Calls torch.cuda.empty_cache() to free GPU memory
    • Applied to both speaker and emotion cache invalidation
  • User benefit: Better memory management during batch processing

4. BigVGAN CUDA Kernel Import Fix (commits: ee23371, e409c4a)

  • Fixed: Corrected import path for BigVGAN custom CUDA kernel
  • Changed: indextts.BigVGAN.alias_free_activation.cuda → indextts.s2mel.modules.bigvgan.alias_free_activation.cuda
  • Purpose: Ensures proper CUDA acceleration when available
  • User benefit: Better performance with CUDA-enabled setups

📋 Implementation Status:

Change Applied to Bundled Code Tested Notes
normalize_emo_vec() ✅ Applied ⏳ Pending Core emotion control improvement
_load_and_cut_audio() ✅ Applied ⏳ Pending Memory stability fix
Cache clearing fixes ✅ Applied ⏳ Pending Critical for batch processing
BigVGAN import fix ✅ Applied ⏳ Pending Performance improvement

🎯 User-Facing Improvements:

  • Better emotion control: Emotion vectors now auto-normalize to prevent voice identity loss
  • Memory stability: Long reference audio automatically truncated to prevent crashes
  • Batch processing: Improved cache management for multiple generations
  • Performance: Fixed CUDA kernel loading for faster inference

🧪 Testing Required:

  • Emotion vector controls in radar chart widget
  • Long reference audio handling (>15 seconds)
  • Multiple sequential generations (cache clearing)
  • CUDA performance with FP16 enabled

2025-11-06: Performance & Stability Improvements

Reference commit range: 64cb31a..1d5d079 (September 18 → November 6, 2025) Upstream commits reviewed: 6 commits affecting infer_v2.py

🔧 New Changes Identified:

1. Empty Generator → IndexError Fix (commit: 750d9d9)

  • Issue: Non-streaming infer() would crash with IndexError on empty generator
  • Fix: Wrapped generator list conversion in try/except, returns None on error
  • Changes to infer_v2.py:
    • Lines 354-364: Added exception handling
    • Lines 680, 690: Changed return to yield for proper generator flow
  • Priority: 🔴 Critical - Prevents runtime crashes

2. Streaming Implementation (commit: b0c6ab8)

  • Added: stream_return and more_segment_before parameters for low-latency streaming
  • Purpose: Enable first-chunk audio output before full generation completes
  • Changes:
    • stream_return (bool): Return generator instead of final result
    • more_segment_before (int): Apply extra segmentation for faster first output (0-80 recommended)
    • Modified indextts/utils/front.py for tokenization changes
  • Priority: 🟡 Important - Enables streaming use cases

3. UNK Token Warnings (commit: 34be9bf)

  • Added: Detection and warning for unknown tokens in input text
  • Behavior:
    • Detects tokens the BPE model cannot encode
    • Prints warnings with token IDs and suggested fixes
    • Helps users debug encoding issues
  • Changes to infer_v2.py: Lines 450-459, added token validation logic
  • Priority: 🟢 Nice-to-have - Improves user experience with error diagnosis

4. S2Mel Stage Optimization (commit: 31e7e85)

  • Focus: Optimized mel-spectrogram generation pipeline
  • Changes:
    • indextts/s2mel/modules/commons.py: 15 lines added/modified
    • indextts/s2mel/modules/diffusion_transformer.py: 2 lines modified
    • indextts/s2mel/modules/flow_matching.py: 15 lines added (new optimization code)
    • indextts/infer_v2.py: 28 lines added/modified
  • Expected Impact: Faster mel-spectrogram generation (stage 2 of 3)
  • Priority: 🟡 Important - Performance improvement

5. GPT2 Inference Acceleration (commit: c1ef414)

  • Focus: Major performance optimization for GPT2 stage (stage 1)
  • Changes:
    • New module: indextts/accel/ directory (609 lines added):
      • accel_engine.py: Core acceleration logic (609 lines)
      • attention.py: Optimized attention mechanism (154 lines)
      • gpt2_accel.py: GPT2-specific optimizations (181 lines)
      • kv_manager.py: Key-value cache management (209 lines)
    • Modified indextts/gpt/model_v2.py: 63 lines changed (+47 new)
    • Modified indextts/infer_v2.py: 17 lines changed
  • Expected Impact: Significantly faster GPT2 inference (first stage bottleneck)
  • Priority: 🔴 Critical - Major performance gain, but new code complexity

6. Merge Commit (commit: 5d67f62 → 1d5d079)

  • Merge of acceleration branch into main
  • Contains all above changes integrated

📊 Summary

Change Files Modified Lines Changed Priority Status
Empty generator fix 1 +12/-9 🔴 Critical ⏳ Pending
Streaming support 2 +72/-18 🟡 Important ⏳ Pending
UNK token warnings 1 +7 🟢 Nice-to-have ⏳ Pending
S2Mel optimization 4 +54/-6 🟡 Important ⏳ Pending
GPT2 acceleration 6 +1228/-14 🔴 Critical ⏳ Pending

🚨 Integration Concerns

  1. New acceleration module complexity - The indextts/accel/ package adds significant complexity with KV-cache management and specialized attention mechanisms. Needs careful testing.

  2. Potential API changes - Streaming parameters (stream_return, more_segment_before) may require adapter updates if our wrapper doesn't expose these.

  3. GPU memory implications - KV-cache optimization might have different memory profiles; needs profiling on different GPU setups.

  4. File structure expansion - New accel/ subdirectory adds to maintenance burden.

🧪 Testing Required

  • Empty generator fix - test non-streaming inference with various inputs
  • Streaming latency - measure first-chunk output time with stream_return=True
  • GPT2 acceleration - benchmark inference speed (expect 1.5-3x speedup)
  • S2Mel optimization - verify audio quality unchanged
  • UNK token handling - test with multilingual/special character inputs
  • Memory profiling - GPU memory usage with new KV-cache manager
  • ComfyUI integration - ensure all changes work through our node interface

✅ Applied Changes Summary (2025-11-06)

All upstream commits successfully integrated:

  1. ✅ Empty generator fix (750d9d9) - Refactored infer() to use generator pattern:

    • Created wrapper infer() method that delegates to infer_generator()
    • Refactored monolithic method into infer_generator() with yield statements
    • Changed final return statements to yield for generator compatibility
    • Handles both streaming (stream_return=True) and non-streaming modes

  2. ✅ Streaming implementation (b0c6ab8) - Added streaming parameters:

    • Added stream_return parameter to infer() and infer_generator()
    • Added more_segment_before parameter for streaming segmentation
    • Integrated with generator-based inference for low-latency streaming

  3. ✅ UNK token warnings (34be9bf) - Added token validation:

    • Added token validation in infer_generator()
    • Warns users about unknown tokens with suggestions

  4. ✅ S2Mel optimization (31e7e85) - Applied all module updates:

    • commons.py: Updated fused_add_tanh_sigmoid_multiply() with torch.split
    • diffusion_transformer.py: Fixed sequence_mask() call with max_length parameter
    • flow_matching.py: Added enable_torch_compile() method to CFM class
    • infer_v2.py: Added use_torch_compile parameter and setup

  5. ✅ GPT2 acceleration (c1ef414) - Applied complete acceleration module:

    • Created /accel/ directory with 5 files (init, accel_engine, attention, gpt2_accel, kv_manager)
    • Modified model_v2.py: Added use_accel parameter to UnifiedVoice
    • Modified model_v2.py: Added accel engine initialization in post_init_gpt2_config()
    • Modified model_v2.py: Added accel-aware inference in inference_speech()

🔧 Custom Modifications Preserved

  • QwenEmotion lazy loading with path fallbacks (lines 81-112 in infer_v2.py)
  • W2V-BERT model loading from TTS folder with unified downloader (lines 145-166)
  • All custom logging and error handling in infer_v2.py and throughout
  • Custom model initialization sequence
  • Custom emotion vector normalization and processing

2026-07-11: Upstream Audit Before Release

Upstream repository checked: index-tts/index-tts (main) Upstream head observed: b5bd657 (2026-07-08) Check performed: 2026-07-11

Relevant upstream changes reviewed

Commit Upstream change Bundled status Release action
843972e Coerce QwenEmotion JSON emotion scores to float and reject non-numeric values clearly Applied in bundled clamp_score() Run a text-emotion generation with numeric-string classifier output
b154a1b WebUI text/vector preset save/load management Already covered differently by the ComfyUI-native preset manager and emotion_presets.json integration No direct port needed
b5bd657 Expose --accel and --torch-compile, add optional acceleration extras and WebUI settings Applied selectively; suite now forwards use_accel into the bundled GPT path, while retaining ComfyUI-owned dependency handling Validate acceleration fallback on compatible and non-accelerated setups
7264ce2 Improve IndexTTS-2 model resource checks and HF cache handling Suite-owned downloader differs and needs a separate comparison if download failures are reported No blind copy into bundled code

Findings

  • No upstream change was found that invalidates the current eight-emotion vector order, Qwen text-emotion syntax, audio-reference blending, or the suite's inline tag format.
  • The upstream QwenEmotion string-score fix is directly relevant to the suite's text-emotion path and should be applied before a release.
  • The upstream acceleration work is not a drop-in replacement because this repository bundles and adapts IndexTTS-2. The suite already contains the acceleration modules, but utils/models/unified_model_interface.py should be checked so use_accel reaches the bundled IndexTTS2 constructor.
  • Upstream WebUI presets are not copied verbatim: the suite's ComfyUI editor has a richer vector/radar, inline-tag, sidebar, and filesystem preset implementation.

Release follow-up checklist

  • Apply the upstream clamp_score() numeric coercion.
  • Pass use_accel through the unified IndexTTS-2 factory; validate fallback behavior.
  • Run a Qwen text-emotion generation using numeric-string classifier output.
  • Verify acceleration flags on a compatible CUDA installation and on a setup without optional acceleration dependencies.
  • Recheck bundled model-resource validation against the current upstream check logic.

Next Update Check: 2025-11-20

Monitoring: Watch for commits to indextts/infer_v2.py in https://github.com/index-tts/index-tts

Update command:

cd /tmp/index_tts_upstream
git pull origin main
git log --oneline 1d5d079..HEAD -- indextts/infer_v2.py

Priority changes to watch for:

  • Additional performance optimizations
  • Bug fixes in new acceleration code
  • Breaking API changes
  • Model loading improvements
Reference in New Issue View Git Blame Copy Permalink