diodiogod-TTS-Audio-Suite/docs/Dev reports/HIGGS_AUDIO_2_INTEGRATION_SPECIFICATION.md

Higgs Audio 2 Integration Specification

TTS Audio Suite v4.4.0 - Technical Documentation

Complete technical specification for Higgs Audio 2 engine integration with unified architecture

---

Overview

The Higgs Audio 2 integration adds a new high-quality TTS engine to the TTS Audio Suite with advanced voice cloning capabilities. This integration follows the established unified architecture patterns while introducing minimal dependencies and maintaining full backward compatibility.

Key Features

• High-Quality TTS Generation: 3B parameter model with advanced audio synthesis
• Voice Cloning: Clone any voice from 30+ seconds of reference audio
• Multi-Language Support: English, Chinese dialects, and extensible to other languages
• Unified Architecture: Seamless integration with existing TTS Text and TTS SRT nodes
• Chunking Support: Handle unlimited text length with intelligent chunking
• Preset Voices: 8 built-in voice presets for instant use
• Caching System: Integrated with existing audio cache for performance
• Minimal Dependencies: Only 2 essential new dependencies added

---

Architecture Design

Modular Components (500-600 lines each)

The Higgs Audio 2 integration follows the established modular architecture principle:

engines/higgs_audio/
├── higgs_audio.py              # Main engine wrapper (688 lines)
├── higgs_audio_downloader.py   # Model downloader (357 lines)
├── boson_multimodal/           # Complete reference implementation
└── __init__.py

engines/adapters/
└── higgs_audio_adapter.py      # Unified adapter (363 lines)

nodes/engines/
└── higgs_audio_engine_node.py  # Configuration node (153 lines)

voices_examples/higgs_audio/
├── config.json                 # Voice preset configuration
├── belinda.wav                 # Voice presets (8 total)
├── en_woman.wav
└── ...

Integration Points

1. Engine Configuration: HiggsAudioEngineNode provides user interface
2. Engine Wrapper: HiggsAudioEngine handles model loading and generation
3. Unified Adapter: HiggsAudioEngineAdapter bridges to unified nodes
4. Unified Nodes: Existing TTS Text and TTS SRT nodes support higgs_audio
5. Model Management: Integrated with unified_downloader system
6. Caching: Uses existing AudioCache with custom key generator

---

Technical Implementation Details

Dependencies

Minimal Essential Dependencies Added:

• descript-audio-codec: Audio codec for Higgs Audio tokenizer
• vector_quantize_pytorch: Vector quantization for audio processing

Avoided Dependencies (handled internally or replaced):

• dacite → Direct dict handling
• boto3, s3fs → Models from HuggingFace, not S3
• json_repair → Python's standard json module
• pydantic → Standard Python dataclasses
• pydub → Existing librosa/torchaudio
• jieba, langid → Optional Chinese text processing
• accelerate → Optional optimization

Text Chunking Strategy

Unified Chunking System:

# Uses superior ImprovedChatterBoxChunker instead of custom implementation
max_chars = tokens_to_chars(max_tokens_per_chunk)  # Convert token limit
chunks = ImprovedChatterBoxChunker.split_into_chunks(text, max_chars)

Benefits:

• Character-based chunking (more predictable than token approximation)
• Better sentence boundary detection
• Consistent across all engines
• Overlap support for context preservation

Voice Cloning Implementation

Voice Priority System:

1. auto: Use preset if selected, otherwise reference audio
2. preset_dropdown: Prefer preset, fallback to reference
3. reference_input: Prefer reference, fallback to preset
4. force_preset: Only use preset, ignore reference

Voice Processing:

# Convert ComfyUI audio to base64 for boson_multimodal
audio_base64 = self._audio_to_base64(reference_audio)
audio_content = AudioContent(raw_audio=audio_base64, audio_url="")
messages.append(Message(role="assistant", content=[audio_content]))

Model Management

Organized Download Structure:

ComfyUI/models/TTS/HiggsAudio/
└── higgs-audio-v2-3B/
    ├── generation/
    │   ├── config.json
    │   ├── model.safetensors
    │   └── generation_config.json
    └── tokenizer/
        ├── config.json
        └── pytorch_model.bin

Auto-Download System:

• Integrated with unified_downloader
• Direct downloads (no HuggingFace cache duplication)
• Organized folder structure
• Model reuse across sessions

Caching System

HiggsAudioCacheKeyGenerator:

cache_data = {
    'text': text,
    'voice_preset': voice_preset,
    'model_path': model_path,
    'tokenizer_path': tokenizer_path,
    'temperature': temperature,
    'top_p': top_p,
    'top_k': top_k,
    'reference_audio_hash': audio_hash,  # If present
    'engine': 'higgs_audio'
}

Cache Integration:

• Registered with AudioCache.register_cache_key_generator()
• Automatic cache key generation
• Reference audio hashing for consistency
• Shared with existing cache system

---

Node Integration Details

Engine Configuration Node

HiggsAudioEngineNode Parameters:

• model: Model selection (higgs-audio-v2-3B, local models)
• device: Device selection (auto, cuda, cpu)
• voice_preset: Voice preset selection (8 presets + voice_clone)
• audio_priority: Voice source priority logic
• system_prompt: Generation guidance prompt
• temperature: Sampling temperature (0.0-2.0)
• top_p: Nucleus sampling (0.1-1.0)
• top_k: Top-k sampling (-1 to 100)
• max_new_tokens: Generation limit (128-4096)
• max_tokens_per_chunk: Chunking limit (100-500)
• silence_between_chunks_ms: Inter-chunk silence (0-2000ms)
• enable_chunking: Chunking toggle
• enable_cache: Cache toggle
• seed: Reproducibility seed

Unified Node Integration

TTS Text Node Integration:

elif engine_type == "higgs_audio":
    # Create wrapper with adapter
    result = engine_instance.generate_tts_audio(
        text=text,
        char_audio=reference_audio_dict,
        char_text=reference_text,
        character="narrator",
        **config_parameters
    )

TTS SRT Node Integration:

elif engine_type == "higgs_audio":
    # Parse SRT and generate per segment
    srt_segments = parse_srt(srt_content)
    for segment in srt_segments:
        segment_audio = engine_instance.generate_srt_audio(...)
        audio_segments.append(segment_audio)
    # Combine with timing logic

---

Voice Preset System

Built-in Voice Presets

1. belinda: Female, clear English pronunciation
2. en_woman: Female, natural English voice
3. en_man: Male, clear English voice
4. chadwick: Male, distinctive character voice
5. vex: Female, character voice with edge
6. mabel: Female, warm conversational voice
7. broom_salesman: Male, character voice
8. zh_man_sichuan: Male, Sichuan Chinese dialect

Voice Configuration

voices_examples/higgs_audio/config.json:

{
  "belinda": {
    "transcript": "This is Belinda's voice with clear pronunciation and natural intonation."
  },
  "en_woman": {
    "transcript": "A natural English woman's voice for general purpose use."
  }
}

Voice Discovery:

• Automatic preset loading from config.json
• Audio file validation (.wav format)
• Integration with Character Voices system
• Custom reference audio support

---

Performance Characteristics

Model Specifications

• Model Size: ~3B parameters
• Memory Requirements: 8GB+ VRAM recommended
• Sample Rate: 24,000 Hz
• Generation Speed: Similar to F5-TTS with chunking
• Voice Quality: High-fidelity with natural prosody

Chunking Performance

Recommended Settings:

• max_tokens_per_chunk: 225 tokens (~787 characters)
• silence_between_chunks_ms: 100ms
• enable_chunking: True for texts >225 tokens

Benefits:

• Unlimited text length support
• Consistent voice quality across chunks
• Natural pauses between segments
• Memory efficiency for long texts

Caching Benefits

• Cache Hits: Instant audio retrieval
• Cache Misses: Full generation with caching
• Storage: Efficient tensor storage
• Cleanup: Automatic cache management

---

Error Handling and Validation

Parameter Validation

HiggsAudioEngineAdapter.validate_parameters():

# Temperature validation
temperature = max(0.0, min(2.0, float(temperature)))

# Top-p validation  
top_p = max(0.1, min(1.0, float(top_p)))

# Voice preset validation
if voice_preset not in available_presets:
    validated["voice_preset"] = "voice_clone"

Error Recovery

1. Model Loading Failures: Clear error messages with resolution steps
2. Audio Processing Errors: Graceful fallback to basic TTS
3. Reference Audio Issues: Automatic fallback to preset voices
4. Memory Issues: Chunking and cache cleanup
5. Network Issues: Retry logic for model downloads

Logging and Diagnostics

print(f"🚀 Loading Higgs Audio 2 engine: {model_path}")
print(f"✅ Higgs Audio 2 engine loaded successfully")
print(f"🎤 Generating Higgs Audio for {len(chunks)} chunk(s)")
print(f"  ✅ Generated audio shape: {audio_tensor.shape}")

---

Compatibility and Migration

Backward Compatibility

• Existing Workflows: No changes required
• Character Voices: Full integration with existing system
• Cache System: Shared cache across all engines
• Audio Processing: Standard ComfyUI audio format

Forward Compatibility

• New Engines: Adapter pattern supports easy addition
• Feature Extensions: Modular design allows enhancement
• Model Updates: Download system supports model versioning
• Voice Expansion: Preset system supports additional voices

Migration Notes

From Direct Higgs Audio Nodes:

• Use Higgs Audio Engine + TTS Text instead of separate nodes
• Voice presets automatically detected
• All parameters preserved with validation

From Other TTS Engines:

• Same unified interface
• Voice switching supported
• Cache keys remain unique per engine

---

Testing and Validation

Manual Testing Approach

1. Model Loading: Verify automatic download and caching
2. Voice Presets: Test all 8 built-in presets
3. Voice Cloning: Test with various reference audio lengths
4. Text Chunking: Test with short and very long texts
5. Audio Quality: Compare with reference implementation
6. Error Handling: Test edge cases and failure modes

Integration Testing

1. Unified Nodes: Verify seamless engine switching
2. Character Voices: Test with Character Voices node
3. SRT Processing: Test with subtitle timing
4. Cache System: Verify cache hits and misses
5. Memory Management: Test cleanup and resource usage

Performance Benchmarks

Text Processing:

• Short text (50 chars): ~2-3 seconds
• Medium text (500 chars): ~5-8 seconds
• Long text (2000+ chars): Chunked processing

Memory Usage:

• Base model: ~6GB VRAM
• Peak generation: ~8GB VRAM
• Cache overhead: ~100MB per hour of audio

---

Future Enhancements

Planned Features

1. Streaming Support: Real-time generation for long texts
2. Voice Training: Custom voice preset creation
3. Multi-Speaker: Support for conversation-style generation
4. Advanced Editing: Word-level editing like F5-TTS
5. Emotion Control: Emotional tone parameters

Extension Points

1. New Voice Presets: Easy addition through config.json
2. Language Models: Support for additional languages
3. Audio Effects: Post-processing pipeline integration
4. Quality Settings: Speed vs quality trade-offs
5. Batch Processing: Multiple texts with different voices

---

Credits and Acknowledgments

Original Implementation

• HiggsAudio Team: Original Higgs Audio v2 model and boson_multimodal implementation
• ShmuelRonen: ComfyUI-HiggsAudio_2 reference implementation
• Anthropic: Claude Code development assistance

Integration Team

• TTS Audio Suite: Unified architecture integration
• Community Contributors: Voice presets and testing

Third-Party Libraries

• descript-audio-codec: Audio tokenization
• vector_quantize_pytorch: Vector quantization
• PyTorch: Deep learning framework
• torchaudio: Audio processing

---

Conclusion

The Higgs Audio 2 integration successfully adds high-quality voice cloning capabilities to the TTS Audio Suite while maintaining the unified architecture principles. The implementation:

✅ Follows Established Patterns: Uses existing adapter and unified node systems
✅ Minimal Dependencies: Only 2 essential new dependencies
✅ Superior Chunking: Uses unified chunking system for consistency
✅ Full Integration: Works with all existing unified nodes
✅ Voice Cloning: Supports both presets and custom reference audio
✅ Performance: Efficient caching and model management
✅ Backward Compatible: No breaking changes to existing workflows

This integration demonstrates the power of the unified architecture, allowing complex new engines to be added with minimal disruption while providing users with a consistent, powerful interface for high-quality TTS generation.

---

Generated for TTS Audio Suite v4.4.0 - Higgs Audio 2 Integration