diodiogod
8a00983418
- Remove orphaned dynamic_worker_manager.py (never imported/used) - Update PROJECT_INDEX.md to reflect file removal - Enhance STREAMING_ARCHITECTURE.md with feasibility analysis for F5-TTS - Document that StreamingWorkQueueProcessor is the active implementation - Confirm streaming architecture is engine-extensible via adapter pattern
130 lines
5.4 KiB
Markdown
130 lines
5.4 KiB
Markdown
# Streaming Parallel Processing Architecture
|
|
|
|
*How we bridged traditional sequential TTS with modern parallel processing*
|
|
|
|
## The Problem
|
|
|
|
Traditional TTS processing was **slow and sequential**:
|
|
|
|
- Process subtitle 1 → wait → process subtitle 2 → wait → process subtitle 3...
|
|
- With 26 subtitles, this could take several minutes
|
|
- Only 1 CPU?? *(not tested with cpu)* core working while others sit idle
|
|
|
|
## The Solution: Streaming Parallel Processing
|
|
|
|
We created a **smart routing system** that automatically chooses the best processing method:
|
|
|
|
### Traditional Mode (batch_size = 0)
|
|
|
|
- **When**: Few subtitles or user prefers sequential
|
|
- **How**: Processes one subtitle at a time, preserves all original functionality
|
|
- **Best for**: Small jobs, debugging, maximum compatibility
|
|
|
|
### Streaming Mode (batch_size > 0)
|
|
|
|
- **When**: Many subtitles and user sets batch workers (e.g., batch_size = 9)
|
|
- **How**: Splits work across multiple parallel workers
|
|
- **Result**: 26 subtitles processed simultaneously instead of sequentially
|
|
|
|
## The Smart Decision Logic
|
|
|
|
```
|
|
if (batch_size > 1):
|
|
→ Use Streaming Parallel Processing
|
|
else:
|
|
→ Use Traditional Sequential Processing
|
|
```
|
|
|
|
**Example**: 26 subtitles with batch_size=9 → Use streaming (much faster?)
|
|
|
|
|
|
## How We Bridged Everything
|
|
|
|
### 1. **Router Pattern**
|
|
|
|
- Created `SRTBatchProcessingRouter` that acts as a traffic controller
|
|
- Decides streaming vs traditional based on workload
|
|
- Both paths produce identical results
|
|
|
|
### 2. **Data Format Compatibility**
|
|
|
|
- Streaming system expects specific data structures (tuples)
|
|
- Traditional system uses different structures (objects)
|
|
- Router translates between formats seamlessly
|
|
|
|
### 3. **Language Optimization**
|
|
|
|
- **Smart grouping**: Process all English subtitles together, then German, etc.
|
|
- **Minimizes expensive model switching** (biggest performance killer)
|
|
- **Character batching**: Group same characters within each language
|
|
|
|
### 4. **Fallback Safety**
|
|
|
|
- If streaming fails for any reason → automatically falls back to traditional
|
|
- User never sees broken functionality
|
|
- Always produces correct output
|
|
|
|
## Real-World Performance
|
|
|
|
**Before**: 26 subtitles = sequential processing
|
|
**After**: 26 subtitles = parallel processing with 9 workers
|
|
**Speedup**: ~10-15% improvement due to model loading overhead in parallel workers
|
|
|
|
## User Experience
|
|
|
|
**Simple**: User just sets `batch_size` parameter:
|
|
|
|
- `0` = Safe traditional mode (works exactly as before)
|
|
- `4` = Use 4 parallel workers (faster) ??? No it's not that much; it/s slows down; so maybe this barely makes a difference
|
|
- `9` = Use 9 parallel workers (much faster)??? No it's not; it/s slows down; so maybe this barely makes a difference
|
|
|
|
The system automatically handles all the complexity behind the scenes.
|
|
|
|
## Technical Architecture
|
|
|
|
```
|
|
User Input → Unified TTS SRT Node → ChatterBox SRT Node → SRT Router
|
|
↓
|
|
┌─ Streaming Path (parallel workers)
|
|
└─ Traditional Path (sequential)
|
|
↓
|
|
Same Output (audio + timing + reports)
|
|
```
|
|
|
|
## Why This Matters
|
|
|
|
1. **Backward Compatibility**: All existing workflows work exactly the same
|
|
2. **Performance Scaling**: Automatically uses available CPU cores efficiently
|
|
3. **Future Proof**: Architecture supports any number of TTS engines
|
|
4. **User Choice**: Power users get speed, casual users get simplicity
|
|
|
|
The streaming architecture transforms TTS from a slow sequential process into a fast??? parallel system while maintaining 100% compatibility with existing workflows.
|
|
|
|
## Feasibility for F5-TTS and Future Engines
|
|
|
|
**Revised Assessment: Streaming IS feasible for F5-TTS and new TTS engines**
|
|
|
|
### Why it works across engines:
|
|
|
|
1. **Shared Language Architecture**: Both ChatterBox and F5-TTS use the unified `language_mapper.py` system:
|
|
- **ChatterBox**: `en → English`, `de → German`, `no → Norwegian`
|
|
- **F5-TTS**: `en → F5TTS_Base`, `de → F5-DE`, `fr → F5-FR`
|
|
- Both need identical model switching for different languages
|
|
|
|
2. **Engine-Agnostic Design**: The `StreamingWorkQueueProcessor` delegates actual TTS generation to the engine's processing method via `_process_single_segment_for_streaming()`. This abstraction makes it engine-independent.
|
|
|
|
3. **Same Optimization Benefits**: F5-TTS benefits from the same "process all German, then all English" language grouping to minimize expensive model switching.
|
|
|
|
### Implementation for New Engines:
|
|
|
|
1. Add `_process_single_segment_for_streaming()` method to new engine nodes
|
|
2. Ensure engine adapters work with the streaming work queue pattern
|
|
3. Test threading overhead vs. inference time for each engine
|
|
|
|
### Actual Streaming Implementation:
|
|
|
|
- **Active**: `StreamingWorkQueueProcessor` (streaming_work_queue.py) - used by ChatterBox SRT processing
|
|
- **Inactive**: `TrueStreamingProcessor` (streaming_processor.py) - experimental implementation
|
|
- **Removed**: `dynamic_worker_manager.py` - orphaned unused code
|
|
|
|
The streaming architecture was designed to be engine-extensible from the start. The time investment for ChatterBox was pioneering the approach - extending to F5-TTS is mostly adapter work. |