Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 diodiogod
|
abca6197d8 | ||
|
diodiogod
|
701d2c97f8 | ||
|
diodiogod
|
965b314fac | ||
|
diodiogod
|
35cbcaca71 | ||
|
diodiogod
|
9275766f9b |
@@ -1,14 +0,0 @@
|
||||
# Registry archives contain runtime files and user documentation.
|
||||
# These exclusions do not remove anything from the Git repository.
|
||||
/.github/
|
||||
/tests/
|
||||
/scripts/
|
||||
/PROJECT_INDEX.md
|
||||
/AGENTS.md
|
||||
/CLAUDE.md
|
||||
/docs/BUMP_SCRIPT_INSTRUCTIONS.md
|
||||
|
||||
# Keep the metadata YAML files: the auxiliary registry reads its YAML at runtime.
|
||||
/docs/Dev reports/*
|
||||
!/docs/Dev reports/tts_audio_suite_engines.yaml
|
||||
!/docs/Dev reports/tts_audio_suite_aux_models.yaml
|
||||
@@ -14,9 +14,6 @@ jobs:
|
||||
steps:
|
||||
- name: Check out code
|
||||
uses: actions/checkout@v4
|
||||
- name: Preserve Unicode filenames in Registry archives
|
||||
# comfy-cli reads git ls-files as plain paths; Git must not escape emoji.
|
||||
run: git config core.quotepath false
|
||||
- name: Publish Custom Node
|
||||
uses: Comfy-Org/publish-node-action@main
|
||||
with:
|
||||
|
||||
-253
@@ -5,259 +5,6 @@ All notable changes to this project will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [5.10.2] - 2026-10-10
|
||||
|
||||
### Added
|
||||
|
||||
- Add Voice Capture playback, waveform preview, and start/end trimming controls
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve Voice Capture so recording saves a clip without running the workflow
|
||||
- Update Voice Capture workflows to use the saved clip only when the user clicks Run; existing workflows need an initial recording
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix Voice Capture unused bottom space and allow resizing down to the preview controls
|
||||
## [5.10.1] - 2026-10-10
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix Voice Capture reusing an old recording when running an unchanged workflow
|
||||
- Fix Voice Capture requiring a setting change before recording again
|
||||
## [5.10.0] - 2026-10-09
|
||||
|
||||
### Added
|
||||
|
||||
- Shared Runtime installation controls and suite security improvements
|
||||
- Add automatic Shared Runtime setup during TTS Audio Suite installation and updates, with a ComfyUI setting and file option to skip it.
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve runtime behavior so workflows use prepared environments without installing Python packages.
|
||||
- Improve file access and model loading across TTS, voice conversion, audio tools, and training to reject unsafe paths and executable checkpoint objects while keeping supported weight formats.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix Registry packages missing guides and workflows with Unicode filenames, and exclude development-only files from published packages.
|
||||
|
||||
### Removed
|
||||
|
||||
- Remove Dedicated Runtime choices from Qwen3-TTS and Step Audio EditX; existing workflows migrate to Shared Runtime without moving other settings.
|
||||
## [5.9.3] - 2026-10-08
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix isolated TTS worker startup failures from conflicting Python modules
|
||||
- Fix Qwen3-TTS and Voice Design failing on first use or after Step Audio EditX with utils is not a package
|
||||
- Fix conflicting module paths affecting all isolated TTS and ASR workers
|
||||
- Fix isolated workers losing ComfyUI imports when their working directory changes
|
||||
- Fix Step Audio EditX importing bundled modules during model discovery
|
||||
- Fix Step Audio EditX CPU and MPS device patch not targeting the bundled encoder
|
||||
## [5.9.2] - 2026-10-04
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix Classic ChatterBox crashing during short or consecutive speech segments (PR #368, thanks @melayyoub)
|
||||
- Fix Classic ChatterBox short segments being replaced with silence by the old workaround
|
||||
- Fix ChatterBox Classic and Official 23-Lang unclear failures when generation produces no usable speech; report a clear error instead
|
||||
|
||||
### Removed
|
||||
|
||||
- Remove Classic ChatterBox's old protection template and automatic short-text padding from Text and SRT; existing workflows remain compatible
|
||||
## [5.9.1] - 2026-10-04
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix ChatterBox Classic SRT cache reuse during character and language switching
|
||||
- Fix unnecessary language model loading when ChatterBox Classic SRT speech is already cached
|
||||
- Fix ChatterBox Classic pause processing ignoring the disabled audio-cache setting
|
||||
## [5.9.0] - 2026-10-03
|
||||
|
||||
### Added
|
||||
|
||||
- Add Step Audio EditX checkpoint selection and expanded sound editing
|
||||
- Add dated 2026-01-23 and 2025-11-28 Step Audio EditX model choices with separate downloads
|
||||
- Add expanded Step Audio EditX sound tags, including giggle, chuckle, clears throat, and new vocal reactions
|
||||
- Add new Step Audio EditX sounds plus fear and humour emotions to the Multiline TTS Tag Editor
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve Step Audio EditX model documentation and editor help while preserving existing workflow widget order and installed weights
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix Step Audio EditX Audio Editor ignoring the connected engine's selected model
|
||||
- Fix stale Step Audio EditX edits when changing models or generation settings
|
||||
- Fix ChatterBox v2/v3 native sound tags being treated as Step Audio EditX edits
|
||||
## [5.8.11] - 2026-09-19
|
||||
|
||||
### Added
|
||||
|
||||
- Add Multiline TTS Tag Editor globe language picker
|
||||
- Add an animated globe for choosing and swapping languages
|
||||
- Add realistic simplified continent outlines and smooth travel between language locations
|
||||
- Add square language buttons with a resizable layout that remembers its size
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve language selection from the node's left panel
|
||||
## [5.8.10] - 2026-09-04
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix Step Audio EditX failing when its speech recognition model is missing
|
||||
- Remove unused dependency constraints that could break installation on newer Python versions
|
||||
- Improve Echo-TTS and FAISS dependency handling through the extension installer
|
||||
## [5.8.9] - 2026-09-03
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix TTS model discovery when extra_model_paths.yaml defines a shared TTS directory
|
||||
- Fix local ComfyUI models/TTS installations being hidden by shared storage configuration.
|
||||
- Fix download path ordering so the configured default directory remains preferred.
|
||||
## [5.8.8] - 2026-08-31
|
||||
|
||||
### Added
|
||||
|
||||
- Clarify that parallel workers are available only for ChatterBox Classic
|
||||
- Prevent unsupported TTS engines from entering experimental parallel processing
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve legacy ChatterBox parallel-worker guidance
|
||||
- Explain that the option is not true model batching and may not improve speed
|
||||
## [5.8.7] - 2026-08-28
|
||||
|
||||
### Added
|
||||
|
||||
- Add native Transformers 5 support for VibeVoice and KugelAudio
|
||||
- Run VibeVoice and KugelAudio directly in the main ComfyUI environment
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix VibeVoice 1.5B and KugelAudio model loading and generation on current ComfyUI installations
|
||||
- Remove misleading compatibility warnings during VibeVoice loading and generation
|
||||
|
||||
### Removed
|
||||
|
||||
- Remove the VibeVoice runtime isolation setting
|
||||
## [5.8.6] - 2026-08-28
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve Vocal Removal cache hashing without temporary WAV encoding
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix Vocal Removal crashing before audio separation on systems with incompatible SoundFile virtual I/O
|
||||
## [5.8.5] - 2026-08-26
|
||||
|
||||
### Added
|
||||
|
||||
- Add support for loading RVC models from nested folders
|
||||
- Add automatic matching for index files stored beside nested RVC models
|
||||
- Preserve support for the existing shared RVC index folder
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve RVC character model organization
|
||||
## [5.8.4] - 2026-08-21
|
||||
|
||||
### Added
|
||||
|
||||
- Keep the example workflow reliable by disabling DeepSpeed by default
|
||||
- Automatically use standard inference when the optional DeepSpeed path is unsupported
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix IndexTTS DeepSpeed failures
|
||||
## [5.8.3] - 2026-08-18
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve long-running ChatterBox session stability
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix ChatterBox generations degrading after extended use
|
||||
- Fix English ChatterBox returning very short audio after many generations
|
||||
- Fix Official 23-Lang v1/v2 accumulating background processing hooks
|
||||
## [5.8.2] - 2026-08-18
|
||||
|
||||
### Added
|
||||
|
||||
- Keep trained adapters visible and selectable with the matching community model
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix MOSS Voice Acting LoRA selection
|
||||
- Fix LoRA controls appearing unavailable when the LAION Voice Acting 8B model is selected
|
||||
## [5.8.1] - 2026-08-11
|
||||
|
||||
### Added
|
||||
|
||||
- Add MOSS-TTS community voice-acting model support
|
||||
- Add the clearly labeled LAION Voice Acting 8B community model with automatic download
|
||||
- Add compatible local full-checkpoint discovery from the MOSS model folder
|
||||
- Support experimental LoRA training with the LAION community checkpoint
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve errors for unsupported local MOSS model layouts
|
||||
## [5.8.0] - 2026-08-11
|
||||
|
||||
### Added
|
||||
|
||||
- Add IndexTTS 2.5 as a new version of the existing IndexTTS engine
|
||||
- Add Chinese, English, Japanese, Spanish, and Arabic generation
|
||||
- Add explicit per-segment language switching for IndexTTS 2.5
|
||||
- Add official duration-factor and text-normalization controls
|
||||
- Keep IndexTTS 2.0 available for workflows that prefer its voice resemblance
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix stale audio or models when switching between IndexTTS 2.0 and 2.5
|
||||
## [5.7.0] - 2026-08-10
|
||||
|
||||
### Added
|
||||
|
||||
- Add integrated DramaBox LoRA model training
|
||||
- Add dataset preparation and training controls for DramaBox voice adapters
|
||||
- Add live training progress and loss reporting in the Model Training panel
|
||||
- Add DramaBox LoRA loading and adjustable adapter strength for inference
|
||||
- Add a ready-to-use DramaBox LoRA training workflow and guide
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve shared speech-clip dataset staging for model training
|
||||
## [5.6.5] - 2026-08-03
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix MOSS-TTS training settings in saved workflows
|
||||
- Fix existing MOSS Dataset Prep workflows loading values into the wrong fields
|
||||
- Fix invalid validation split and preparation batch size errors after updating
|
||||
- Fix MOSS training tensor shape errors caused by shifted codec settings
|
||||
## [5.6.4] - 2026-08-03
|
||||
|
||||
### Added
|
||||
|
||||
- Add MOSS-TTS training dataset folder support
|
||||
- Add direct loading of matching audio and transcript files from a folder
|
||||
- Support WAV, FLAC, MP3, OGG, and M4A training clips
|
||||
- Add optional recursive scanning for datasets organized into subfolders
|
||||
- Preserve existing JSONL manifest workflows
|
||||
## [5.6.3] - 2026-08-01
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve runtime availability checks so package startup code is not executed during installation
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix TTS Audio Suite installer validation failures
|
||||
- Fix ComfyUI Desktop installation failing on supported PyTorch and TorchAudio combinations
|
||||
## [5.6.2] - 2026-07-30
|
||||
|
||||
### Changed
|
||||
|
||||
@@ -36,7 +36,7 @@ The project code is MIT. Model weights carry their own licenses:
|
||||
VibeVoice MIT (research-only per model card) No
|
||||
Higgs Audio 2 Boson Higgs Audio 2 Community License Conditional
|
||||
Higgs Audio v3 Boson Higgs Audio v3 Research and Non-Commercial License No
|
||||
IndexTTS 2 / 2.5 bilibili Model Use License Conditional
|
||||
IndexTTS-2 bilibili Model Use License Conditional
|
||||
CosyVoice3 Apache-2.0 Yes
|
||||
Qwen3-TTS Apache-2.0 Yes
|
||||
Granite ASR Apache-2.0 Yes
|
||||
@@ -44,6 +44,7 @@ The project code is MIT. Model weights carry their own licenses:
|
||||
Echo-TTS CC-BY-NC-SA-4.0 No
|
||||
Fish Audio S2 Pro Fish Audio Research License No
|
||||
Dots TTS Apache-2.0 Yes
|
||||
TADA MIT code / Llama 3.2 weights Conditional
|
||||
DramaBox LTX-2 Community License Conditional
|
||||
OmniVoice Apache-2.0 Yes
|
||||
MOSS-TTS Apache-2.0 Yes
|
||||
|
||||
+14
-28
@@ -17,7 +17,7 @@
|
||||
**Key architectural rules:**
|
||||
- Chunking happens in the **processor**, not the adapter (`generate_single()` on adapter = raw single call)
|
||||
- Runtime routing happens through `ModelLoadConfig.runtime_mode` + `runtime_profile`, not ad-hoc subprocess calls
|
||||
- Shared runtime workers are currently used for fragile engine families such as Qwen3-TTS / ASR, Granite forced alignment, and Higgs Audio 2. VibeVoice/Kugel and other engines that support the modern stack run natively in the main Transformers 5 environment.
|
||||
- Shared runtime workers are currently used for fragile engine families such as VibeVoice, Qwen3-TTS / ASR, Granite forced alignment, Higgs Audio 2, and TADA. Engines that support the modern stack run natively in the main Transformers 5 environment.
|
||||
- YAML (`docs/Dev reports/tts_audio_suite_engines.yaml`) is source of truth for engine doc tables → run `python3 scripts/generate_engine_tables.py --readme` to regenerate
|
||||
- Auxiliary YAML (`docs/Dev reports/tts_audio_suite_aux_models.yaml`) is source of truth for helper/post-process model docs → run `python3 scripts/generate_aux_model_docs.py`
|
||||
- All models download to `ComfyUI/models/TTS/<model-name>/`
|
||||
@@ -25,7 +25,7 @@
|
||||
|
||||
## Engines
|
||||
|
||||
19 engines follow the pattern above:
|
||||
20 engines follow the pattern above:
|
||||
|
||||
| Engine | Adapter | Processor | SRT Processor | Engine Node |
|
||||
|--------|---------|-----------|---------------|-------------|
|
||||
@@ -44,13 +44,14 @@
|
||||
| Echo-TTS | `echo_tts_adapter.py` | `nodes/echo_tts/echo_tts_processor.py` | `echo_tts_srt_processor.py` | `echo_tts_engine_node.py` |
|
||||
| Fish Audio S2 Pro | `fish_audio_s2_adapter.py` | `nodes/fish_audio_s2/fish_audio_s2_processor.py` | `fish_audio_s2_srt_processor.py` | `fish_audio_s2_engine_node.py` |
|
||||
| Dots TTS | `dots_tts_adapter.py` | `nodes/dots_tts/dots_tts_processor.py` | `dots_tts_srt_processor.py` | `dots_tts_engine_node.py` |
|
||||
| TADA | `tada_adapter.py` | `nodes/tada/tada_processor.py` | `tada_srt_processor.py` | `tada_engine_node.py` |
|
||||
| DramaBox | `dramabox_adapter.py` | `nodes/dramabox/dramabox_processor.py` | `dramabox_srt_processor.py` | `dramabox_engine_node.py` |
|
||||
| OmniVoice | `omnivoice_adapter.py` | `nodes/omnivoice/omnivoice_processor.py` | `omnivoice_srt_processor.py` | `omnivoice_engine_node.py` |
|
||||
| MOSS-SoundEffect v2 | `moss_soundeffect_v2_adapter.py` | — | — | `moss_soundeffect_v2_engine_node.py` |
|
||||
| RVC | — | `engines/rvc/` | — | `rvc_engine_node.py` |
|
||||
|
||||
**Engine implementations live in:**
|
||||
- `engines/chatterbox/`, `engines/chatterbox_official_23lang/`, `engines/f5tts/`, `engines/higgs_audio/`, `engines/higgs_audio_v3/`, `engines/vibevoice_engine/`, `engines/step_audio_editx/`, `engines/cosyvoice/`, `engines/qwen3_tts/`, `engines/qwen3_asr/`, `engines/moss_tts/`, `engines/moss_soundeffect_v2/`, `engines/granite_asr/`, `engines/echo_tts/`, `engines/fish_audio_s2/`, `engines/dots_tts/`, `engines/dramabox/`, `engines/omnivoice/`, `engines/rvc/`
|
||||
- `engines/chatterbox/`, `engines/chatterbox_official_23lang/`, `engines/f5tts/`, `engines/higgs_audio/`, `engines/higgs_audio_v3/`, `engines/vibevoice_engine/`, `engines/step_audio_editx/`, `engines/cosyvoice/`, `engines/qwen3_tts/`, `engines/qwen3_asr/`, `engines/moss_tts/`, `engines/moss_soundeffect_v2/`, `engines/granite_asr/`, `engines/echo_tts/`, `engines/fish_audio_s2/`, `engines/dots_tts/`, `engines/tada/`, `engines/dramabox/`, `engines/omnivoice/`, `engines/rvc/`
|
||||
|
||||
## Documentation Files
|
||||
|
||||
@@ -67,6 +68,7 @@
|
||||
- `OMNIVOICE_TAGS_GUIDE.md` - OmniVoice native non-verbal tags and pronunciation overrides
|
||||
- `MOSS_TTS_PROMPT_FIELDS_GUIDE.md` - Official MOSS whole-segment prompt fields and inline `<>` translation limits
|
||||
- `DRAMABOX_PROMPTING_GUIDE.md` - DramaBox expressive scene prompts, voice references, controls, hardware, and license
|
||||
- `TADA_ENGINE_GUIDE.md` - TADA models, gated tokenizer setup, exact-transcript voice cloning, native controls, limits, and licensing
|
||||
- `COSYVOICE3_TAGS_GUIDE.md` - CosyVoice3 native paralinguistic tags
|
||||
- `CHATTERBOX_V2_SPECIAL_TOKENS.md` - ChatterBox v2 emotion tokens
|
||||
- `IndexTTS2_Emotion_Control_Guide.md` - IndexTTS-2 vector, text, audio, and blended emotion controls
|
||||
@@ -78,7 +80,6 @@
|
||||
- `ENGINE_COMPARISON.md`, `LANGUAGE_SUPPORT.md`, `FEATURE_COMPARISON.md` - Auto-generated tables
|
||||
|
||||
### Dev Docs (`docs/Dev reports/`)
|
||||
- `REGISTRY_COMPATIBILITY_PLAN.md` - File access, safe loading, config parsing, runtime installation/preferences, archive fixes, verification and Registry approval evidence
|
||||
- `tts_audio_suite_engines.yaml` - **Source of truth** for all engine metadata
|
||||
- `tts_audio_suite_aux_models.yaml` - **Source of truth** for helper/post-process model metadata
|
||||
- `SRT_IMPLEMENTATION.md` - SRT timing technical details
|
||||
@@ -120,13 +121,13 @@
|
||||
- `nodes/subtitles/text_to_srt_builder_node.py` - Build SRT from transcript text plus timing data, or estimate timings from plain text
|
||||
- `nodes/subtitles/srt_advanced_options_node.py` - Subtitle readability / segmentation policy options
|
||||
- `nodes/text/tts_tag_editor_node.py` - 🏷️ Multiline TTS Tag Editor: rich text editor with character/language/parameter dropdowns, preset system, syntax highlighting, undo/redo — pairs with `web/string_multiline_tag_editor.js`
|
||||
- `nodes/step_audio_editx_special/step_audio_editx_audio_editor_node.py` - 🎨 Audio Editor: post-process ANY engine's audio with Step Audio EditX (emotion/style, paralinguistic effects, speed); dated checkpoint choices and tags documented in `docs/STEP_AUDIO_EDITX_CHECKPOINTS.md`
|
||||
- `nodes/step_audio_editx_special/step_audio_editx_audio_editor_node.py` - 🎨 Audio Editor: post-process ANY engine's audio with Step Audio EditX (14 emotions, 32 styles, paralinguistic effects like `<Laughter>`, speed control) — universal, not just for Step Audio EditX engine
|
||||
- `nodes/engines/index_tts_emotion_options_node.py` - IndexTTS-2 emotion radar chart
|
||||
|
||||
### Audio / Video Nodes
|
||||
- `nodes/audio/analyzer_node.py` - Audio Wave Analyzer
|
||||
- `nodes/audio/vocal_removal_node.py` - Vocal/instrumental separation
|
||||
- `nodes/audio/recorder_node.py` - Saved microphone clip output with independent recording, preview, and trimming controls
|
||||
- `nodes/audio/recorder_node.py` - Microphone recording
|
||||
- `nodes/audio/merge_audio_node.py` - Audio mixing
|
||||
- `nodes/video/mouth_movement_analyzer_node.py` - Silent speech timing extractor
|
||||
- `nodes/models/load_rvc_model_node.py` - RVC model loader
|
||||
@@ -137,26 +138,25 @@
|
||||
- `unified_model_interface.py` - Universal factory pattern for all engines
|
||||
- `engine_registry.py` - Engine capability definitions
|
||||
- `factory_config.py` - standardized model load config, runtime mode/profile normalization
|
||||
- `step_audio_editx_checkpoints.py` - Dated Step Audio EditX choices and pinned Hugging Face revisions
|
||||
- `manager.py` - Model discovery and caching
|
||||
- `comfyui_model_wrapper/` - ComfyUI native model management integration
|
||||
- `extra_paths.py` - extra_model_paths.yaml support
|
||||
|
||||
### Isolated Runtimes (`utils/runtimes/`)
|
||||
- `profiles.py` - named runtime profiles (shared/dedicated legacy Transformers 4 environments and engine-specific runtimes)
|
||||
- `launcher.py` - worker environment construction, suite-first import paths, Windows toolchain env setup
|
||||
- `profiles.py` - named runtime profiles (`vibevoice_transformers4_shared`, dedicated variants, etc.)
|
||||
- `launcher.py` - runtime bootstrap, venv creation, Windows toolchain env setup
|
||||
- `session.py`, `protocol.py` - JSONL worker transport and message protocol
|
||||
- `bootstrap.py` - shared runtime bootstrap helpers
|
||||
- `vibevoice_proxy.py`, `qwen3_tts_proxy.py`, `qwen3_asr_proxy.py`, `higgs_audio_proxy.py` - parent-process proxies
|
||||
- `workers/` - worker subprocess entrypoints for VibeVoice, Qwen3-TTS, Qwen3-ASR/aligner, Higgs Audio, Fish Audio S2, and Step Audio EditX; each prioritizes suite imports before third-party dependencies
|
||||
- `vibevoice_proxy.py`, `qwen3_tts_proxy.py`, `qwen3_asr_proxy.py`, `higgs_audio_proxy.py`, `tada_proxy.py` - parent-process proxies
|
||||
- `workers/` - worker subprocess entrypoints for VibeVoice, Qwen3-TTS, Qwen3-ASR/aligner, Higgs Audio, and TADA
|
||||
- Current shared legacy T4 runtime profile is reused by:
|
||||
- VibeVoice / Kugel
|
||||
- Qwen3-TTS
|
||||
- Qwen3-ASR and Granite's optional Qwen forced aligner
|
||||
- Higgs Audio 2
|
||||
- TADA
|
||||
|
||||
### Audio (`utils/audio/`)
|
||||
- `voice_capture.py`, `voice_capture_api.py` - Background microphone recording and standalone capture controls; saved clips in ComfyUI input/voice_capture/
|
||||
- `trim.py` - Shared audio-range selection for Character Voices and Voice Capture
|
||||
### Audio (`utils/audio/`)
|
||||
- `processing.py` - Tensor manipulation, normalization, format conversion
|
||||
- `cache.py` - Unified TTS caching system
|
||||
- `chunk_combiner.py` - Smart chunk combination (auto/concatenate/crossfade/silence_padding)
|
||||
@@ -206,20 +206,6 @@
|
||||
- `web/asr_srt_preset_widgets.js` - ASR SRT preset locking
|
||||
|
||||
## Scripts & Config
|
||||
- `.comfyignore` - Registry archive exclusions for development tools and reports; keeps runtime metadata YAML, user documentation, and engine/training code
|
||||
- `scripts/bump_version_enhanced.py` - Version bump with changelog (use `patch`/`minor`/`major`)
|
||||
- `scripts/generate_engine_tables.py` - Regenerate all docs from YAML (`--readme` flag for README too)
|
||||
- `requirements.txt`, `pyproject.toml` - Dependencies and project metadata
|
||||
|
||||
## Registry compatibility helpers
|
||||
|
||||
- `utils/security/path_access.py` - Configured data-folder containment, link checks and safe output paths
|
||||
- `utils/security/data_loading.py` - Restricted checkpoint/dictionary/NumPy loading and scoped dependency adapters
|
||||
- `utils/security/demucs_loading.py`, `audio_separator_loading.py` - Preserve legacy separator metadata with restricted dependency loaders
|
||||
- `utils/downloads/unified_downloader.py` - Model download destinations contained in registered data folders
|
||||
- `utils/security/config_values.py` - Data-only config expressions with bounded list arithmetic
|
||||
- `utils/runtimes/bootstrap.py` - Runtime lookup/readiness only; generation never installs packages
|
||||
- `utils/runtimes/installation.py` - Installation-time preparation invoked by `install.py`
|
||||
- `utils/runtimes/settings.py`, `settings_api.py` - One persisted installation preference and same-origin UI writes
|
||||
- `web/runtime_installation_settings.js` - Shared runtime installation setting and status/cleanup details
|
||||
- `web/runtime_mode_compatibility.js` - Dedicated-to-shared migration for older workflows
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
[![Dynamic TOML Badge][version-shield]][version-url]
|
||||
[](https://ko-fi.com/diogogo)
|
||||
|
||||
# TTS Audio Suite v5.10.2
|
||||
# TTS Audio Suite v5.6.2
|
||||
|
||||
[](https://ko-fi.com/diogogo)
|
||||
|
||||
@@ -23,7 +23,7 @@ Subtitle workflows are still a core focus: the suite can transcribe to SRT, rebu
|
||||
|
||||
<!-- ENGINE_COMPARISON_START -->
|
||||
|
||||
## Quick Engine Comparison — 19 Engines
|
||||
## Quick Engine Comparison — 20 Engines
|
||||
|
||||
| Engine | Languages | Size | Key Features |
|
||||
|--------|-----------|------|--------------|
|
||||
@@ -33,7 +33,7 @@ Subtitle workflows are still a core focus: the suite can transcribe to SRT, rebu
|
||||
| **VibeVoice** | 🇺🇸🇨🇳🇩🇪🇪🇸🇫🇷🇮🇹 +21 | 5.4GB / 18GB | 90-min long-form, Native 4-speaker (Base models) |
|
||||
| **Higgs Audio 2** | 🇺🇸🇨🇳🇩🇪🇪🇸🇰🇷 | ~9GB | 3 multi-speaker, CUDA graphs (55+ tokens/sec) |
|
||||
| **Higgs Audio v3** | 🌐 100+ languages | ~8GB | Native inline emotion/style/prosody/SFX tags |
|
||||
| **IndexTTS 2 / 2.5** | 🇺🇸🇨🇳🇪🇸🇯🇵🇸🇦 | ~4.7GB / ~5.49GB | Emotion Control: 8 vectors, Text as reference |
|
||||
| **IndexTTS-2** | 🇺🇸🇨🇳🇯🇵 | ~4.7GB | Emotion Control: 8 vectors, Text as reference |
|
||||
| **CosyVoice3** | 🇺🇸🇨🇳🇯🇵🇰🇷 | ~5.4GB | Paralinguistic tags |
|
||||
| **Qwen3-TTS** | 🇺🇸🇨🇳🇩🇪🇪🇸🇫🇷🇮🇹 +4 | ~3-6GB | Voice design, ASR (Automatic Speech Recognition) |
|
||||
| **Granite ASR** | 🇺🇸🇩🇪🇪🇸🇫🇷🇯🇵🇵🇹 | ~4.6GB | Native speaker attribution / diarization (plus model variant), Native word-level timestamps (plus model variant) |
|
||||
@@ -41,6 +41,7 @@ Subtitle workflows are still a core focus: the suite can transcribe to SRT, rebu
|
||||
| **Echo-TTS** | 🇺🇸 | ~5.3GB + ~1.8GB | Diffusion-based (~30s best), Force Speaker KV (speaker drift control) |
|
||||
| **Fish Audio S2 Pro** | 🌐 80+ languages | ~10.3GB / ~8.0GB | Free-form sub-word emotion/prosody tags, Native multi-speaker and multi-turn dialogue with dynamic speaker references |
|
||||
| **Dots TTS** | 🇺🇸🇨🇳🇩🇪🇪🇸🇫🇷🇮🇹 +13 | ~6GB | Official auto language detect / language control, SOAR and MeanFlow distilled variants |
|
||||
| **TADA** | 🇺🇸🇨🇳🇩🇪🇪🇸🇫🇷🇮🇹 +4 | ~6.1GB / ~11GB with selected codec assets | Voice cloning, Reference-conditioned expression transfer |
|
||||
| **DramaBox** | 🇺🇸 | ~16.4GB | Expressive scene prompting and stage directions, Native and SRT-aware duration targeting |
|
||||
| **OmniVoice** | 🌐 600+ languages | ~3.7GB | Inline non-verbal tags and pronunciation overrides, Reference-free voice design |
|
||||
| **MOSS-TTS** | 🇺🇸🇨🇳🇩🇪🇪🇸🇫🇷🇮🇹 +18 | ~8.5GB tokenizer + ~6.1GB/17GB/18GB model | Reference-free voice design with MOSS-VoiceGenerator, Native 1-5 speaker TTSD dialogue |
|
||||
@@ -246,8 +247,8 @@ For comprehensive technical information, refer to the [SRT_IMPLEMENTATION.md](do
|
||||
This is the new architectural baseline for the suite.
|
||||
|
||||
* **Main environment moved forward**: the primary ComfyUI environment is now meant to run on **Transformers 5**
|
||||
* **Isolation for fragile engines**: engines that still behave better on the older stack can use **a shared** legacy runtime instead of forcing the whole suite backward
|
||||
* **Cleaner engine strategy**: modern engines such as **Higgs Audio v3**, **MOSS-TTS**, and other compatible stacks can stay native in the main environment
|
||||
* **Isolation for fragile engines**: engines that still behave better on the older stack can use **shared** or **dedicated** legacy runtimes instead of forcing the whole suite backward
|
||||
* **Cleaner engine strategy**: modern engines such as **Higgs Audio v3**, **Step Audio EditX**, **MOSS-TTS**, and other compatible stacks can stay native in the main environment
|
||||
* **Less dependency deadlock**: adding new engines no longer has to mean globally freezing the entire project to one old Transformers version
|
||||
|
||||
This matters because the suite now has a clearer split:
|
||||
@@ -277,9 +278,6 @@ This matters because the suite now has a clearer split:
|
||||
FP8-cast transformer storage, and optional `torch.compile`
|
||||
* **Generation diagnostics**: conservative near-silence detection in console
|
||||
output, TTS generation information, and SRT timing reports
|
||||
* **LoRA training**: official DramaBox audio-branch IC-LoRA training through
|
||||
the unified training nodes, with normalized manifest/index input and managed
|
||||
adapter export
|
||||
|
||||
**Important limitations:**
|
||||
|
||||
@@ -291,8 +289,6 @@ This matters because the suite now has a clearer split:
|
||||
|
||||
See the **[DramaBox Prompting Guide](docs/DRAMABOX_PROMPTING_GUIDE.md)** for
|
||||
prompt syntax, controls, memory modes, duration behavior, and examples.
|
||||
See the **[DramaBox LoRA Training Guide](docs/DRAMABOX_LORA_GUIDE.md)** for
|
||||
dataset formats, training workflow, adapter loading, and CPU-safe preflight.
|
||||
|
||||
</details>
|
||||
|
||||
@@ -429,7 +425,7 @@ dataset formats, training workflow, adapter loading, and CPU-safe preflight.
|
||||
3. Choose between Custom Character Switching (recommended) or Native Multi-Speaker mode
|
||||
4. Generate long-form content with automatic voice cloning from your voices folder
|
||||
|
||||
VibeVoice and KugelAudio run directly in the suite's Transformers 5 main environment.
|
||||
**Isolation note:** On current ComfyUI stacks, **Kugel/VibeVoice is usually meant to run with `⚠️ Runtime Isolation = Shared Runtime`**. If you force `Main Environment`, you are opting back into the dependency-conflict path.
|
||||
|
||||
**Perfect for:**
|
||||
|
||||
@@ -762,7 +758,7 @@ Both versions fully support character switching, language switching, and pause t
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><h3>IndexTTS 2 / 2.5 With Emotion Control</h3></summary>
|
||||
<summary><h3>IndexTTS-2 With Emotion Control</h3></summary>
|
||||
|
||||
**NEW in v4.9.0**: Revolutionary IndexTTS-2 engine with advanced emotion control and dual-source emotion blending!
|
||||
|
||||
@@ -772,12 +768,7 @@ Both versions fully support character switching, language switching, and pause t
|
||||
* **Character Voices Integration**: Use Character Voices `opt_narrator` on `emotion_audio`, including per-character `[Character:emotion_ref]` references
|
||||
* **8-Emotion Vector Control**: Manual precision control over Happy, Angry, Sad, Surprised, Afraid, Disgusted, Calm, and Melancholic emotions
|
||||
* **Character Tag Emotions**: Per-character audio emotion control using `[Character:emotion_ref]` syntax, blendable with vector/text emotion
|
||||
* **Emotion Alpha Control**: Fine-tune emotion conditioning from 0.0 to the official 1.0 maximum
|
||||
* **IndexTTS-2.5 Multilingual Generation**: Explicit Chinese, English, Japanese, Spanish, and Arabic selection
|
||||
* **Official 2.5 Duration Factor**: `duration_factor` scales the internal semantic feature sequence (`0.5` shorter/faster, `1.0` unchanged, `2.0` longer/slower). It is not natural prosody or exact-duration planning, does not apply to 2.0, and is not used by SRT native-duration targeting
|
||||
* **Pronunciation Overrides**: Preserve official `<word|pronunciation>` annotations through suite text processing
|
||||
|
||||
> **2.0 versus 2.5:** Treat 2.5 as a multilingual/efficiency alternative, not an automatic voice-cloning quality upgrade. In our manual listening, legacy 2.0 preserved speaker resemblance better when transferring a strong emotion from a different reference voice; 2.5 may still be preferable for Japanese, Spanish, Arabic, or cross-lingual generation. Strong external emotion settings can reduce perceived speaker identity, so compare both models for the target voice.
|
||||
* **Emotion Alpha Control**: Fine-tune emotion intensity from 0.0 (neutral) to 2.0 (maximum dramatic expression)
|
||||
|
||||
**Key Features:**
|
||||
|
||||
@@ -811,10 +802,9 @@ Welcome to our show! [Alice:happy_sarah] I'm so excited to be here!
|
||||
**NEW in v4.15**: Revolutionary LLM-based audio post-processing with emotion, style, and paralinguistic control!
|
||||
|
||||
* **🎨 Step Audio EditX - Audio Editor Node**: Post-process ANY TTS audio with advanced editing capabilities
|
||||
* **📅 Dated Checkpoints**: Choose `2026-01-23` (default for new engine nodes) or `2025-11-28` in the existing model dropdown. Downloads use pinned revisions and separate folders; existing unversioned/local selections keep their files. [Checkpoint guide](docs/STEP_AUDIO_EDITX_CHECKPOINTS.md).
|
||||
* **🗣️ Paralinguistic Effects**: Original tags plus January sounds such as `<giggle>`, `<chuckle>`, `<inhale>`, `<exhale>`, `<cough>`, and `<clears_throat>`; iteration syntax remains available, e.g. `<giggle:2>`.
|
||||
* **😊 Emotion Controls**: happy, sad, angry, excited, fear, humour, surprised, disgusted, confusion, empathy, embarrass, depressed, coldness, admiration; older `calm` and `fearful` options remain accepted
|
||||
* **🎭 Speaking Styles**: whisper, serious, child, older, girl, pure, sister, sweet, exaggerated, ethereal, generous, recite, act_coy, warm, shy, comfort, authority, chat, radio, soulful, gentle, story, vivid, program, news, advertising, roar, murmur, shout, deeply, loudly, arrogant, friendly
|
||||
* **🗣️ Paralinguistic Effects**: Insert natural sounds - Laughter, Breathing, Sigh, Uhm, Surprise (oh/ah/wa), Confirmation, Question, Dissatisfaction
|
||||
* **😊 14 Emotion Controls**: happy, sad, angry, excited, calm, fearful, surprised, disgusted, confusion, empathy, embarrass, depressed, coldness, admiration
|
||||
* **🎭 32 Speaking Styles**: whisper, serious, child, older, girl, pure, sister, sweet, exaggerated, ethereal, generous, recite, act_coy, warm, shy, comfort, authority, chat, radio, soulful, gentle, story, vivid, program, news, advertising, roar, murmur, shout, deeply, loudly, arrogant, friendly
|
||||
* **⚡ Speed Control**: faster, slower, more_faster, more_slower with multi-iteration support
|
||||
* **🔊 Voice Restoration**: ChatterBox VC integration to restore original voice resemblance after editing
|
||||
* **🏷️ Inline Edit Tags**: Apply effects directly in text using `<Laughter:2>`, `<emotion:happy>`, `<style:whisper>` tags
|
||||
@@ -991,15 +981,11 @@ Use the built-in OmniVoice preset in **📐 Visual Tag Builder** for the canonic
|
||||
|
||||
* **1.7B**: `MOSS-TTS-Local-Transformer`
|
||||
* **v1.5 8B**: `MOSS-TTS-v1.5` — 31 languages and more stable cloning
|
||||
* **Voice Acting 8B (Community - LAION)**: optional third-party full v1.5 fine-tune for expressive delivery; selecting it downloads `laion/moss-tts-v1.5-8b-voice-acting`
|
||||
* **v1 8B**: `MOSS-TTS`
|
||||
* **Native 8B Dialogue**: `MOSS-TTSD-v1.0`
|
||||
* **Voice Designer 1.7B**: `MOSS-VoiceGenerator` — select it in the MOSS engine for Voice Designer
|
||||
* **Shared Codec**: `MOSS-Audio-Tokenizer`
|
||||
|
||||
Compatible community full checkpoints can also be placed in `models/TTS/moss_tts/<model-name>/`.
|
||||
They are listed as `local:<model-name>` and classified from `config.json`; unsupported layouts fail explicitly.
|
||||
|
||||
**Supported Native Input Forms (TTSD):**
|
||||
|
||||
* `[Character]` tags
|
||||
@@ -1041,7 +1027,6 @@ Per-segment overrides are supported with `[]` parameter syntax for whole-segment
|
||||
|
||||
* **Initial MOSS LoRA training support is now integrated** through the unified `🎓 Model Training` flow.
|
||||
* Current scope is **MOSS-TTS 8B (Delay) LoRA training** with local adapter export into `models/TTS/moss_tts/loras/`.
|
||||
* The LAION Voice Acting 8B community checkpoint is accepted by the same training path because it uses the v1.5 Delay architecture, but full inference/training validation is pending community feedback.
|
||||
* Dataset-building UX is still early and will need refinement, but the end-to-end workflow is functional.
|
||||
|
||||
</details>
|
||||
@@ -1218,31 +1203,6 @@ Perfect for:
|
||||
- 🟢 **OpenSeeFace mouth movement**: ✅ Working (experimental)
|
||||
- 🔴 **MediaPipe mouth movement**: ❌ Incompatible (use OpenSeeFace)
|
||||
|
||||
### Shared Runtime installation
|
||||
|
||||
**Most users can keep the defaults.** Manager prepares one Shared Runtime during installation or updates. Qwen3-TTS/ASR, Step Audio EditX, Higgs Audio 2, and forced alignment reuse it. Model weights download only when needed. Running a workflow or restarting ComfyUI does not install Python packages.
|
||||
|
||||
If an engine reports that Shared Runtime is missing or needs repair, enable **Install shared runtime automatically** under **Settings > TTS Audio Suite > Runtime installation**, then repair/reinstall TTS Audio Suite through Manager. Restart when Manager requests it.
|
||||
|
||||
<details>
|
||||
<summary>Advanced: skip installation or remove an unused shared runtime</summary>
|
||||
|
||||
Turn off **Install shared runtime automatically** in the setting above to skip preparation during future installs and updates. Engines still set to Shared Runtime can use an existing environment; the setting does not switch engines to Main Environment.
|
||||
|
||||
Open **TTS Audio Suite > Runtime installation details** from ComfyUI's menu to see the configuration file, runtime folder, and installation status. To reclaim the runtime's space, turn off automatic installation, close ComfyUI, and manually delete the displayed **shared_legacy_t4** folder. Model weights and voice files are separate. Engines set to Shared Runtime need that environment restored before they can run again.
|
||||
|
||||
To skip the first installation, create this file before installing the suite:
|
||||
|
||||
**ComfyUI/user/__tts_audio_suite/runtime_settings.json**
|
||||
|
||||
```json
|
||||
{"install_shared_runtime": false}
|
||||
```
|
||||
|
||||
The UI and installer read the same file. With a custom ComfyUI user directory, use its **__tts_audio_suite** folder. After the suite has loaded, it remembers that location for Manager updates and reinstalls. Before the first installation with a custom user directory, set **TTS_AUDIO_SUITE_USER_DIRECTORY** to that directory in the installer's environment.
|
||||
|
||||
</details>
|
||||
|
||||
### Option 2: Manual Installation
|
||||
|
||||
**Same intelligent installer, manual setup:**
|
||||
@@ -1442,12 +1402,7 @@ Core TTS installation will continue; only the listed features may be unavailable
|
||||
|
||||
A common problem is installing dependencies in the wrong Python environment. Always ensure you are installing dependencies within your ComfyUI's Python environment.
|
||||
|
||||
Where an engine offers **⚠️ Runtime Isolation**, there are two choices:
|
||||
|
||||
- **Main Environment** uses ComfyUI's installed Python packages.
|
||||
- **⚠️ Shared Runtime** uses one support environment shared by compatible engines, helping avoid dependency conflicts. Keep the engine's default unless you need to change it.
|
||||
|
||||
Older workflows that selected Dedicated Runtime now use Shared Runtime. Their other settings stay in place. Existing dedicated folders are kept; after closing ComfyUI, you can delete an unused dedicated folder to reclaim its space. Keep the **shared_legacy_t4** folder if you use Shared Runtime.
|
||||
If the engine comparison table shows **`Shared`** or **`Dedicated`** in the **Isolation** column, that engine has its own secondary-environment path for dependency conflicts. Configure that on the engine node with `⚠️ Runtime Isolation` instead of trying to downgrade your main ComfyUI environment.
|
||||
|
||||
* **Verify your Python environment:** After activating your venv or navigating to your portable ComfyUI installation, check the Python executable being used:
|
||||
|
||||
@@ -1565,7 +1520,7 @@ For offline/manual setup:
|
||||
| VibeVoice | `ComfyUI/models/TTS/VibeVoice/` | ✅ | 1.5B and 7B variants |
|
||||
| RVC | `ComfyUI/models/TTS/RVC/` | ✅* | Base models auto; character `.pth` can be user-provided |
|
||||
| IndexTTS-2 | `ComfyUI/models/TTS/IndexTTS/` | ✅ | Emotion components included |
|
||||
| Step Audio EditX | `ComfyUI/models/TTS/step_audio_editx/` | ✅ | Lazy dated checkpoints + tokenizer stack; unversioned installs retained |
|
||||
| Step Audio EditX | `ComfyUI/models/TTS/step_audio_editx/` | ✅ | Main model + tokenizer stack |
|
||||
| CosyVoice3 | `ComfyUI/models/TTS/CosyVoice/` | ✅ | Variant-specific lazy downloads |
|
||||
| Qwen3-TTS / ASR | `ComfyUI/models/TTS/qwen3_tts/` | ✅ | Per-variant download + shared tokenizer |
|
||||
| MOSS-TTS | `ComfyUI/models/TTS/moss_tts/` | ✅ | Local/Delay/VoiceGenerator/SoundEffect v1/TTSD models plus shared MOSS-Audio-Tokenizer codec |
|
||||
@@ -1573,7 +1528,8 @@ For offline/manual setup:
|
||||
| Granite ASR | `ComfyUI/models/TTS/granite_asr/` | ✅ | Granite ASR models; plus adds native diarization/timestamps, optional Qwen forced aligner reused lazily for timestamps/SRT fallback |
|
||||
| Echo-TTS | `ComfyUI/models/TTS/echo-tts-base/` | ✅ | ~7.1GB total (base + dac); CC-BY-NC-SA |
|
||||
| Dots TTS | `ComfyUI/models/TTS/dots_tts/` | ✅ | Official base / soar / mf checkpoints with tokenizer, vocoder, speaker encoder |
|
||||
| DramaBox | `ComfyUI/models/TTS/dramabox/DramaBox/` | ✅ | ~16.4GB download; fast mode roughly 24GB VRAM; experimental FP8, staged, and sequential options can reduce VRAM, but no minimum GPU size is guaranteed; conditional LTX-2 Community License |
|
||||
| TADA | `ComfyUI/models/TTS/tada/` | ✅ | Selected 1B or 3B-ML model plus language-specific codec assets and an exact hash-verified ungated Llama tokenizer redistribution |
|
||||
| DramaBox | `ComfyUI/models/TTS/dramabox/DramaBox/` | ✅ | ~16.4GB download; fast mode roughly 24GB VRAM; experimental FP8 peaks on RTX 4090: staged ~15.1GB allocated, sequential ~11.7GB allocated / ~12.4GB reserved; conditional LTX-2 Community License |
|
||||
| Fish Audio S2 Pro | `ComfyUI/models/TTS/fish_audio_s2_pro/` | ✅ | Official BF16 or optional community FP8 checkpoint; the official checkpoint can be quantized on load with BNB INT8/NF4; main T5 environment with process teardown for Clear VRAM; Fish Audio Research License |
|
||||
| OmniVoice | `ComfyUI/models/TTS/omnivoice/` | ✅ | Official OmniVoice model. Voice cloning in this suite requires explicit reference text. |
|
||||
|
||||
@@ -1614,7 +1570,6 @@ Your support helps maintain and improve this project for the entire community!
|
||||
| Workflow | Description | Status | Files |
|
||||
| ---------------------------------------------- | ---------------------------------------------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------- |
|
||||
| **🤐 Voice Cleaning** | Audio restoration & cleanup with dual tool pipeline | ✅ **New in v4.13** | [📁 JSON](example_workflows/Voice%20Cleaning%20-%20🤐%20Noise%20or%20Vocal%20Removal%20+%20🤐%20Voice%20Fixer.json) |
|
||||
| **DramaBox LoRA 🎓 Model Training** | DramaBox IC-LoRA training workflow from staged speech clips | ✅ **New** | [📁 JSON](example_workflows/DramaBox%20LoRA%20🎓%20Model%20Training.json) |
|
||||
| **MOSS LoRA 🎓 Model Training** | Initial MOSS LoRA training workflow from clipped speech dataset | ✅ **New in v4.27** | [📁 JSON](example_workflows/MOSS%20LoRA%20🎓%20Model%20Training.json) |
|
||||
| **RVC 🎓 Model Training** | RVC voice model training workflow | ✅ **New in v4.25** | [📁 JSON](example_workflows/RVC%20🎓%20Model%20Training.json) |
|
||||
| **🎨 Step Audio EditX - Audio Editor** | Step Audio EditX audio editing with inline edit tags | ✅ **New in v4.14** | [📁 JSON](example_workflows/🎨%20Step%20Audio%20EditX%20-%20Audio%20Editor%20+%20Inline%20Edit%20Tags.json) |
|
||||
|
||||
+65
-13
@@ -344,7 +344,6 @@ def setup_api_routes():
|
||||
import folder_paths
|
||||
from server import PromptServer
|
||||
from aiohttp import web
|
||||
from utils.security.path_access import resolve_input_path, filename_component, child_path
|
||||
|
||||
def _get_ui_data_dir():
|
||||
base_dir = os.path.join(folder_paths.get_system_user_directory("tts_audio_suite"), "ui")
|
||||
@@ -356,10 +355,6 @@ def setup_api_routes():
|
||||
|
||||
from utils.voice.alias_api import register_character_alias_routes
|
||||
register_character_alias_routes(PromptServer.instance.routes, web)
|
||||
from utils.runtimes.settings_api import register_runtime_settings_routes
|
||||
register_runtime_settings_routes(PromptServer.instance.routes, web)
|
||||
from utils.audio.voice_capture_api import register_voice_capture_routes
|
||||
register_voice_capture_routes(PromptServer.instance.routes, web)
|
||||
|
||||
@PromptServer.instance.routes.get("/api/tts-audio-suite/index-tts-emotion-presets")
|
||||
async def get_index_tts_emotion_presets_endpoint(request):
|
||||
@@ -465,6 +460,66 @@ def setup_api_routes():
|
||||
print(f"⚠️ Error saving OmniVoice preset library: {e}")
|
||||
return web.json_response({"status": "error", "error": str(e)}, status=500)
|
||||
|
||||
@PromptServer.instance.routes.get("/api/tts-audio-suite/voice-input-devices")
|
||||
async def get_voice_input_devices_endpoint(request):
|
||||
"""Return input devices without risking a main-process PortAudio hang."""
|
||||
try:
|
||||
import json
|
||||
import subprocess
|
||||
|
||||
probe_script = r"""
|
||||
import json
|
||||
import sounddevice as sd
|
||||
|
||||
devices = []
|
||||
seen = set()
|
||||
for device in sd.query_devices():
|
||||
try:
|
||||
max_input_channels = int(device.get("max_input_channels", 0))
|
||||
except Exception:
|
||||
max_input_channels = 0
|
||||
if max_input_channels <= 0:
|
||||
continue
|
||||
|
||||
name = str(device.get("name", "")).strip()
|
||||
if not name or name in seen:
|
||||
continue
|
||||
|
||||
seen.add(name)
|
||||
devices.append(name)
|
||||
|
||||
print(json.dumps({"devices": devices}))
|
||||
"""
|
||||
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-c", probe_script],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=8,
|
||||
check=False,
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
stderr = (result.stderr or "").strip()
|
||||
stdout = (result.stdout or "").strip()
|
||||
error_message = stderr or stdout or f"device probe exited with code {result.returncode}"
|
||||
return web.json_response({"devices": [], "error": error_message}, status=500)
|
||||
|
||||
payload = json.loads(result.stdout or "{}")
|
||||
devices = payload.get("devices", [])
|
||||
if not isinstance(devices, list):
|
||||
devices = []
|
||||
|
||||
return web.json_response({"devices": devices})
|
||||
except subprocess.TimeoutExpired:
|
||||
return web.json_response(
|
||||
{"devices": [], "error": "Timed out while probing audio input devices. Leaving the dropdown on system default avoids startup hangs."},
|
||||
status=504,
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"⚠️ Error retrieving voice input devices: {e}")
|
||||
return web.json_response({"devices": [], "error": str(e)}, status=500)
|
||||
|
||||
@PromptServer.instance.routes.post("/api/tts-audio-suite/settings")
|
||||
async def set_inline_tag_settings_endpoint(request):
|
||||
"""API endpoint to receive settings from frontend for inline edit tags and restore VC"""
|
||||
@@ -586,8 +641,7 @@ def setup_api_routes():
|
||||
if not audio_file:
|
||||
return web.json_response({"error": "audio_file is required for preview analysis"}, status=400)
|
||||
|
||||
audio_file = resolve_input_path(audio_file)
|
||||
node_id = filename_component(data.get("node_id") or "preview")
|
||||
node_id = str(data.get("node_id") or "preview")
|
||||
|
||||
analyzer_node_path = os.path.join(os.path.dirname(__file__), "nodes", "audio", "analyzer_node.py")
|
||||
spec = importlib.util.spec_from_file_location("tts_audio_suite_audio_analyzer_node", analyzer_node_path)
|
||||
@@ -595,7 +649,7 @@ def setup_api_routes():
|
||||
spec.loader.exec_module(analyzer_node_module)
|
||||
|
||||
analyzer_node = analyzer_node_module.AudioAnalyzerNode()
|
||||
analysis_result = analyzer_node.analyze_audio(
|
||||
analyzer_node.analyze_audio(
|
||||
audio_file=audio_file,
|
||||
analysis_method=data.get("analysis_method", "silence"),
|
||||
precision_level=data.get("precision_level", "milliseconds"),
|
||||
@@ -609,9 +663,7 @@ def setup_api_routes():
|
||||
)
|
||||
|
||||
import folder_paths
|
||||
if analysis_result[1].startswith("Error:"):
|
||||
return web.json_response({"error": analysis_result[1]}, status=400)
|
||||
cache_file = child_path(folder_paths.get_output_directory(), f"audio_analyzer_cache_{node_id}.json")
|
||||
cache_file = os.path.join(folder_paths.get_output_directory(), f"audio_analyzer_cache_{node_id}.json")
|
||||
with open(cache_file, "r", encoding="utf-8") as f:
|
||||
payload = json.load(f)
|
||||
|
||||
@@ -620,7 +672,7 @@ def setup_api_routes():
|
||||
return response
|
||||
except Exception as e:
|
||||
print(f"⚠️ Audio analyzer preview failed: {e}")
|
||||
return web.json_response({"error": str(e)}, status=400 if isinstance(e, (ValueError, FileNotFoundError)) else 500)
|
||||
return web.json_response({"error": str(e)}, status=500)
|
||||
|
||||
@PromptServer.instance.routes.get("/api/tts-audio-suite/training-progress")
|
||||
async def get_training_progress_endpoint(request):
|
||||
@@ -642,4 +694,4 @@ def setup_api_routes():
|
||||
# Setup API routes when extension loads
|
||||
setup_api_routes()
|
||||
|
||||
# nodes.py already handles all the startup output and status reporting
|
||||
# nodes.py already handles all the startup output and status reporting
|
||||
|
||||
@@ -1,99 +0,0 @@
|
||||
# DramaBox LoRA training
|
||||
|
||||
TTS Audio Suite exposes the official DramaBox audio-branch IC-LoRA trainer
|
||||
through the unified `🎓 Model Training` flow. The bundled scripts are pinned to
|
||||
the same upstream DramaBox revision as the inference implementation.
|
||||
|
||||
See the official DramaBox
|
||||
[LoRA training guide](https://github.com/resemble-ai/DramaBox#training-a-lora-on-top-of-dramabox)
|
||||
for the upstream dataset format and training behavior.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Build a `⚙️ DramaBox Engine`.
|
||||
2. Create the dataset either externally or entirely inside ComfyUI:
|
||||
`🎞️ Training Clip Staging` → `🧾 DramaBox Dataset Rows`.
|
||||
3. Connect the resulting manifest to `📦 DramaBox Dataset Prep` and keep
|
||||
`dataset_type` set to `manifest`.
|
||||
4. Provide at least two clips per speaker.
|
||||
5. Connect the dataset to `🎛️ DramaBox Training Config` and then to `🎓 Model Training`.
|
||||
6. Select the resulting adapter in the DramaBox engine, or enter its path in the
|
||||
advanced LoRA override field.
|
||||
|
||||
The dataset node accepts:
|
||||
|
||||
- JSONL/JSON manifests with `audio_filepath` (or `audio_path`) and `text` (or
|
||||
`transcript`)
|
||||
- TSV rows with audio path and text
|
||||
- the official `gemini_synthetic` and `libriheavy` index formats
|
||||
|
||||
Manifest rows may include `speaker`, `speaker_id`, `language`, and `duration`.
|
||||
If `speaker` is omitted, rows are grouped as `speaker_1`. Duration and audio
|
||||
metadata are measured without loading the waveform into the GPU. The suite
|
||||
converts all accepted formats into the `~`-delimited speaker index required by
|
||||
the upstream training loop. Clips are restricted to 2–20 seconds by default.
|
||||
|
||||
For an all-ComfyUI dataset, connect one or more `AUDIO` sources to
|
||||
`🎞️ Training Clip Staging`, then enter one transcript per clip in
|
||||
`🧾 DramaBox Dataset Rows`. Speaker and language lines are optional; shared
|
||||
defaults are used when those lines are blank.
|
||||
|
||||
### Transcripts and scene descriptions
|
||||
|
||||
The official trainer accepts either plain spoken transcripts or the same
|
||||
scene-style prompt format used for inference. For example, both of these are
|
||||
valid training text:
|
||||
|
||||
```text
|
||||
This is the spoken sentence.
|
||||
A woman speaks warmly, "This is the spoken sentence."
|
||||
```
|
||||
|
||||
Use scene descriptions only when they accurately describe the clip. Plain
|
||||
transcripts remain valid and are the safer choice when no reliable style or
|
||||
scene annotation is available.
|
||||
|
||||
## What training does
|
||||
|
||||
The first preprocessing pass uses Gemma and the DramaBox audio VAE to create
|
||||
cached conditions and audio latents. The training process then attaches a LoRA
|
||||
to the audio transformer branch. It saves periodic checkpoints and exports the
|
||||
selected adapter to:
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/dramabox/loras/<adapter_name>/
|
||||
```
|
||||
|
||||
The job directory, normalized index, preprocessing cache, progress file, and
|
||||
logs are stored under:
|
||||
|
||||
```text
|
||||
ComfyUI/output/tts_audio_suite_training/dramabox/
|
||||
```
|
||||
|
||||
`continue_from` is a warm start from an existing LoRA checkpoint; it is not an
|
||||
exact optimizer-state resume. Use saved checkpoints to compare quality rather
|
||||
than assuming the last step is best. Optional upstream validation can be
|
||||
enabled with a `val_config` YAML path, but it launches full DramaBox inference
|
||||
at each save step. It requires a second GPU: set `validation_gpu` to that
|
||||
physical CUDA device index. The suite rejects validation on the training GPU
|
||||
instead of allowing both full model processes to compete for the same VRAM.
|
||||
|
||||
DramaBox LoRA inference supports normal transformer precision, `fp8_cast`, and
|
||||
the optional `torch.compile` path. With normal precision the live adapter is
|
||||
reversibly merged for fast inference. With FP8 storage the BF16 adapter remains
|
||||
unmerged above the immutable FP8 base weights, avoiding unsafe mixed-dtype
|
||||
weight fusion while retaining the main FP8 memory saving.
|
||||
|
||||
The base DramaBox runtime is reused when the selected adapter or LoRA strength
|
||||
changes. Strength updates are applied directly to the live PEFT adapter, while
|
||||
the generated-audio cache still treats adapter path, file revision, and strength
|
||||
as distinct generation settings. Replacing an adapter with a different rank may
|
||||
retrace compiled transformer blocks, but does not reload the base checkpoint.
|
||||
|
||||
## CPU-safe preflight
|
||||
|
||||
Training and Gemma/VAE preprocessing are GPU workloads. For development or
|
||||
validation without touching CUDA, enable `dry_run` in the training config and
|
||||
the dataset node's `dry_run`/`preprocess_now` controls. This writes the
|
||||
normalized index and official command/config without loading DramaBox weights.
|
||||
@@ -140,8 +140,9 @@ The segment override ends at the next character tag.
|
||||
VRAM. It additionally keeps the diffusion transformer in system RAM while
|
||||
another major stage uses CUDA. It transfers the transformer for every
|
||||
generated segment or long-form chunk and is therefore substantially slower.
|
||||
Actual peak usage varies with the environment, generation settings, and
|
||||
other loaded components; no minimum GPU size is guaranteed.
|
||||
With `fp8_cast`,
|
||||
this measured about 11.7GB peak allocated and 12.4GB peak reserved VRAM on
|
||||
an RTX 4090; leave additional headroom for ComfyUI and other loaded models.
|
||||
System RAM must hold the offloaded transformer (about 3.4GB with FP8 or
|
||||
6.6GB without it).
|
||||
- `fp8_cast` uses the official LTX FP8 transformer weight-storage policy and
|
||||
|
||||
@@ -1,9 +1,5 @@
|
||||
# Isolated Runtimes Plan
|
||||
|
||||
Historical design. The current implementation uses Main Environment or one Shared Runtime,
|
||||
prepared by install.py. Dedicated profiles and first-use package installation were removed;
|
||||
see [Registry compatibility work](REGISTRY_COMPATIBILITY_PLAN.md).
|
||||
|
||||
## Goal
|
||||
|
||||
Keep ComfyUI in one modern main runtime while routing fragile engines into
|
||||
|
||||
@@ -1,248 +0,0 @@
|
||||
# Registry compatibility work
|
||||
|
||||
Scope implemented locally on 2026-10-09: file containment, restricted data loading,
|
||||
configuration parsing, installation-time Shared Runtime setup, user preferences,
|
||||
dedicated-runtime migration, and Registry archive cleanup. Local verification is recorded below.
|
||||
These verification records do not constitute Registry approval.
|
||||
|
||||
## 1. File access
|
||||
|
||||
- Restrict workflow and HTTP-supplied file paths to the configured ComfyUI
|
||||
input/output/temp or explicitly registered model/voice folders, as appropriate
|
||||
for the operation. Resolve links before checking containment.
|
||||
- Keep the audio analyzer's existing widget and browser upload/drag-and-drop.
|
||||
Uploaded audio is already placed in input by ComfyUI's upload endpoint.
|
||||
- Reject paths outside the permitted folders. Copying an arbitrary server-side
|
||||
path into input would still read that path and expose its contents.
|
||||
- Audit the analyzer preview route and node execution together, plus the suite's
|
||||
other file-backed widgets and routes. Protect generated cache/output names
|
||||
from raw node IDs and other request values.
|
||||
- Preserve explicitly configured external model and voice directories; do not
|
||||
turn their normal use into an unrestricted path exception.
|
||||
- Restrict model download destinations before network access, writes, or replacement;
|
||||
reject traversal and links escaping the permitted data roots. Custom-node code
|
||||
folders are excluded from data roots even though ComfyUI registers them.
|
||||
|
||||
## 2. Checkpoint and dictionary loading
|
||||
|
||||
- Prefer restricted PyTorch loading for state dictionaries; keep existing model
|
||||
formats where safe loading supports them. Do not require conversion or copies
|
||||
of upstream weights as the default solution.
|
||||
- The local format audit covered 52 PyTorch files and one plain-pickle language
|
||||
dictionary: 50 checkpoints opened directly with weights_only=True, two Dots
|
||||
latent-statistics files needed a scoped NumPy type allowlist, and the dictionary
|
||||
loaded with a data-only unpickler that refuses class/global lookups.
|
||||
- The audit used the configured test interpreter with Torch 2.10.0+cu130. It did
|
||||
not establish full engine inference, every remote variant, TorchScript safety,
|
||||
or compatibility with old whole-model HuBERT files and all training resumes.
|
||||
- Review RVC inference/separation/training loaders, DramaBox training, Dots
|
||||
integration, optional Russian stress dictionaries, and remaining unsafe
|
||||
helpers. No automatic fallback to unrestricted loading for an untrusted file.
|
||||
- Keep dependency patches scoped to suite integration rather than changing
|
||||
unrelated installed packages or globally weakening deserialization.
|
||||
|
||||
## 3. Configuration expressions
|
||||
|
||||
- Replace the Qwen tokenizer's constant eval with its literal value.
|
||||
- Replace bundled FunASR configuration eval calls with restricted data parsing.
|
||||
Preserve required list/tuple expressions, including existing list addition and
|
||||
repetition; literal_eval alone does not cover those expressions.
|
||||
- Keep neural-network model.eval() calls. Add the required adjacent TTS Audio
|
||||
Suite patch comments for bundled-code changes.
|
||||
|
||||
## 4. Optional isolated environments
|
||||
|
||||
The maintainer rejected a manual setup command as the normal user experience.
|
||||
Required behavior: users get automatic setup or a clear install action inside
|
||||
ComfyUI, without editing files, running terminal commands, or learning about
|
||||
Python environments. Keep existing workflow widgets and their order. Generation now checks prepared-runtime metadata and never installs packages.
|
||||
The installer prepares or reuses Shared Runtime independently of its main-dependency fast path.
|
||||
|
||||
Implemented: prepare the single shared legacy environment automatically from the existing install.py hook during normal
|
||||
Manager installation/update. Default Qwen3-TTS, Qwen ASR, Step Audio EditX, and
|
||||
Higgs Audio 2 already select this environment. Users keep the existing experience
|
||||
without terminal setup or additional support packages. Model weights still
|
||||
download only when the corresponding engine is used.
|
||||
|
||||
The local shared environment measured 139.1 MB (132.7 MiB), including 6,429
|
||||
files. It inherits the main environment's heavy packages rather than installing
|
||||
another PyTorch copy. This measures the current machine, not download size or
|
||||
a universal upper bound; missing/incompatible host dependencies can change it.
|
||||
|
||||
Keep an advanced opt-out for automatic shared-environment preparation. A local
|
||||
installer configuration can apply before first installation; a ComfyUI setting
|
||||
can control subsequent installation/update/repair runs. Skipping preparation
|
||||
must neither delete an existing environment nor silently switch engines to an
|
||||
incompatible main environment. A missing required environment should report
|
||||
that support must be enabled and installation repaired through Manager.
|
||||
|
||||
### Implemented advanced controls
|
||||
|
||||
- Add a default-on "Install shared runtime automatically" setting under
|
||||
ComfyUI Settings > TTS Audio Suite > Runtime installation.
|
||||
- Store the preference in one server-side JSON file read by the installer.
|
||||
The UI edits that same file and shows its actual location; a hand-edited
|
||||
preference must not be overwritten with stale browser defaults on page load.
|
||||
Prefer the suite's existing ComfyUI System User storage, outside package files
|
||||
replaced during updates. Resolve the configured user directory consistently
|
||||
in the running suite and the separate installer process.
|
||||
Default location: ComfyUI/user/__tts_audio_suite/runtime_settings.json.
|
||||
Document the equivalent location under a custom user directory so pre-install
|
||||
opt-out does not depend on opening a UI that is not installed yet.
|
||||
- File schema:
|
||||
|
||||
```json
|
||||
{
|
||||
"install_shared_runtime": false
|
||||
}
|
||||
```
|
||||
|
||||
- Only accept the defined boolean. Do not
|
||||
accept executable paths, package names, URLs, or shell commands from this file,
|
||||
a workflow, or a management request. Protect settings-management writes.
|
||||
- Turning shared preparation off affects later installer runs. It does not
|
||||
remove the installed environment, stop its use by existing workflows, or
|
||||
select Main Environment instead. The file option supports pre-install opt-out;
|
||||
the UI becomes available after the suite is installed.
|
||||
- Do not add automatic deletion on restart. Show whether the environment is
|
||||
installed, its folder location, and instructions to close ComfyUI before
|
||||
manually deleting runtimes/shared_legacy_t4. Explain that engines still set
|
||||
to Shared Runtime then need that environment restored or another prepared,
|
||||
compatible runtime selected. Model weights and voice files are separate.
|
||||
|
||||
### Runtime choices implemented locally
|
||||
|
||||
- Remove Dedicated Runtime from Qwen3-TTS and Step Audio EditX. Runtime widgets
|
||||
stay in their existing positions with Main Environment and Shared Runtime.
|
||||
- Old Dedicated Runtime widget/API values and old dedicated profile names map
|
||||
to Shared Runtime. Frontend migration also updates saved workflow widgets.
|
||||
- Remove the separate Qwen, Step, and VibeVoice legacy profile definitions; keep
|
||||
the shared profile unchanged so existing installations can be reused.
|
||||
- Leave existing dedicated environment folders on disk. Do not delete user files
|
||||
as part of this change. Users can remove unused folders after closing ComfyUI.
|
||||
- No dedicated installation controls or file settings are needed. Reuse valid
|
||||
shared runtime metadata on later updates, checking profile changes separately
|
||||
from the main dependency installer's fast path.
|
||||
|
||||
Manager can defer its installation jobs until restart, including on Windows.
|
||||
That is still Manager-requested package installation. A workflow node recording
|
||||
flags and our prestartup/import code calling pip on every boot is a different
|
||||
boundary; timing alone does not make it acceptable. No new installer node is
|
||||
needed for the recommended default. The implementation still requires Registry review; it does not assert approval under the latest policy.
|
||||
|
||||
Reuse existing compatible environments. An update must not silently delete a
|
||||
working environment during generation. Model downloads remain separate from
|
||||
installing executable Python dependencies. An HTTP route or workflow node that
|
||||
invokes pip after a click would still be runtime installation; renaming pip or
|
||||
hiding it in a helper is not a policy fix.
|
||||
|
||||
Evidence checked:
|
||||
|
||||
- [Official standards](https://docs.comfy.org/registry/standards) prohibit runtime
|
||||
package installation through subprocess calls.
|
||||
- [Manager's install queue](https://github.com/Comfy-Org/ComfyUI-Manager/blob/main/glob/manager_server.py)
|
||||
accepts registered node IDs/versions and delegates to its package installer.
|
||||
Its [special-purpose files documentation](https://github.com/Comfy-Org/ComfyUI-Manager#custom-node-support)
|
||||
documents automatic install.py execution during installation.
|
||||
- [Manager's deferred installer](https://github.com/Comfy-Org/ComfyUI-Manager/blob/main/glob/manager_core.py#L2070)
|
||||
schedules installation jobs for restart when appropriate, rather than requiring
|
||||
each node pack to create an independent startup installer.
|
||||
- [SAM3 Registry versions](https://api.comfy.org/nodes/comfyui-sam3/versions?include_status_reason=true):
|
||||
0.1.21 is Active, with a manual SAFE decision by drltdata@comfy.org under
|
||||
policy-v0.1. Its [published archive](https://cdn.comfy.org/pznodes/comfyui-sam3/0.1.21/node.zip)
|
||||
contains install.py calling comfy_env.install(), plus isolated-environment
|
||||
dependency configuration. This is an accepted installation-time precedent,
|
||||
not a blanket exception for our current first-use bootstrap or newer policies.
|
||||
- [comfy-env installation documentation](https://docs.comfy-forge.org/comfy-env/install/)
|
||||
describes installation-time creation rather than environment creation during
|
||||
node execution. Adoption of that library is not required for this plan.
|
||||
- Registry review requests [226](https://github.com/Comfy-Org/registry-backend/issues/226),
|
||||
[261](https://github.com/Comfy-Org/registry-backend/issues/261), and
|
||||
[210](https://github.com/Comfy-Org/registry-backend/issues/210) are unanswered;
|
||||
they do not prove approval of optional runtime installers.
|
||||
|
||||
### User documentation
|
||||
|
||||
README.md now describes automatic installation, the UI/file opt-out, custom user
|
||||
directories, repair through Manager, and manual cleanup. The details command shows
|
||||
the actual preference file and runtime folder; no UI route or node runs pip.
|
||||
|
||||
## 5. Registry archive contents
|
||||
|
||||
- Added .comfyignore to exclude root tests/scripts, GitHub automation, the project
|
||||
index, and development reports from Registry archives. Repository contents
|
||||
remain available for development.
|
||||
- Keep the two metadata YAML files in docs/Dev reports. The auxiliary model
|
||||
registry reads its YAML at runtime. Keep user guides, example workflows,
|
||||
frontend assets, and all engine/training code.
|
||||
- The publisher's current Git filename handling skips quoted emoji paths.
|
||||
Configure core.quotepath=false in publishing so those guides/examples ship.
|
||||
|
||||
## Verification for the implementation
|
||||
|
||||
- Compare an archive with an independently collected, NUL-delimited Git source
|
||||
manifest (git ls-files -z), not the packer's own parsed filename list. Require
|
||||
every missing file to have an intentional exclusion and verify retained bytes.
|
||||
This catches silent omissions from quoting, decoding, and path parsing.
|
||||
- Exercise uploads, allowed typed paths, rejected traversal/link escapes, and
|
||||
malformed node IDs through both node execution and HTTP preview routes.
|
||||
- Check restricted loaders against the actual local formats, then exercise the
|
||||
affected engine inference and training paths using the configured test setup.
|
||||
- Check existing prepared environments and missing/stale profiles. Confirm that
|
||||
generation cannot install packages or replace an environment.
|
||||
- Recheck the published Registry status separately from successful upload;
|
||||
packaging cleanup by itself does not fix the runtime findings.
|
||||
|
||||
## Local checks for dedicated runtime removal
|
||||
|
||||
- The existing runtime unit test module passed all 27 cases, including legacy
|
||||
mode/profile migration and rejection of unsupported values.
|
||||
- JavaScript migration checks covered 20 hook/value combinations and confirmed
|
||||
other widget positions/values and unrelated nodes remain unchanged.
|
||||
- The restarted ComfyUI server exposes only Main Environment and Shared Runtime
|
||||
for Qwen3-TTS and Step Audio EditX, with the original input order. Its prompt
|
||||
validation accepts old dedicated values and rejects unknown modes. Requests
|
||||
deliberately omitted required downstream inputs, so no generation was queued.
|
||||
- ComfyUI serves the new frontend extension. Full-fix checks are recorded below.
|
||||
|
||||
- Source rewrites must match AST call identity. Compare preserved attribute-call
|
||||
identities against the original source, independently of edit selection; builtin
|
||||
eval removal must not rename model.eval() or other unrelated methods.
|
||||
|
||||
## Completed local verification
|
||||
|
||||
- 59 targeted existing unit cases passed, covering runtime/profile migration,
|
||||
installer repairs, audio processing, voice compatibility, and dependency guidance.
|
||||
- An empty temporary Shared Runtime was created by the real installer using the
|
||||
configured test Python. It installed Transformers 4.57.3, inherited host PyTorch,
|
||||
passed import/readiness/reuse checks, and left the host packages unchanged. The
|
||||
temporary environment was removed. Windows console logging uses ASCII output.
|
||||
- Installer hook checks covered enabled preparation and file-based opt-out.
|
||||
Missing/stale runtime checks cannot run subprocess installers during generation;
|
||||
simulated failed installation restores the previous runtime.
|
||||
- Live API checks passed for uploaded audio analysis, permitted paths, blocked
|
||||
traversal/external paths/node IDs, strict settings values, same-origin writes,
|
||||
preference persistence, and preservation of installed runtime files.
|
||||
- The real browser showed the default-on setting and the details menu with the
|
||||
correct file/folder paths and manual cleanup instructions. Backend file edits
|
||||
override stale browser preferences. Frontend checks also covered rapid toggles
|
||||
and restoring the checkbox after a failed save.
|
||||
- Live Step TTS followed by a giggle edit succeeded through the Shared Runtime,
|
||||
including migration from the old dedicated API value. The final run produced
|
||||
non-silent 24 kHz audio lasting 3.12 seconds.
|
||||
- Restricted Dots latent statistics, the 3,060,182-entry Russian dictionary, NumPy
|
||||
data/DAC formats, and Demucs architecture metadata passed. A small real Demucs
|
||||
model restored its state through audio-separator's existing load_model entry
|
||||
point. Malicious pickle/NumPy/Torch payloads and model-constructor REDUCE calls
|
||||
were rejected. Dependency patches preserve global torch/importlib references.
|
||||
- Python source compilation and independent AST comparison preserved every
|
||||
existing neural model.eval() call. Required config list arithmetic remains usable.
|
||||
- The candidate archive includes intended uncommitted source without staging it:
|
||||
1,936 packaged files and 104 intentional exclusions. Runtime files, Unicode
|
||||
paths, user guides, and both metadata YAML files retain exact bytes. The auxiliary
|
||||
model registry reads its YAML successfully from the packaged layout.
|
||||
|
||||
Limits: Manager is absent from the test installation, so its real queue/restart
|
||||
flow was not exercised; the install.py hook and fresh runtime installation were
|
||||
tested directly. This does not establish every engine's inference/training resume,
|
||||
support arbitrary whole-object checkpoints, or constitute Registry approval.
|
||||
@@ -109,7 +109,7 @@ Decision
|
||||
|
||||
Operational Recommendation
|
||||
- Treat Qwen3-TTS as a legacy-stack engine for now.
|
||||
- Route it into the shared legacy Transformers 4 runtime used by compatible engines (the same runtime family already proven for Qwen in prior testing).
|
||||
- Route it into the shared legacy Transformers 4 runtime used by compatible engines (same family already proven for VibeVoice/Qwen in prior testing).
|
||||
- Keep the main environment on Transformers 5.
|
||||
- Revisit native Transformers 5 support only if Qwen3-TTS becomes strategically important enough to justify deeper behavior-level compatibility work.
|
||||
|
||||
|
||||
@@ -12,6 +12,10 @@ runtime_isolation_modes:
|
||||
label: "Shared Runtime"
|
||||
short_label: "Shared"
|
||||
description: "Use a secondary shared runtime for engines with dependency conflicts. This runtime reuses heavy base packages like PyTorch from the main environment when configured."
|
||||
dedicated_runtime:
|
||||
label: "Dedicated Runtime"
|
||||
short_label: "Dedicated"
|
||||
description: "Use a dedicated secondary runtime only for one engine when the shared runtime is not enough."
|
||||
|
||||
engines:
|
||||
- id: f5-tts
|
||||
@@ -372,6 +376,17 @@ engines:
|
||||
- "Multilingual (KugelAudio variants)"
|
||||
- "4-bit quantization"
|
||||
|
||||
runtime_isolation:
|
||||
default_mode: "shared_runtime"
|
||||
supported_modes:
|
||||
- "main_environment"
|
||||
- "shared_runtime"
|
||||
- "dedicated_runtime"
|
||||
status: "implemented"
|
||||
runtime_profile: "vibevoice_transformers4_shared"
|
||||
dedicated_profile: "vibevoice_transformers4_dedicated"
|
||||
notes: "Recommended on modern ComfyUI stacks because VibeVoice/Kugel currently needs a secondary dependency layer. The shared runtime reuses heavy base packages like PyTorch from the main environment to avoid full duplication."
|
||||
|
||||
model_sources:
|
||||
- component: "vibevoice-1.5B"
|
||||
source_name: "microsoft/VibeVoice-1.5B"
|
||||
@@ -672,9 +687,9 @@ engines:
|
||||
reference_free_tts: { supported: true, notes: "(zero-shot)" }
|
||||
|
||||
- id: indextts-2
|
||||
name: IndexTTS 2 / 2.5
|
||||
models: "IndexTTS-2, IndexTTS-2.5"
|
||||
size: "~4.7GB / ~5.49GB"
|
||||
name: IndexTTS-2
|
||||
models: "IndexTTS-2"
|
||||
size: "~4.7GB"
|
||||
license: "bilibili Model Use License"
|
||||
commercial: "conditional"
|
||||
|
||||
@@ -689,8 +704,6 @@ engines:
|
||||
- "Emotion Control: 8 vectors"
|
||||
- "Text as reference"
|
||||
- "Audio as reference"
|
||||
- "IndexTTS-2.5 official internal feature-duration scaling (not prosody planning)"
|
||||
- "IndexTTS-2.5 pronunciation annotations"
|
||||
|
||||
model_sources:
|
||||
- component: "IndexTTS-2"
|
||||
@@ -699,12 +712,6 @@ engines:
|
||||
size: "Multiple files"
|
||||
auto_download: true
|
||||
notes: "Main TTS engine"
|
||||
- component: "IndexTTS-2.5"
|
||||
source_name: "IndexTeam/IndexTTS-2.5"
|
||||
source_url: "https://huggingface.co/IndexTeam/IndexTTS-2.5"
|
||||
size: "~5.49GB"
|
||||
auto_download: true
|
||||
notes: "Multilingual backend with bundled codec and official feature-duration scaling"
|
||||
- component: "w2v-bert-2.0"
|
||||
source_name: "facebook/w2v-bert-2.0"
|
||||
source_url: "https://huggingface.co/facebook/w2v-bert-2.0"
|
||||
@@ -721,16 +728,16 @@ engines:
|
||||
en: { supported: true, flag: "🇺🇸", notes: "" }
|
||||
zh: { supported: true, flag: "🇨🇳", notes: "" }
|
||||
de: { supported: false, flag: "🇩🇪", notes: "" }
|
||||
es: { supported: true, flag: "🇪🇸", notes: "IndexTTS-2.5" }
|
||||
es: { supported: false, flag: "🇪🇸", notes: "" }
|
||||
fr: { supported: false, flag: "🇫🇷", notes: "" }
|
||||
it: { supported: false, flag: "🇮🇹", notes: "" }
|
||||
ja: { supported: true, flag: "🇯🇵", notes: "IndexTTS-2.5" }
|
||||
ja: { supported: true, flag: "🇯🇵", notes: "?" }
|
||||
ko: { supported: false, flag: "🇰🇷", notes: "" }
|
||||
ru: { supported: false, flag: "🇷🇺", notes: "" }
|
||||
pt: { supported: false, flag: "🇧🇷", notes: "" }
|
||||
pl: { supported: false, flag: "🇵🇱", notes: "" }
|
||||
hi: { supported: false, flag: "🇮🇳", notes: "" }
|
||||
ar: { supported: true, flag: "🇸🇦", notes: "IndexTTS-2.5" }
|
||||
ar: { supported: false, flag: "��", notes: "" }
|
||||
tr: { supported: false, flag: "🇹🇷", notes: "" }
|
||||
th: { supported: false, flag: "🇹🇭", notes: "" }
|
||||
no: { supported: false, flag: "🇳🇴", notes: "" }
|
||||
@@ -850,8 +857,10 @@ engines:
|
||||
supported_modes:
|
||||
- "main_environment"
|
||||
- "shared_runtime"
|
||||
- "dedicated_runtime"
|
||||
status: "implemented"
|
||||
runtime_profile: "vibevoice_transformers4_shared"
|
||||
dedicated_profile: "qwen3_tts_transformers4_dedicated"
|
||||
notes: "Recommended on modern ComfyUI stacks because Qwen3-TTS, Qwen3-ASR, and the reused Qwen forced aligner currently rely on the shared legacy T4 runtime for stable operation."
|
||||
|
||||
model_sources:
|
||||
@@ -1044,35 +1053,19 @@ engines:
|
||||
asr: false
|
||||
training: false
|
||||
|
||||
runtime_isolation:
|
||||
default_mode: "shared_runtime"
|
||||
supported_modes:
|
||||
- "main_environment"
|
||||
- "shared_runtime"
|
||||
status: "implemented"
|
||||
runtime_profile: "vibevoice_transformers4_shared"
|
||||
notes: "Shared Transformers 4 is recommended; the main Transformers 5 stack can produce invalid audio tokens."
|
||||
|
||||
special_features:
|
||||
- "Second Pass Speech Editing Node: 14 emotions"
|
||||
- "32 speaking styles"
|
||||
- "Dated checkpoints: 2026-01-23 / 2025-11-28"
|
||||
- "Expanded January paralinguistic effects with legacy tag compatibility"
|
||||
- "Selectable main or shared Python runtime (shared Transformers 4 runtime recommended)"
|
||||
- "Paralinguistic effects"
|
||||
- "Selectable main, shared, or dedicated Python runtime (shared Transformers 4 runtime recommended)"
|
||||
|
||||
model_sources:
|
||||
- component: "Step-Audio-EditX-2026-01-23"
|
||||
- component: "Step-Audio-EditX"
|
||||
source_name: "stepfun-ai/Step-Audio-EditX"
|
||||
source_url: "https://huggingface.co/stepfun-ai/Step-Audio-EditX"
|
||||
size: "~7GB"
|
||||
auto_download: true
|
||||
notes: "Default 3B checkpoint; pinned to 5fe2f8a05c2353301ad47d3c1747b262115da138"
|
||||
- component: "Step-Audio-EditX-2025-11-28"
|
||||
source_name: "stepfun-ai/Step-Audio-EditX"
|
||||
source_url: "https://huggingface.co/stepfun-ai/Step-Audio-EditX/tree/7f3de603ae46c96dff6f06f47b1d5a45aabd34fe"
|
||||
size: "~7GB"
|
||||
auto_download: true
|
||||
notes: "Legacy checkpoint; pinned November weights in a separate folder"
|
||||
notes: "Main 3B audio editing model"
|
||||
- component: "Step-Audio-Tokenizer"
|
||||
source_name: "stepfun-ai/Step-Audio-Tokenizer"
|
||||
source_url: "https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer"
|
||||
@@ -1216,6 +1209,7 @@ engines:
|
||||
default_mode: "main_environment"
|
||||
main_environment: true
|
||||
shared_runtime: false
|
||||
dedicated_runtime: false
|
||||
|
||||
special_features:
|
||||
- "Free-form sub-word emotion/prosody tags"
|
||||
@@ -1345,6 +1339,110 @@ engines:
|
||||
speed_performance: { supported: "partial", notes: "Moderate; mf variant is faster" }
|
||||
reference_free_tts: { supported: true, notes: "(default speaker)" }
|
||||
|
||||
- id: tada
|
||||
name: TADA
|
||||
models: "TADA 1B (English), TADA 3B-ML"
|
||||
size: "~6.1GB / ~11GB with selected codec assets"
|
||||
license: "MIT code / Llama 3.2 weights"
|
||||
commercial: conditional
|
||||
|
||||
capabilities:
|
||||
tts: true
|
||||
srt: true
|
||||
vc: false
|
||||
asr: false
|
||||
training: false
|
||||
|
||||
special_features:
|
||||
- "Zero-shot voice cloning"
|
||||
- "Reference-conditioned expression transfer"
|
||||
|
||||
readme_key_features:
|
||||
- "Voice cloning"
|
||||
- "Reference-conditioned expression transfer"
|
||||
|
||||
runtime_isolation:
|
||||
default_mode: "shared_runtime"
|
||||
supported_modes:
|
||||
- "shared_runtime"
|
||||
status: "implemented"
|
||||
runtime_profile: "vibevoice_transformers4_shared"
|
||||
notes: "The supported path uses the shared Transformers 4 runtime with hume-tada 0.1.9. TADA is incompatible with the suite's main Transformers 5 environment."
|
||||
|
||||
model_sources:
|
||||
- component: "TADA 1B"
|
||||
source_name: "HumeAI/tada-1b"
|
||||
source_url: "https://huggingface.co/HumeAI/tada-1b"
|
||||
size: "~3.9GB"
|
||||
auto_download: true
|
||||
notes: "English-only model; downloaded only when selected"
|
||||
- component: "TADA 3B-ML"
|
||||
source_name: "HumeAI/tada-3b-ml"
|
||||
source_url: "https://huggingface.co/HumeAI/tada-3b-ml"
|
||||
size: "~8.9GB"
|
||||
auto_download: true
|
||||
notes: "Multilingual model; downloaded only when selected"
|
||||
- component: "TADA codec"
|
||||
source_name: "HumeAI/tada-codec"
|
||||
source_url: "https://huggingface.co/HumeAI/tada-codec"
|
||||
size: "~2.15GB per selected language"
|
||||
auto_download: true
|
||||
notes: "Downloads shared encoder/decoder plus only the aligner required by the selected language"
|
||||
- component: "Llama tokenizer"
|
||||
source_name: "onnx-community/Llama-3.2-1B"
|
||||
source_url: "https://huggingface.co/onnx-community/Llama-3.2-1B"
|
||||
size: "Tokenizer files only"
|
||||
auto_download: true
|
||||
notes: "Ungated redistribution; runtime files are verified against Meta's official Git blob hashes and remain under the Llama 3.2 Community License"
|
||||
- component: "Wav2Vec aligner configuration"
|
||||
source_name: "facebook/wav2vec2-large"
|
||||
source_url: "https://huggingface.co/facebook/wav2vec2-large"
|
||||
size: "Configuration file only"
|
||||
auto_download: true
|
||||
notes: "Downloads config.json only; no Wav2Vec model weights"
|
||||
|
||||
languages:
|
||||
en: { supported: true, flag: "🇺🇸", notes: "(1B and 3B-ML)" }
|
||||
zh: { supported: true, flag: "🇨🇳", notes: "(3B-ML; backend code `ch`)" }
|
||||
de: { supported: true, flag: "🇩🇪", notes: "(3B-ML)" }
|
||||
es: { supported: true, flag: "🇪🇸", notes: "(3B-ML)" }
|
||||
fr: { supported: true, flag: "🇫🇷", notes: "(3B-ML)" }
|
||||
it: { supported: true, flag: "🇮🇹", notes: "(3B-ML)" }
|
||||
ja: { supported: true, flag: "🇯🇵", notes: "(3B-ML)" }
|
||||
ko: { supported: false, flag: "🇰🇷", notes: "" }
|
||||
ru: { supported: false, flag: "🇷🇺", notes: "" }
|
||||
pt: { supported: true, flag: "🇵🇹", notes: "(3B-ML)" }
|
||||
pl: { supported: true, flag: "🇵🇱", notes: "(3B-ML)" }
|
||||
hi: { supported: false, flag: "🇮🇳", notes: "" }
|
||||
ar: { supported: true, flag: "🇦🇪", notes: "(3B-ML)" }
|
||||
tr: { supported: false, flag: "🇹🇷", notes: "" }
|
||||
th: { supported: false, flag: "🇹🇭", notes: "" }
|
||||
no: { supported: false, flag: "🇳🇴", notes: "" }
|
||||
vi: { supported: false, flag: "🇻🇳", notes: "" }
|
||||
hy: { supported: false, flag: "🇦🇲", notes: "" }
|
||||
ka: { supported: false, flag: "🇬🇪", notes: "" }
|
||||
da: { supported: false, flag: "🇩🇰", notes: "" }
|
||||
fi: { supported: false, flag: "🇫🇮", notes: "" }
|
||||
el: { supported: false, flag: "🇬🇷", notes: "" }
|
||||
he: { supported: false, flag: "🇮🇱", notes: "" }
|
||||
ms: { supported: false, flag: "🇲🇾", notes: "" }
|
||||
nl: { supported: false, flag: "🇳🇱", notes: "" }
|
||||
sv: { supported: false, flag: "🇸🇪", notes: "" }
|
||||
sw: { supported: false, flag: "🇰🇪", notes: "" }
|
||||
|
||||
features:
|
||||
voice_cloning: { supported: true, notes: "Reference audio plus exact transcript" }
|
||||
reference_transcript: { requirement: required }
|
||||
native_multi_speaker: { supported: false, notes: "" }
|
||||
voice_conversion: { supported: false, notes: "" }
|
||||
asr_transcribe: { supported: false, notes: "" }
|
||||
emotion_control: { supported: false, notes: "No direct emotion control; expression may transfer from the reference audio" }
|
||||
native_long_form: { supported: false, notes: "Suite text chunking is used" }
|
||||
community_finetunes: { supported: false, notes: "No suite integration" }
|
||||
vram_efficient: { supported: "partial", notes: "1B is the lighter option; suite VRAM measurements are pending" }
|
||||
speed_performance: { supported: "partial", notes: "No suite benchmark; no streaming or batch inference" }
|
||||
reference_free_tts: { supported: false, notes: "This integration requires a reference voice and exact transcript" }
|
||||
|
||||
- id: dramabox
|
||||
name: DramaBox
|
||||
models: "DramaBox 3.3B"
|
||||
@@ -1357,7 +1455,7 @@ engines:
|
||||
srt: true
|
||||
vc: false
|
||||
asr: false
|
||||
training: true
|
||||
training: false
|
||||
|
||||
special_features:
|
||||
- "Expressive scene prompting and stage directions"
|
||||
@@ -1368,7 +1466,6 @@ engines:
|
||||
- "Explicit generation/reference durations, CFG rescale control, and optional Perth watermark"
|
||||
- "Experimental staged and sequential strategies for lowering peak VRAM; official LTX FP8-cast storage"
|
||||
- "Optional official torch.compile path"
|
||||
- "Official audio-branch IC-LoRA training workflow"
|
||||
|
||||
model_sources:
|
||||
- component: "DramaBox DiT + audio components"
|
||||
@@ -1422,8 +1519,8 @@ engines:
|
||||
emotion_control: { supported: true, notes: "Natural-language scene prompt and stage directions" }
|
||||
native_long_form: { supported: true, notes: "Official duration-aware quote-group chunking; ~37s target / 45s cap" }
|
||||
native_srt_duration_targeting: { supported: true, notes: "Subtitle duration is passed as gen_duration before the selected SRT timing mode applies final correction" }
|
||||
community_finetunes: { supported: true, notes: "Official audio-branch IC-LoRA adapters can be trained and loaded" }
|
||||
vram_efficient: { supported: true, notes: "Fast mode is ~24GB; experimental FP8, staged, and sequential options can reduce VRAM, but practical peak usage varies by environment and loaded components" }
|
||||
community_finetunes: { supported: false, notes: "Not integrated" }
|
||||
vram_efficient: { supported: true, notes: "Fast mode is ~24GB; experimental FP8 peaks on RTX 4090: staged ~15.1GB allocated, sequential ~11.7GB allocated / ~12.4GB reserved" }
|
||||
speed_performance: { supported: "partial", notes: "Fast favors repeated generation; staged reloads temporary components; sequential also transfers the transformer; optional block-level torch.compile accelerates repeat denoising" }
|
||||
reference_free_tts: { supported: true, notes: "Voice reference is optional" }
|
||||
|
||||
@@ -1502,7 +1599,7 @@ engines:
|
||||
|
||||
- id: moss-tts
|
||||
name: MOSS-TTS
|
||||
models: "Local 1.7B, Delay 8B v1.5/1.0, LAION Voice Acting 8B community fine-tune, VoiceGenerator 1.7B, SoundEffect 8B v1, TTSD 8B"
|
||||
models: "Local 1.7B, Delay 8B v1.5/1.0, VoiceGenerator 1.7B, SoundEffect 8B v1, TTSD 8B"
|
||||
size: "~8.5GB tokenizer + ~6.1GB/17GB/18GB model"
|
||||
license: "Apache-2.0"
|
||||
commercial: true
|
||||
@@ -1520,8 +1617,6 @@ engines:
|
||||
- "Reference-free voice design with MOSS-VoiceGenerator"
|
||||
- "Native 1-5 speaker TTSD dialogue"
|
||||
- "31-language generation with MOSS-TTS-v1.5"
|
||||
- "Optional LAION community 8B voice-acting fine-tune"
|
||||
- "Config-based discovery of compatible local MOSS full checkpoints"
|
||||
- "Prompt-only sound-effect generation with MOSS-SoundEffect v1"
|
||||
- "Long-form generation (TTSD/Delay)"
|
||||
- "Duration token hint"
|
||||
@@ -1547,12 +1642,6 @@ engines:
|
||||
size: "~17GB"
|
||||
auto_download: true
|
||||
notes: "Current official 8B delay model with 31 languages and more stable voice cloning"
|
||||
- component: "MOSS-TTS v1.5 Voice Acting 8B (Community - LAION)"
|
||||
source_name: "laion/moss-tts-v1.5-8b-voice-acting"
|
||||
source_url: "https://huggingface.co/laion/moss-tts-v1.5-8b-voice-acting"
|
||||
size: "~17GB"
|
||||
auto_download: true
|
||||
notes: "Third-party full MOSS-TTS v1.5 fine-tune for expressive voice acting; not an official OpenMOSS model"
|
||||
- component: "MOSS-VoiceGenerator"
|
||||
source_name: "OpenMOSS-Team/MOSS-VoiceGenerator"
|
||||
source_url: "https://huggingface.co/OpenMOSS-Team/MOSS-VoiceGenerator"
|
||||
@@ -1615,7 +1704,7 @@ engines:
|
||||
asr_transcribe: { supported: false, notes: "" }
|
||||
emotion_control: { supported: true, notes: "(MOSS-VoiceGenerator instruction-conditioned voice design)" }
|
||||
native_long_form: { supported: true, notes: "(TTSD/Delay long-form; use chunk orchestration for very long inputs)" }
|
||||
community_finetunes: { supported: true, notes: "(Compatible full local checkpoints, LAION Voice Acting 8B auto-download, and LoRA adapter inference/training supported)" }
|
||||
community_finetunes: { supported: true, notes: "(LoRA adapter inference supported; initial integrated LoRA training support added for MOSS-TTS Delay 8B)" }
|
||||
vram_efficient: { supported: "partial", notes: "(Local 1.7B smaller; tokenizer is large)" }
|
||||
speed_performance: { supported: true, notes: "Fast with CUDA/FlashAttention" }
|
||||
reference_free_tts: { supported: true, notes: "(direct TTS and prompt-only generation)" }
|
||||
@@ -1919,13 +2008,15 @@ table_notes:
|
||||
Conditional means the transcript is required only for the specific mode:
|
||||
CosyVoice3 zero-shot, Qwen3-TTS full Base cloning, or MOSS-TTSD cloned-speaker
|
||||
dialogue. Higgs Audio 2, Higgs Audio v3, and Dots TTS accept matching text
|
||||
when provided but do not require it.
|
||||
when provided but do not require it. TADA always requires an exact transcript
|
||||
for its reference audio in this integration.
|
||||
language_support:
|
||||
- "**CosyVoice3 Chinese**: Includes 18+ dialects (Cantonese, Sichuan, Dongbei, Shanghai, etc.)"
|
||||
- "**Higgs Audio 2**: Trained on EN, ZH (Mandarin), KO, DE, ES (English majority) - 10M hours AudioVerse dataset"
|
||||
- "**Higgs Audio v3**: Official model card lists 102 languages total: 85 polished-production languages and 17 usable but less polished languages."
|
||||
- "**IndexTTS-2**: Trained on 55K+ hours - ZH, EN, JA primary."
|
||||
- "**MOSS-TTS**: Official list includes ZH, EN, DE, ES, FR, JA, IT, HU, KO, RU, FA, AR, PL, PT, CS, DA, SV, EL, TR; HU/FA/CS are not separate columns in this matrix."
|
||||
- "**TADA**: TADA 1B is English-only. TADA 3B-ML supports EN, AR, ZH, DE, ES, FR, IT, JA, PL, and PT; the upstream backend calls Chinese `ch`."
|
||||
- "**OmniVoice**: Official model supports 600+ languages; this matrix only shows the suite's comparison subset."
|
||||
- "**RVC**: Language-agnostic voice conversion/post-processing; it is marked supported for every language row."
|
||||
- "Qwen3-TTS: Supports pt-BR only with instruction (voice design or harcoded voices). Base can't do instructions, so it will always ouput pt-PT."
|
||||
@@ -1966,7 +2057,7 @@ readme_model_download_table:
|
||||
- engine: "Step Audio EditX"
|
||||
primary_model_path: "ComfyUI/models/TTS/step_audio_editx/"
|
||||
auto_download: "✅"
|
||||
notes: "Lazy dated checkpoints + tokenizer stack; unversioned installs retained"
|
||||
notes: "Main model + tokenizer stack"
|
||||
- engine: "CosyVoice3"
|
||||
primary_model_path: "ComfyUI/models/TTS/CosyVoice/"
|
||||
auto_download: "✅"
|
||||
@@ -1995,10 +2086,14 @@ readme_model_download_table:
|
||||
primary_model_path: "ComfyUI/models/TTS/dots_tts/"
|
||||
auto_download: "✅"
|
||||
notes: "Official base / soar / mf checkpoints with tokenizer, vocoder, speaker encoder"
|
||||
- engine: "TADA"
|
||||
primary_model_path: "ComfyUI/models/TTS/tada/"
|
||||
auto_download: "✅"
|
||||
notes: "Selected 1B or 3B-ML model plus language-specific codec assets and an exact hash-verified ungated Llama tokenizer redistribution"
|
||||
- engine: "DramaBox"
|
||||
primary_model_path: "ComfyUI/models/TTS/dramabox/DramaBox/"
|
||||
auto_download: "✅"
|
||||
notes: "~16.4GB download; fast mode roughly 24GB VRAM; experimental FP8, staged, and sequential options can reduce VRAM, but no minimum GPU size is guaranteed; conditional LTX-2 Community License"
|
||||
notes: "~16.4GB download; fast mode roughly 24GB VRAM; experimental FP8 peaks on RTX 4090: staged ~15.1GB allocated, sequential ~11.7GB allocated / ~12.4GB reserved; conditional LTX-2 Community License"
|
||||
- engine: "Fish Audio S2 Pro"
|
||||
primary_model_path: "ComfyUI/models/TTS/fish_audio_s2_pro/"
|
||||
auto_download: "✅"
|
||||
@@ -2205,20 +2300,14 @@ model_layouts_markdown: |
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/step_audio_editx/
|
||||
├── Step-Audio-EditX-2026-01-23/
|
||||
│ └── CosyVoice-300M-25Hz/
|
||||
├── Step-Audio-EditX-2025-11-28/ # downloaded only when selected
|
||||
│ └── CosyVoice-300M-25Hz/
|
||||
├── Step-Audio-EditX/ # existing unversioned installations
|
||||
├── Step-Audio-EditX/
|
||||
│ └── CosyVoice-300M-25Hz/
|
||||
└── FunASR-Paraformer/
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Selected checkpoints and tokenizer assets auto-download at pinned revisions.
|
||||
- Existing unversioned/local weights are retained and folders are not relabeled; known config compatibility fields may be normalized.
|
||||
- See [Step Audio EditX checkpoints](STEP_AUDIO_EDITX_CHECKPOINTS.md) for selection and expanded tags.
|
||||
- Main model, tokenizer assets, and speech stack auto-download.
|
||||
|
||||
## Higgs Audio v3
|
||||
|
||||
@@ -2244,7 +2333,7 @@ model_layouts_markdown: |
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/dramabox/
|
||||
├── DramaBox/
|
||||
└── DramaBox/
|
||||
├── dramabox-dit-v1.safetensors
|
||||
├── dramabox-audio-components.safetensors
|
||||
├── assets/
|
||||
@@ -2255,10 +2344,6 @@ model_layouts_markdown: |
|
||||
├── model-00002-of-00002.safetensors
|
||||
├── model.safetensors.index.json
|
||||
└── tokenizer and processor files...
|
||||
└── loras/
|
||||
└── <adapter_name>/
|
||||
├── adapter_config.json
|
||||
└── adapter_model.safetensors
|
||||
```
|
||||
|
||||
Notes:
|
||||
@@ -2266,8 +2351,6 @@ model_layouts_markdown: |
|
||||
- Both repositories download directly into the organized suite folder.
|
||||
- Transformers is forced into local-only loading after download.
|
||||
- Requires NVIDIA CUDA. Fast mode targets approximately 24GB VRAM; experimental staged/sequential modes can run with less.
|
||||
- Integrated training exports LoRA adapters into `dramabox/loras/<adapter_name>/`.
|
||||
- Training jobs, normalized datasets, preprocessing caches, checkpoints, and logs are stored under `ComfyUI/output/tts_audio_suite_training/dramabox/`.
|
||||
- The LTX-2 Community License requires a paid license for entities with at
|
||||
least USD 10 million in annual revenue.
|
||||
|
||||
@@ -2312,7 +2395,6 @@ model_layouts_markdown: |
|
||||
ComfyUI/models/TTS/moss_tts/
|
||||
├── MOSS-TTS-Local-Transformer/
|
||||
├── MOSS-TTS-v1.5/
|
||||
├── moss-tts-v1.5-8b-voice-acting/ # Community - LAION
|
||||
├── MOSS-TTS/
|
||||
├── MOSS-VoiceGenerator/
|
||||
├── MOSS-SoundEffect/
|
||||
@@ -2329,8 +2411,6 @@ model_layouts_markdown: |
|
||||
- `MOSS-Audio-Tokenizer` is required by the official TTS and TTSD variants.
|
||||
- `MOSS-TTS-Local-Transformer` is the smaller 1.7B model.
|
||||
- `MOSS-TTS-v1.5` is the current 8B delay model with 31-language support.
|
||||
- `moss-tts-v1.5-8b-voice-acting` is an optional third-party LAION full fine-tune for expressive speech, not an official OpenMOSS model.
|
||||
- Other compatible full checkpoints placed here are discovered from their `config.json`; unsupported MOSS architectures are rejected explicitly.
|
||||
- `MOSS-TTS` is the legacy official 8B delay model.
|
||||
- `MOSS-VoiceGenerator` is the 1.7B voice-design provider used by Voice Designer.
|
||||
- `MOSS-SoundEffect` is the v1 sound-effect checkpoint used through the MOSS-TTS engine and 🌩️ Sound Effects.
|
||||
@@ -2453,6 +2533,28 @@ model_layouts_markdown: |
|
||||
- Native sample rate is 48kHz.
|
||||
- Main-environment support works on Transformers 5; on Windows, `normalize_text` falls back to no-op if `WeTextProcessing` is unavailable.
|
||||
|
||||
## TADA
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/tada/
|
||||
├── TADA-1B/ # downloaded only if selected
|
||||
├── TADA-3B-ML/ # downloaded only if selected
|
||||
├── tada-codec/
|
||||
│ ├── encoder/
|
||||
│ ├── decoder/
|
||||
│ ├── aligner[-<language>]/ # selected language only
|
||||
│ └── wav2vec2-large/
|
||||
└── llama-3.2-1b-tokenizer/ # tokenizer files only
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- TADA runs in the shared Transformers 4 runtime with `hume-tada==0.1.9`.
|
||||
- The selected 1B or 3B-ML model, shared codec components, and selected language aligner download on demand.
|
||||
- The Llama tokenizer downloads from an ungated redistribution and is verified against Meta's official Git blob hashes. It remains under the Llama 3.2 Community License.
|
||||
- Reference audio and its exact transcript are required by this integration.
|
||||
- Native output is mono 24kHz.
|
||||
|
||||
## OmniVoice
|
||||
|
||||
```text
|
||||
|
||||
@@ -7,21 +7,22 @@
|
||||
| **F5-TTS** | Main | Base, v1, E2TTS + 8 lang models | ~1.2GB each | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | CC-BY-NC-4.0 | Targeted Word/Speech Editing, Speed control | 10 |
|
||||
| **ChatterBox** | Main | EN, DE×3, IT, FR, RU, HY, KA, JA, KO, NO | ~4.3GB | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | MIT | Expressiveness slider | 10 |
|
||||
| **ChatterBox 23L** | Main | v1, v2, v3, Vietnamese (Viterbox), Egyptian Arabic (oddadmix) | ~4.3GB | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | MIT | V1, V2, and V3 official checkpoints, Emotion tokens (v2; currently ineffective), V3 skips the legacy alignment analyzer and trims the final token artifact | 25 |
|
||||
| **VibeVoice** | Main | 1.5B, 7B, KugelAudio-0 (7B), kugel-2 (7B), Hindi-1.5B/7B | 5.4GB / 18GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | MIT (research-only per model card) | 90-min long-form, Native 4-speaker (Base models), Multilingual (KugelAudio variants), 4-bit quantization | 27 |
|
||||
| **VibeVoice** | Shared | 1.5B, 7B, KugelAudio-0 (7B), kugel-2 (7B), Hindi-1.5B/7B | 5.4GB / 18GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | MIT (research-only per model card) | 90-min long-form, Native 4-speaker (Base models), Multilingual (KugelAudio variants), 4-bit quantization | 27 |
|
||||
| **Higgs Audio 2** | Shared | 3B | ~9GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Boson Higgs Audio 2 Community License | 3 multi-speaker, CUDA graphs (55+ tokens/sec) | 5 |
|
||||
| **Higgs Audio v3** | Main | 4B | ~8GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Boson Higgs Audio v3 Research and Non-Commercial License | Native inline emotion/style/prosody/SFX tags, Zero-shot voice cloning, 100+ language support | 100+ |
|
||||
| **IndexTTS 2 / 2.5** | Main | IndexTTS-2, IndexTTS-2.5 | ~4.7GB / ~5.49GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | bilibili Model Use License | Emotion Control: 8 vectors, Text as reference, Audio as reference, IndexTTS-2.5 official internal feature-duration scaling (not prosody planning), IndexTTS-2.5 pronunciation annotations | 5 |
|
||||
| **IndexTTS-2** | Main | IndexTTS-2 | ~4.7GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | bilibili Model Use License | Emotion Control: 8 vectors, Text as reference, Audio as reference | 3 |
|
||||
| **CosyVoice3** | Main | 0.5B, 0.5B-RL | ~5.4GB | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | Apache-2.0 | Paralinguistic tags | 4 |
|
||||
| **Qwen3-TTS** | Shared | 0.6B, 1.7B (CustomVoice/VoiceDesign/Base) | ~3-6GB | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | Apache-2.0 | Voice design, ASR (Automatic Speech Recognition) | 10 |
|
||||
| **Granite ASR** | Main | granite-4.0-1b-speech, granite-speech-4.1-2b, granite-speech-4.1-2b-plus | ~4.6GB | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | Apache-2.0 | Native speaker attribution / diarization (plus model variant), Native word-level timestamps (plus model variant), ASR (Automatic Speech Recognition), Custom timestamps/SRT via reused Qwen forced aligner, Speech translation (experimental), Optional forced aligner auto-routed through shared legacy T4 runtime | 6 |
|
||||
| **Step Audio EditX** | Shared | 3B LLM + CosyVoice | ~7GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 (verify before commercial use) | Second Pass Speech Editing Node: 14 emotions, 32 speaking styles, Dated checkpoints: 2026-01-23 / 2025-11-28, Expanded January paralinguistic effects with legacy tag compatibility, Selectable main or shared Python runtime (shared Transformers 4 runtime recommended) | 4 |
|
||||
| **Step Audio EditX** | Main | 3B LLM + CosyVoice | ~7GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 (verify before commercial use) | Second Pass Speech Editing Node: 14 emotions, 32 speaking styles, Paralinguistic effects, Selectable main, shared, or dedicated Python runtime (shared Transformers 4 runtime recommended) | 4 |
|
||||
| **Echo-TTS** | Main | echo-tts-base + fish-s1-dac-min | ~5.3GB + ~1.8GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | CC-BY-NC-SA-4.0 | Diffusion-based (~30s best), Force Speaker KV (speaker drift control) | 1 |
|
||||
| **Fish Audio S2 Pro** | Main | S2 Pro 4B / FP8 | ~10.3GB / ~8.0GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Fish Audio Research License | Free-form sub-word emotion/prosody tags, Native multi-speaker and multi-turn dialogue with dynamic speaker references, Zero-shot voice cloning, Optional per-segment custom character switching, Configurable 4K-32K native context with reduced KV-cache VRAM, Optional community FP8 weight-only checkpoint with BF16 activations, Optional on-the-fly BitsAndBytes INT8/NF4 for the official checkpoint | 80+ languages |
|
||||
| **Dots TTS** | Main | dots.tts-base, dots.tts-soar, dots.tts-mf | ~6GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 | Official auto language detect / language control, SOAR and MeanFlow distilled variants | 19 |
|
||||
| **DramaBox** | Main | DramaBox 3.3B | ~16.4GB | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | LTX-2 Community License | Expressive scene prompting and stage directions, Native and SRT-aware duration targeting, Official duration-aware long-form chunking with scene-prefix preservation, Optional 10-second zero-shot voice reference, CFG negative prompt with per-segment switching, Explicit generation/reference durations, CFG rescale control, and optional Perth watermark, Experimental staged and sequential strategies for lowering peak VRAM; official LTX FP8-cast storage, Optional official torch.compile path, Official audio-branch IC-LoRA training workflow | 1 |
|
||||
| **TADA** | Shared | TADA 1B (English), TADA 3B-ML | ~6.1GB / ~11GB with selected codec assets | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | MIT code / Llama 3.2 weights | Zero-shot voice cloning, Reference-conditioned expression transfer | 10 |
|
||||
| **DramaBox** | Main | DramaBox 3.3B | ~16.4GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | LTX-2 Community License | Expressive scene prompting and stage directions, Native and SRT-aware duration targeting, Official duration-aware long-form chunking with scene-prefix preservation, Optional 10-second zero-shot voice reference, CFG negative prompt with per-segment switching, Explicit generation/reference durations, CFG rescale control, and optional Perth watermark, Experimental staged and sequential strategies for lowering peak VRAM; official LTX FP8-cast storage, Optional official torch.compile path | 1 |
|
||||
| **OmniVoice** | Main | OmniVoice | ~3.7GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 | Inline non-verbal tags and pronunciation overrides, Reference-free voice design, 600+ language support, Upstream long-form chunk orchestration | 600+ |
|
||||
| **MOSS-TTS** | Main | Local 1.7B, Delay 8B v1.5/1.0, LAION Voice Acting 8B community fine-tune, VoiceGenerator 1.7B, SoundEffect 8B v1, TTSD 8B | ~8.5GB tokenizer + ~6.1GB/17GB/18GB model | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | Apache-2.0 | Reference-free voice design with MOSS-VoiceGenerator, Native 1-5 speaker TTSD dialogue, 31-language generation with MOSS-TTS-v1.5, Optional LAION community 8B voice-acting fine-tune, Config-based discovery of compatible local MOSS full checkpoints, Prompt-only sound-effect generation with MOSS-SoundEffect v1, Long-form generation (TTSD/Delay), Duration token hint, Local/Delay/TTSD variants, Initial integrated LoRA training workflow (Delay 8B) | 24 |
|
||||
| **MOSS-TTS** | Main | Local 1.7B, Delay 8B v1.5/1.0, VoiceGenerator 1.7B, SoundEffect 8B v1, TTSD 8B | ~8.5GB tokenizer + ~6.1GB/17GB/18GB model | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | Apache-2.0 | Reference-free voice design with MOSS-VoiceGenerator, Native 1-5 speaker TTSD dialogue, 31-language generation with MOSS-TTS-v1.5, Prompt-only sound-effect generation with MOSS-SoundEffect v1, Long-form generation (TTSD/Delay), Duration token hint, Local/Delay/TTSD variants, Initial integrated LoRA training workflow (Delay 8B) | 24 |
|
||||
| **MOSS-SoundEffect v2** | Main | MOSS-SoundEffect-v2.0 | ~11.2GB | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | Apache-2.0 | Durations up to 30 seconds, Native negative prompting, CFG, flow shift, and diffusion-step controls, Prompt-only text-to-sound generation, 48 kHz mono output, Seeded generation | 2 |
|
||||
| **RVC** | Main | Community .pth | 100-300MB | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | MIT (framework); community models vary | Real-time VC, Integrated training workflow, Pitch shift (±14), 6 HuBERT models, Language-independent | Any |
|
||||
|
||||
*Isolation column: `Main` runs in the main ComfyUI environment. `Shared` uses a shared secondary runtime reused by multiple engines.*
|
||||
*Isolation column: `Main` runs in the main ComfyUI environment. `Shared` uses a shared secondary runtime reused by multiple engines. `Dedicated` uses an engine-specific secondary runtime.*
|
||||
+18
-18
@@ -2,22 +2,22 @@
|
||||
|
||||
## Feature Comparison Matrix
|
||||
|
||||
| Feature | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS 2 / 2.5 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | DramaBox | OmniVoice | MOSS-TTS | MOSS-SoundEffect v2 | RVC |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| **TTS** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| **SRT** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| **Voice Conversion** | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| **ASR (Transcribe)** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Sound Effects** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ |
|
||||
| **Training** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ✅ |
|
||||
| **Voice Cloning** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (Base model) | ❌ | ✅ | ✅ | ✅ Reference audio plus exact transcript | ✅ | ✅ Optional reference audio; configurable 3-30 second window from the beginning of the source, default 10 seconds | ✅ | ✅ | ❌ | ⚠️ (needs training) |
|
||||
| **Reference Transcript†** | **Required** | Not used | Not used | Not used | Optional | Optional | Not used | Conditional | Conditional | N/A | **Required** | Not used | **Required** | Optional | Not used | **Required** | Conditional | Not used | N/A |
|
||||
| **Native Multi-Speaker** | ❌ | ❌ | ❌ | ✅ (Base only, Kugel uses fallback) | ✅ | ❌ | ❌ | ❌ | ❌ | ⚠️ (Plus variant speaker attribution / diarization) | ❌ | ❌ | ✅ Native dialogue is default; optional custom mode generates each [Character] segment independently as local speaker 0 | ❌ | ❌ | ❌ | ✅ (TTSD v1.0; 1-5 speakers) | ❌ | ❌ |
|
||||
| **Emotion Control** | ❌ | ❌ | ⚠️ (v2 tags - doesn't work) | ❌ | ⚠️ (via prompt) | ✅ (native inline tags) | ✅ (8 emotions) | ⚠️ (via instruct) | ⚠️ (via instruct) | ❌ | ✅ (14 emotions) | ❌ | ✅ Free-form inline natural-language tags | ❌ | ✅ Natural-language scene prompt and stage directions | ⚠️ (voice-design instruct + inline non-verbal tags) | ✅ (MOSS-VoiceGenerator instruction-conditioned voice design) | ❌ | ❌ |
|
||||
| **Native Long-form** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Configurable 4K-32K native context; suite text chunking is bypassed | ❌ | ✅ Official duration-aware quote-group chunking; ~37s target / 45s cap | ✅ (uses upstream audio_chunk_duration / audio_chunk_threshold orchestration; bypasses suite char-based chunk splitting) | ✅ (TTSD/Delay long-form; use chunk orchestration for very long inputs) | ❌ | N/A |
|
||||
| **Community Finetunes** | ✅ | ✅ | ✅ | ✅ KugelAudio, Hindi | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Official audio-branch IC-LoRA adapters can be trained and loaded | ❌ | ✅ (Compatible full local checkpoints, LAION Voice Acting 8B auto-download, and LoRA adapter inference/training supported) | ❌ | ✅ |
|
||||
| **VRAM Efficient** | ✅ | ✅ | ✅ | ⚠️ (5-18GB) | ⚠️ (9GB) | ⚠️ (~8-10GB) | ⚠️ (9-12GB) | ✅ (5.4GB) | ✅ (3-6GB) | ✅ (~4.6GB) | ⚠️ (7GB) | ⚠️ (~7GB total) | ⚠️ 8K context measured at ~15.2GB BF16, ~11.2GB FP8, ~11.2GB BNB INT8, or ~8.9GB BNB NF4; BF16 codec and activations; BNB is a load-time option for the official checkpoint | ⚠️ (main env works; 2B-class model is not lightweight) | ✅ Fast mode is ~24GB; experimental FP8, staged, and sequential options can reduce VRAM, but practical peak usage varies by environment and loaded components | ⚠️ (main env works; 2B-class model is not lightweight) | ⚠️ (Local 1.7B smaller; tokenizer is large) | ⚠️ (runs in the main ComfyUI environment but remains GPU-heavy) | ✅ |
|
||||
| **Speed/Performance** | ✅ Very Fast | ✅ Fast | ✅ Fast | ⚠️ | ⚠️ | ⚠️ CUDA recommended | ⚠️ | ✅ Fast | ⚠️ | ⚠️ Moderate | ⚠️ | ✅ Fast (diffusion, realtime-capable) | ✅ Main-environment subprocess with reliable teardown; local compile measurements: ~40 it/s BF16, ~11.8 it/s NF4 at ~8.9GB VRAM, and ~3.7 it/s INT8 at ~11.2GB VRAM; quality comparison pending | ⚠️ Moderate; mf variant is faster | ⚠️ Fast favors repeated generation; staged reloads temporary components; sequential also transfers the transformer; optional block-level torch.compile accelerates repeat denoising | ✅ Fast; upstream reports sub-realtime RTF | ✅ Fast with CUDA/FlashAttention | ⚠️ (100 diffusion steps by default) | ✅ Fast |
|
||||
| **No Narrator Required** | ❌ | ✅ (default speaker) | ✅ (default speaker) | ✅ (zero-shot / default speaker) | ✅ (basic TTS if no narrator/reference is provided) | ✅ (zero-shot) | ❌ | ✅ (cross-lingual or instruct mode) | ✅ (Base default voice or CustomVoice presets) | N/A | ❌ | ❌ | ✅ Reference audio is optional | ✅ (default speaker) | ✅ Voice reference is optional | ✅ (native default voice; instruct also works without narrator) | ✅ (direct TTS and prompt-only generation) | ❌ | N/A |
|
||||
| Feature | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | TADA | DramaBox | OmniVoice | MOSS-TTS | MOSS-SoundEffect v2 | RVC |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| **TTS** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| **SRT** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| **Voice Conversion** | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| **ASR (Transcribe)** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Sound Effects** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ |
|
||||
| **Training** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
|
||||
| **Voice Cloning** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (Base model) | ❌ | ✅ | ✅ | ✅ Reference audio plus exact transcript | ✅ | ✅ Reference audio plus exact transcript | ✅ Optional reference audio; configurable 3-30 second window from the beginning of the source, default 10 seconds | ✅ | ✅ | ❌ | ⚠️ (needs training) |
|
||||
| **Reference Transcript†** | **Required** | Not used | Not used | Not used | Optional | Optional | Not used | Conditional | Conditional | N/A | **Required** | Not used | **Required** | Optional | **Required** | Not used | **Required** | Conditional | Not used | N/A |
|
||||
| **Native Multi-Speaker** | ❌ | ❌ | ❌ | ✅ (Base only, Kugel uses fallback) | ✅ | ❌ | ❌ | ❌ | ❌ | ⚠️ (Plus variant speaker attribution / diarization) | ❌ | ❌ | ✅ Native dialogue is default; optional custom mode generates each [Character] segment independently as local speaker 0 | ❌ | ❌ | ❌ | ❌ | ✅ (TTSD v1.0; 1-5 speakers) | ❌ | ❌ |
|
||||
| **Emotion Control** | ❌ | ❌ | ⚠️ (v2 tags - doesn't work) | ❌ | ⚠️ (via prompt) | ✅ (native inline tags) | ✅ (8 emotions) | ⚠️ (via instruct) | ⚠️ (via instruct) | ❌ | ✅ (14 emotions) | ❌ | ✅ Free-form inline natural-language tags | ❌ | ❌ | ✅ Natural-language scene prompt and stage directions | ⚠️ (voice-design instruct + inline non-verbal tags) | ✅ (MOSS-VoiceGenerator instruction-conditioned voice design) | ❌ | ❌ |
|
||||
| **Native Long-form** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Configurable 4K-32K native context; suite text chunking is bypassed | ❌ | ❌ | ✅ Official duration-aware quote-group chunking; ~37s target / 45s cap | ✅ (uses upstream audio_chunk_duration / audio_chunk_threshold orchestration; bypasses suite char-based chunk splitting) | ✅ (TTSD/Delay long-form; use chunk orchestration for very long inputs) | ❌ | N/A |
|
||||
| **Community Finetunes** | ✅ | ✅ | ✅ | ✅ KugelAudio, Hindi | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (LoRA adapter inference supported; initial integrated LoRA training support added for MOSS-TTS Delay 8B) | ❌ | ✅ |
|
||||
| **VRAM Efficient** | ✅ | ✅ | ✅ | ⚠️ (5-18GB) | ⚠️ (9GB) | ⚠️ (~8-10GB) | ⚠️ (9-12GB) | ✅ (5.4GB) | ✅ (3-6GB) | ✅ (~4.6GB) | ⚠️ (7GB) | ⚠️ (~7GB total) | ⚠️ 8K context measured at ~15.2GB BF16, ~11.2GB FP8, ~11.2GB BNB INT8, or ~8.9GB BNB NF4; BF16 codec and activations; BNB is a load-time option for the official checkpoint | ⚠️ (main env works; 2B-class model is not lightweight) | ⚠️ 1B is the lighter option; suite VRAM measurements are pending | ✅ Fast mode is ~24GB; experimental FP8 peaks on RTX 4090: staged ~15.1GB allocated, sequential ~11.7GB allocated / ~12.4GB reserved | ⚠️ (main env works; 2B-class model is not lightweight) | ⚠️ (Local 1.7B smaller; tokenizer is large) | ⚠️ (runs in the main ComfyUI environment but remains GPU-heavy) | ✅ |
|
||||
| **Speed/Performance** | ✅ Very Fast | ✅ Fast | ✅ Fast | ⚠️ | ⚠️ | ⚠️ CUDA recommended | ⚠️ | ✅ Fast | ⚠️ | ⚠️ Moderate | ⚠️ | ✅ Fast (diffusion, realtime-capable) | ✅ Main-environment subprocess with reliable teardown; local compile measurements: ~40 it/s BF16, ~11.8 it/s NF4 at ~8.9GB VRAM, and ~3.7 it/s INT8 at ~11.2GB VRAM; quality comparison pending | ⚠️ Moderate; mf variant is faster | ⚠️ No suite benchmark; no streaming or batch inference | ⚠️ Fast favors repeated generation; staged reloads temporary components; sequential also transfers the transformer; optional block-level torch.compile accelerates repeat denoising | ✅ Fast; upstream reports sub-realtime RTF | ✅ Fast with CUDA/FlashAttention | ⚠️ (100 diffusion steps by default) | ✅ Fast |
|
||||
| **No Narrator Required** | ❌ | ✅ (default speaker) | ✅ (default speaker) | ✅ (zero-shot / default speaker) | ✅ (basic TTS if no narrator/reference is provided) | ✅ (zero-shot) | ❌ | ✅ (cross-lingual or instruct mode) | ✅ (Base default voice or CustomVoice presets) | N/A | ❌ | ❌ | ✅ Reference audio is optional | ✅ (default speaker) | ❌ | ✅ Voice reference is optional | ✅ (native default voice; instruct also works without narrator) | ✅ (direct TTS and prompt-only generation) | ❌ | N/A |
|
||||
|
||||
† **Reference Transcript:** Conditional means the transcript is required only for the specific mode: CosyVoice3 zero-shot, Qwen3-TTS full Base cloning, or MOSS-TTSD cloned-speaker dialogue. Higgs Audio 2, Higgs Audio v3, and Dots TTS accept matching text when provided but do not require it.
|
||||
† **Reference Transcript:** Conditional means the transcript is required only for the specific mode: CosyVoice3 zero-shot, Qwen3-TTS full Base cloning, or MOSS-TTSD cloned-speaker dialogue. Higgs Audio 2, Higgs Audio v3, and Dots TTS accept matching text when provided but do not require it. TADA always requires an exact transcript for its reference audio in this integration.
|
||||
@@ -11,8 +11,6 @@ The multiline TTS tag editor now has an engine-aware `Inline Tags` panel with se
|
||||
|
||||
Use the Step mode when you want post-process editing tags. Use the Higgs or Cosy modes when you want those engines' native generation tags.
|
||||
|
||||
The existing **model_path** dropdown now offers `Step-Audio-EditX-2026-01-23` and `Step-Audio-EditX-2025-11-28`. The January checkpoint is the default for new engine nodes and supports the expanded sound vocabulary below. Existing unversioned and local selections keep their files. See [checkpoint selection and compatibility](STEP_AUDIO_EDITX_CHECKPOINTS.md).
|
||||
|
||||
## 💡 Inline vs Manual Node Workflow
|
||||
|
||||
**ComfyUI is modular** - you can manually chain **TTS → 🎨 Step Audio EditX - Audio Editor** nodes for full control.
|
||||
@@ -61,19 +59,6 @@ Insert non-verbal sounds at specific positions in the audio:
|
||||
| `<Question-ei>` | Question sound | `<Question-ei> Really?` |
|
||||
| `<Dissatisfaction-hnn>` | Dissatisfied "hnn" | `<Dissatisfaction-hnn> Not good` |
|
||||
|
||||
The **2026-01-23** checkpoint additionally supports:
|
||||
|
||||
| Tag | Effect |
|
||||
|---|---|
|
||||
| `<inhale>`, `<exhale>`, `<breath>` | Breathing sounds |
|
||||
| `<laugh>`, `<chuckle>`, `<giggle>` | Laughter variants |
|
||||
| `<clears throat>` or `<clears_throat>` | Throat clearing |
|
||||
| `<snort>`, `<cough>` | Snorting and coughing |
|
||||
| `<Surprise-yo>` | Additional surprise vocalization |
|
||||
| `<Question-ah>`, `<Question-en>`, `<Question-yi>`, `<Question-oh>` | Additional questioning vocalizations |
|
||||
|
||||
These support the same iteration and pipe syntax: `<giggle:2>` and `<clears_throat:1|style:serious>`. Tags are case-insensitive. For ChatterBox v2/v3, specify an iteration for Step effects (`<giggle:1>`); bare overlapping tags retain their native meaning.
|
||||
|
||||
**Position matters:** The sound is inserted where you place the tag.
|
||||
|
||||
**💡 Pro Tip - Stronger Effects:** For more reliable and pronounced effects, include relevant text alongside the tag:
|
||||
@@ -93,7 +78,7 @@ Apply emotional tone to the entire audio segment:
|
||||
|
||||
`<emotion:VALUE>` or `<emotion:VALUE:ITERATIONS>`
|
||||
|
||||
**Available emotions:** happy, sad, angry, excited, fear, humour, surprised, disgusted, confusion, empathy, embarrass, depressed, coldness, admiration. Older suite options `calm` and `fearful` remain accepted.
|
||||
**Available emotions:** happy, sad, angry, excited, calm, fearful, surprised, disgusted, confusion, empathy, embarrass, depressed, coldness, admiration
|
||||
|
||||
**Examples:**
|
||||
```
|
||||
@@ -638,13 +623,9 @@ Choose appropriate effects for the situation:
|
||||
### All Paralinguistic Tags
|
||||
`<Breathing>`, `<Laughter>`, `<Sigh>`, `<Uhm>`, `<Surprise-oh>`, `<Surprise-ah>`, `<Surprise-wa>`, `<Confirmation-en>`, `<Question-ei>`, `<Dissatisfaction-hnn>`
|
||||
|
||||
January checkpoint vocabulary: `<inhale>`, `<exhale>`, `<laugh>`, `<chuckle>`, `<clears throat>` (alias `<clears_throat>`), `<snort>`, `<giggle>`, `<cough>`, `<breath>`, `<Surprise-yo>`, `<Question-ah>`, `<Question-en>`, `<Question-yi>`, `<Question-oh>`.
|
||||
|
||||
### All Emotions
|
||||
`<emotion:happy>`, `<emotion:sad>`, `<emotion:angry>`, `<emotion:excited>`, `<emotion:calm>`, `<emotion:fearful>`, `<emotion:surprised>`, `<emotion:disgusted>`, `<emotion:confusion>`, `<emotion:empathy>`, `<emotion:embarrass>`, `<emotion:depressed>`, `<emotion:coldness>`, `<emotion:admiration>`
|
||||
|
||||
Also available: `<emotion:fear>` and `<emotion:humour>`.
|
||||
|
||||
### All Styles
|
||||
`<style:whisper>`, `<style:serious>`, `<style:child>`, `<style:older>`, `<style:girl>`, `<style:pure>`, `<style:sister>`, `<style:sweet>`, `<style:exaggerated>`, `<style:ethereal>`, `<style:generous>`, `<style:recite>`, `<style:act_coy>`, `<style:warm>`, `<style:shy>`, `<style:comfort>`, `<style:authority>`, `<style:chat>`, `<style:radio>`, `<style:soulful>`, `<style:gentle>`, `<style:story>`, `<style:vivid>`, `<style:program>`, `<style:news>`, `<style:advertising>`, `<style:roar>`, `<style:murmur>`, `<style:shout>`, `<style:deeply>`, `<style:loudly>`, `<style:arrogant>`, `<style:friendly>`
|
||||
|
||||
@@ -668,7 +649,7 @@ A: Slightly. The TTS generates normally, then Step Audio EditX loads once and pr
|
||||
A: Yes! Batch processing is automatically applied - all subtitles are generated first, then all edits are applied at once.
|
||||
|
||||
**Q: What if I make a typo in a tag?**
|
||||
A: The Step parser leaves unknown tags in the text and does not create an edit for them. Correct the spelling before generation; the TTS engine may interpret or speak the remaining text.
|
||||
A: Unknown tags are ignored and stripped from the text. Check console for warnings.
|
||||
|
||||
**Q: Can I nest tags?**
|
||||
A: No. Use pipe syntax or separate tags: `<tag1|tag2>` or `<tag1><tag2>`
|
||||
|
||||
+105
-104
@@ -2,110 +2,110 @@
|
||||
|
||||
## Language Support by Engine
|
||||
|
||||
| Language | Code | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS 2 / 2.5 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | DramaBox | OmniVoice | MOSS-TTS | MOSS-SoundEffect v2 | RVC |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 🇺🇸 **English** | EN | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ Tier 1 | ✅ | ✅ Official model is English-only | ✅ | ✅ | ✅ Officially demonstrated prompt language | ✅ |
|
||||
| 🇨🇳 **Chinese** | ZH | ❌ | ❌ | ✅ | ✅ | ✅ (Mandarin) | ✅ | ✅ | ✅ + 18 dialects | ✅ | ❌ | ✅ (Mandarin + Sichuanese, Cantonese) | ❌ | ✅ Tier 1 | ✅ (Mandarin; official also exposes YUE separately outside this matrix) | ❌ | ✅ | ✅ | ✅ Officially demonstrated prompt language | ✅ |
|
||||
| 🇩🇪 **German** | DE | ✅ | ✅ (×3) | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇪🇸 **Spanish** | ES | ✅ | ❌ | ✅ | ✅ (Kugel) | ✅ | ✅ | ✅ IndexTTS-2.5 | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇫🇷 **French** | FR | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇮🇹 **Italian** | IT | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇯🇵 **Japanese** | JA | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ✅ IndexTTS-2.5 | ✅ | ✅ | ✅ (4.1-2b supports Japanese; plus variant does not) | ✅ | ❌ | ✅ Tier 1 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇰🇷 **Korean** | KO | ❌ | ✅ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇷🇺 **Russian** | RU | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇧🇷 **Portuguese** | PT | ✅ (BR) | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ (EU/BR*) | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ (Official PT tag is generic; upstream does not expose separate PT-BR/PT-PT tags and it may lean more European Portuguese than Brazilian Portuguese) | ❌ | ✅ (generic PT; official language space is much broader than this matrix) | ✅ | ❌ | ✅ |
|
||||
| 🇵🇱 **Polish** | PL | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇮🇳 **Hindi** | HI | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇪🇬 **Arabic** | AR | ❌ | ❌ | ✅ (Egyptian) | ✅ (Kugel) | ❌ | ✅ | ✅ IndexTTS-2.5 | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇹🇷 **Turkish** | TR | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇹🇭 **Thai** | TH | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇳🇴 **Norwegian** | NO | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
|
||||
| 🇻🇳 **Vietnamese** | VI | ❌ | ❌ | ✅ (Viterbox) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇦🇲 **Armenian** | HY | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
|
||||
| 🇬🇪 **Georgian** | KA | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
|
||||
| 🇩🇰 **Danish** | DA | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇫🇮 **Finnish** | FI | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇬🇷 **Greek** | EL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇮🇱 **Hebrew** | HE | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇲🇾 **Malay** | MS | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇳🇱 **Dutch** | NL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇸🇪 **Swedish** | SV | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇰🇪 **Swahili** | SW | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇿🇦 **Afrikaans** | AF | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇱 **Albanian** | SQ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Assamese** | AS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Asturian** | AST | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇿 **Azerbaijani** | AZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇺 **Bashkir** | BA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Basque** | EU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇾 **Belarusian** | BE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇩 **Bengali** | BN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇦 **Bosnian** | BS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇬 **Bulgarian** | BG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Catalan** | CA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇭 **Cebuano** | CEB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇶 **Central Kurdish** | CKB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇷 **Croatian** | HR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇿 **Czech** | CS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇺 **Eastern Mari** | MHR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🌍 **Esperanto** | EO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇪 **Estonian** | ET | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Galician** | GL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Gujarati** | GU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇹 **Haitian Creole** | HT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇬 **Hausa** | HA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇺 **Hungarian** | HU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇩 **Indonesian** | ID | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇩 **Javanese** | JV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Kannada** | KN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇿 **Kazakh** | KK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇼 **Kinyarwanda** | RW | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇬 **Kyrgyz** | KY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇻 **Latvian** | LV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇩 **Lingala** | LN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇹 **Lithuanian** | LT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇪 **Luo** | LUO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇰 **Macedonian** | MK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Malayalam** | ML | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇹 **Maltese** | MT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇿 **Māori** | MI | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Marathi** | MR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇳 **Mongolian** | MN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇵 **Nepali** | NE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇫🇷 **Occitan** | OC | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇷 **Persian** | FA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇴 **Romanian** | RO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Sepedi** | NSO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇸 **Serbian** | SR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇼 **Shona** | SN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇰 **Slovak** | SK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇮 **Slovene** | SL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇭 **Tagalog** | TL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇹🇯 **Tajik** | TG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Tamil** | TA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Telugu** | TE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇦 **Ukrainian** | UK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇰 **Urdu** | UR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇳 **Uyghur** | UG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇿 **Uzbek** | UZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Xhosa** | XH | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Zulu** | ZU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇼 **Chichewa/Nyanja** | NY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Eastern Punjabi** | PA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇬 **Ganda** | LG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇸 **Icelandic** | IS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇪 **Irish** | GA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇩🇿 **Kabyle** | KAB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇻 **Kabuverdianu** | KEA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇪 **Kamba** | KAM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇻🇦 **Latin** | LA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇺 **Luxembourgish** | LB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇹 **Oromo** | OM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇫 **Pashto** | PS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇰 **Sindhi** | SD | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇴 **Somali** | SO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇴 **Umbundu** | UMB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🏴 **Welsh** | CY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| Language | Code | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | TADA | DramaBox | OmniVoice | MOSS-TTS | MOSS-SoundEffect v2 | RVC |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 🇺🇸 **English** | EN | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ Tier 1 | ✅ | ✅ (1B and 3B-ML) | ✅ Official model is English-only | ✅ | ✅ | ✅ Officially demonstrated prompt language | ✅ |
|
||||
| 🇨🇳 **Chinese** | ZH | ❌ | ❌ | ✅ | ✅ | ✅ (Mandarin) | ✅ | ✅ | ✅ + 18 dialects | ✅ | ❌ | ✅ (Mandarin + Sichuanese, Cantonese) | ❌ | ✅ Tier 1 | ✅ (Mandarin; official also exposes YUE separately outside this matrix) | ✅ (3B-ML; backend code `ch`) | ❌ | ✅ | ✅ | ✅ Officially demonstrated prompt language | ✅ |
|
||||
| 🇩🇪 **German** | DE | ✅ | ✅ (×3) | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ (3B-ML) | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇪🇸 **Spanish** | ES | ✅ | ❌ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ (3B-ML) | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇫🇷 **French** | FR | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ (3B-ML) | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇮🇹 **Italian** | IT | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (3B-ML) | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇯🇵 **Japanese** | JA | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ✅ ? | ✅ | ✅ | ✅ (4.1-2b supports Japanese; plus variant does not) | ✅ | ❌ | ✅ Tier 1 | ✅ | ✅ (3B-ML) | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇰🇷 **Korean** | KO | ❌ | ✅ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ Tier 2 | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇷🇺 **Russian** | RU | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇧🇷 **Portuguese** | PT | ✅ (BR) | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ (EU/BR*) | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ (Official PT tag is generic; upstream does not expose separate PT-BR/PT-PT tags and it may lean more European Portuguese than Brazilian Portuguese) | ✅ (3B-ML) | ❌ | ✅ (generic PT; official language space is much broader than this matrix) | ✅ | ❌ | ✅ |
|
||||
| 🇵🇱 **Polish** | PL | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (3B-ML) | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇮🇳 **Hindi** | HI | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇪🇬 **Arabic** | AR | ❌ | ❌ | ✅ (Egyptian) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ (3B-ML) | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇹🇷 **Turkish** | TR | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇹🇭 **Thai** | TH | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇳🇴 **Norwegian** | NO | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
|
||||
| 🇻🇳 **Vietnamese** | VI | ❌ | ❌ | ✅ (Viterbox) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇦🇲 **Armenian** | HY | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
|
||||
| 🇬🇪 **Georgian** | KA | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
|
||||
| 🇩🇰 **Danish** | DA | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇫🇮 **Finnish** | FI | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇬🇷 **Greek** | EL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇮🇱 **Hebrew** | HE | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇲🇾 **Malay** | MS | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇳🇱 **Dutch** | NL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇸🇪 **Swedish** | SV | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇰🇪 **Swahili** | SW | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇿🇦 **Afrikaans** | AF | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇱 **Albanian** | SQ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Assamese** | AS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Asturian** | AST | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇿 **Azerbaijani** | AZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇺 **Bashkir** | BA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Basque** | EU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇾 **Belarusian** | BE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇩 **Bengali** | BN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇦 **Bosnian** | BS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇬 **Bulgarian** | BG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Catalan** | CA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇭 **Cebuano** | CEB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇶 **Central Kurdish** | CKB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇷 **Croatian** | HR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇿 **Czech** | CS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇺 **Eastern Mari** | MHR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🌍 **Esperanto** | EO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇪 **Estonian** | ET | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Galician** | GL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Gujarati** | GU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇹 **Haitian Creole** | HT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇬 **Hausa** | HA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇺 **Hungarian** | HU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇩 **Indonesian** | ID | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇩 **Javanese** | JV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Kannada** | KN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇿 **Kazakh** | KK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇼 **Kinyarwanda** | RW | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇬 **Kyrgyz** | KY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇻 **Latvian** | LV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇩 **Lingala** | LN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇹 **Lithuanian** | LT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇪 **Luo** | LUO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇰 **Macedonian** | MK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Malayalam** | ML | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇹 **Maltese** | MT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇿 **Māori** | MI | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Marathi** | MR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇳 **Mongolian** | MN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇵 **Nepali** | NE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇫🇷 **Occitan** | OC | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇷 **Persian** | FA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇴 **Romanian** | RO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Sepedi** | NSO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇸 **Serbian** | SR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇼 **Shona** | SN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇰 **Slovak** | SK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇮 **Slovene** | SL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇭 **Tagalog** | TL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇹🇯 **Tajik** | TG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Tamil** | TA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Telugu** | TE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇦 **Ukrainian** | UK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇰 **Urdu** | UR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇳 **Uyghur** | UG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇿 **Uzbek** | UZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Xhosa** | XH | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Zulu** | ZU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇼 **Chichewa/Nyanja** | NY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Eastern Punjabi** | PA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇬 **Ganda** | LG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇸 **Icelandic** | IS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇪 **Irish** | GA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇩🇿 **Kabyle** | KAB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇻 **Kabuverdianu** | KEA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇪 **Kamba** | KAM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇻🇦 **Latin** | LA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇺 **Luxembourgish** | LB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇹 **Oromo** | OM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇫 **Pashto** | PS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇰 **Sindhi** | SD | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇴 **Somali** | SO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇴 **Umbundu** | UMB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🏴 **Welsh** | CY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
|
||||
**Notes:**
|
||||
|
||||
@@ -114,6 +114,7 @@
|
||||
- **Higgs Audio v3**: Official model card lists 102 languages total: 85 polished-production languages and 17 usable but less polished languages.
|
||||
- **IndexTTS-2**: Trained on 55K+ hours - ZH, EN, JA primary.
|
||||
- **MOSS-TTS**: Official list includes ZH, EN, DE, ES, FR, JA, IT, HU, KO, RU, FA, AR, PL, PT, CS, DA, SV, EL, TR; HU/FA/CS are not separate columns in this matrix.
|
||||
- **TADA**: TADA 1B is English-only. TADA 3B-ML supports EN, AR, ZH, DE, ES, FR, IT, JA, PL, and PT; the upstream backend calls Chinese `ch`.
|
||||
- **OmniVoice**: Official model supports 600+ languages; this matrix only shows the suite's comparison subset.
|
||||
- **RVC**: Language-agnostic voice conversion/post-processing; it is marked supported for every language row.
|
||||
- Qwen3-TTS: Supports pt-BR only with instruction (voice design or harcoded voices). Base can't do instructions, so it will always ouput pt-PT.
|
||||
@@ -65,12 +65,11 @@ Use this as the canonical list of model repositories/links for offline setup.
|
||||
|---|---|---|---|---|
|
||||
| higgs-audio-v3-tts-4b | [bosonai/higgs-audio-v3-tts-4b](https://huggingface.co/bosonai/higgs-audio-v3-tts-4b) | ~8GB | ✅ | Official 4B multilingual controllable TTS model |
|
||||
|
||||
## IndexTTS 2 / 2.5
|
||||
## IndexTTS-2
|
||||
|
||||
| Component | Source | Size | Auto-Download | Notes |
|
||||
|---|---|---|---|---|
|
||||
| IndexTTS-2 | [IndexTeam/IndexTTS-2](https://huggingface.co/IndexTeam/IndexTTS-2) | Multiple files | ✅ | Main TTS engine |
|
||||
| IndexTTS-2.5 | [IndexTeam/IndexTTS-2.5](https://huggingface.co/IndexTeam/IndexTTS-2.5) | ~5.49GB | ✅ | Multilingual backend with bundled codec and official feature-duration scaling |
|
||||
| w2v-bert-2.0 | [facebook/w2v-bert-2.0](https://huggingface.co/facebook/w2v-bert-2.0) | ~2GB | ✅ | Semantic feature extractor |
|
||||
| qwen0.6bemo4-merge | Included with IndexTTS-2 | Included | ✅ | Text emotion model bundle |
|
||||
|
||||
@@ -105,8 +104,7 @@ Use this as the canonical list of model repositories/links for offline setup.
|
||||
|
||||
| Component | Source | Size | Auto-Download | Notes |
|
||||
|---|---|---|---|---|
|
||||
| Step-Audio-EditX-2026-01-23 | [stepfun-ai/Step-Audio-EditX](https://huggingface.co/stepfun-ai/Step-Audio-EditX) | ~7GB | ✅ | Default 3B checkpoint; pinned to 5fe2f8a05c2353301ad47d3c1747b262115da138 |
|
||||
| Step-Audio-EditX-2025-11-28 | [stepfun-ai/Step-Audio-EditX](https://huggingface.co/stepfun-ai/Step-Audio-EditX/tree/7f3de603ae46c96dff6f06f47b1d5a45aabd34fe) | ~7GB | ✅ | Legacy checkpoint; pinned November weights in a separate folder |
|
||||
| Step-Audio-EditX | [stepfun-ai/Step-Audio-EditX](https://huggingface.co/stepfun-ai/Step-Audio-EditX) | ~7GB | ✅ | Main 3B audio editing model |
|
||||
| Step-Audio-Tokenizer | [stepfun-ai/Step-Audio-Tokenizer](https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer) | Included | ✅ | Tokenizer bundle used by Step EditX |
|
||||
|
||||
## Echo-TTS
|
||||
@@ -131,6 +129,16 @@ Use this as the canonical list of model repositories/links for offline setup.
|
||||
| dots.tts-soar | [rednote-hilab/dots.tts-soar](https://huggingface.co/rednote-hilab/dots.tts-soar) | ~6GB | ✅ | Official SOAR checkpoint for higher-quality zero-shot cloning |
|
||||
| dots.tts-mf | [rednote-hilab/dots.tts-mf](https://huggingface.co/rednote-hilab/dots.tts-mf) | ~6GB | ✅ | Official MeanFlow-distilled checkpoint for faster inference |
|
||||
|
||||
## TADA
|
||||
|
||||
| Component | Source | Size | Auto-Download | Notes |
|
||||
|---|---|---|---|---|
|
||||
| TADA 1B | [HumeAI/tada-1b](https://huggingface.co/HumeAI/tada-1b) | ~3.9GB | ✅ | English-only model; downloaded only when selected |
|
||||
| TADA 3B-ML | [HumeAI/tada-3b-ml](https://huggingface.co/HumeAI/tada-3b-ml) | ~8.9GB | ✅ | Multilingual model; downloaded only when selected |
|
||||
| TADA codec | [HumeAI/tada-codec](https://huggingface.co/HumeAI/tada-codec) | ~2.15GB per selected language | ✅ | Downloads shared encoder/decoder plus only the aligner required by the selected language |
|
||||
| Llama tokenizer | [onnx-community/Llama-3.2-1B](https://huggingface.co/onnx-community/Llama-3.2-1B) | Tokenizer files only | ✅ | Ungated redistribution; runtime files are verified against Meta's official Git blob hashes and remain under the Llama 3.2 Community License |
|
||||
| Wav2Vec aligner configuration | [facebook/wav2vec2-large](https://huggingface.co/facebook/wav2vec2-large) | Configuration file only | ✅ | Downloads config.json only; no Wav2Vec model weights |
|
||||
|
||||
## DramaBox
|
||||
|
||||
| Component | Source | Size | Auto-Download | Notes |
|
||||
@@ -151,7 +159,6 @@ Use this as the canonical list of model repositories/links for offline setup.
|
||||
| MOSS-TTS-Local-Transformer | [OpenMOSS-Team/MOSS-TTS-Local-Transformer](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Local-Transformer) | ~6.1GB | ✅ | Official 1.7B local-transformer model |
|
||||
| MOSS-TTS | [OpenMOSS-Team/MOSS-TTS](https://huggingface.co/OpenMOSS-Team/MOSS-TTS) | ~17GB | ✅ | Official 8B delay model |
|
||||
| MOSS-TTS-v1.5 | [OpenMOSS-Team/MOSS-TTS-v1.5](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-v1.5) | ~17GB | ✅ | Current official 8B delay model with 31 languages and more stable voice cloning |
|
||||
| MOSS-TTS v1.5 Voice Acting 8B (Community - LAION) | [laion/moss-tts-v1.5-8b-voice-acting](https://huggingface.co/laion/moss-tts-v1.5-8b-voice-acting) | ~17GB | ✅ | Third-party full MOSS-TTS v1.5 fine-tune for expressive voice acting; not an official OpenMOSS model |
|
||||
| MOSS-VoiceGenerator | [OpenMOSS-Team/MOSS-VoiceGenerator](https://huggingface.co/OpenMOSS-Team/MOSS-VoiceGenerator) | ~4.2GB | ✅ | Official 1.7B reference-free voice-design model |
|
||||
| MOSS-TTSD-v1.0 | [OpenMOSS-Team/MOSS-TTSD-v1.0](https://huggingface.co/OpenMOSS-Team/MOSS-TTSD-v1.0) | ~18GB | ✅ | Official 8B native multi-speaker dialogue model |
|
||||
| MOSS-SoundEffect | [OpenMOSS-Team/MOSS-SoundEffect](https://huggingface.co/OpenMOSS-Team/MOSS-SoundEffect) | ~17GB | ✅ | Official MOSS v1 prompt-only sound-effect checkpoint; uses the shared MOSS audio tokenizer |
|
||||
|
||||
+25
-18
@@ -194,20 +194,14 @@ Notes:
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/step_audio_editx/
|
||||
├── Step-Audio-EditX-2026-01-23/
|
||||
│ └── CosyVoice-300M-25Hz/
|
||||
├── Step-Audio-EditX-2025-11-28/ # downloaded only when selected
|
||||
│ └── CosyVoice-300M-25Hz/
|
||||
├── Step-Audio-EditX/ # existing unversioned installations
|
||||
├── Step-Audio-EditX/
|
||||
│ └── CosyVoice-300M-25Hz/
|
||||
└── FunASR-Paraformer/
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Selected checkpoints and tokenizer assets auto-download at pinned revisions.
|
||||
- Existing unversioned/local weights are retained and folders are not relabeled; known config compatibility fields may be normalized.
|
||||
- See [Step Audio EditX checkpoints](STEP_AUDIO_EDITX_CHECKPOINTS.md) for selection and expanded tags.
|
||||
- Main model, tokenizer assets, and speech stack auto-download.
|
||||
|
||||
## Higgs Audio v3
|
||||
|
||||
@@ -233,7 +227,7 @@ Notes:
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/dramabox/
|
||||
├── DramaBox/
|
||||
└── DramaBox/
|
||||
├── dramabox-dit-v1.safetensors
|
||||
├── dramabox-audio-components.safetensors
|
||||
├── assets/
|
||||
@@ -244,10 +238,6 @@ ComfyUI/models/TTS/dramabox/
|
||||
├── model-00002-of-00002.safetensors
|
||||
├── model.safetensors.index.json
|
||||
└── tokenizer and processor files...
|
||||
└── loras/
|
||||
└── <adapter_name>/
|
||||
├── adapter_config.json
|
||||
└── adapter_model.safetensors
|
||||
```
|
||||
|
||||
Notes:
|
||||
@@ -255,8 +245,6 @@ Notes:
|
||||
- Both repositories download directly into the organized suite folder.
|
||||
- Transformers is forced into local-only loading after download.
|
||||
- Requires NVIDIA CUDA. Fast mode targets approximately 24GB VRAM; experimental staged/sequential modes can run with less.
|
||||
- Integrated training exports LoRA adapters into `dramabox/loras/<adapter_name>/`.
|
||||
- Training jobs, normalized datasets, preprocessing caches, checkpoints, and logs are stored under `ComfyUI/output/tts_audio_suite_training/dramabox/`.
|
||||
- The LTX-2 Community License requires a paid license for entities with at
|
||||
least USD 10 million in annual revenue.
|
||||
|
||||
@@ -301,7 +289,6 @@ Notes:
|
||||
ComfyUI/models/TTS/moss_tts/
|
||||
├── MOSS-TTS-Local-Transformer/
|
||||
├── MOSS-TTS-v1.5/
|
||||
├── moss-tts-v1.5-8b-voice-acting/ # Community - LAION
|
||||
├── MOSS-TTS/
|
||||
├── MOSS-VoiceGenerator/
|
||||
├── MOSS-SoundEffect/
|
||||
@@ -318,8 +305,6 @@ Notes:
|
||||
- `MOSS-Audio-Tokenizer` is required by the official TTS and TTSD variants.
|
||||
- `MOSS-TTS-Local-Transformer` is the smaller 1.7B model.
|
||||
- `MOSS-TTS-v1.5` is the current 8B delay model with 31-language support.
|
||||
- `moss-tts-v1.5-8b-voice-acting` is an optional third-party LAION full fine-tune for expressive speech, not an official OpenMOSS model.
|
||||
- Other compatible full checkpoints placed here are discovered from their `config.json`; unsupported MOSS architectures are rejected explicitly.
|
||||
- `MOSS-TTS` is the legacy official 8B delay model.
|
||||
- `MOSS-VoiceGenerator` is the 1.7B voice-design provider used by Voice Designer.
|
||||
- `MOSS-SoundEffect` is the v1 sound-effect checkpoint used through the MOSS-TTS engine and 🌩️ Sound Effects.
|
||||
@@ -442,6 +427,28 @@ Notes:
|
||||
- Native sample rate is 48kHz.
|
||||
- Main-environment support works on Transformers 5; on Windows, `normalize_text` falls back to no-op if `WeTextProcessing` is unavailable.
|
||||
|
||||
## TADA
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/tada/
|
||||
├── TADA-1B/ # downloaded only if selected
|
||||
├── TADA-3B-ML/ # downloaded only if selected
|
||||
├── tada-codec/
|
||||
│ ├── encoder/
|
||||
│ ├── decoder/
|
||||
│ ├── aligner[-<language>]/ # selected language only
|
||||
│ └── wav2vec2-large/
|
||||
└── llama-3.2-1b-tokenizer/ # tokenizer files only
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- TADA runs in the shared Transformers 4 runtime with `hume-tada==0.1.9`.
|
||||
- The selected 1B or 3B-ML model, shared codec components, and selected language aligner download on demand.
|
||||
- The Llama tokenizer downloads from an ungated redistribution and is verified against Meta's official Git blob hashes. It remains under the Llama 3.2 Community License.
|
||||
- Reference audio and its exact transcript are required by this integration.
|
||||
- Native output is mono 24kHz.
|
||||
|
||||
## OmniVoice
|
||||
|
||||
```text
|
||||
|
||||
@@ -9,13 +9,10 @@ Use this if `🧾 MOSS Dataset Rows` feels unclear.
|
||||
Current first training slice supports:
|
||||
|
||||
- **MOSS-TTS 8B v1.0 and v1.5 (Delay)**
|
||||
- **LAION MOSS-TTS v1.5 Voice Acting 8B community full checkpoint (Delay, compatibility path; training results not yet validated by the suite maintainers)**
|
||||
- **LoRA adapter training**
|
||||
|
||||
The model selected on the connected MOSS engine is used for dataset preparation and training. Prepare the dataset again after switching between v1.0 and v1.5.
|
||||
|
||||
The LAION Voice Acting checkpoint uses the same Delay architecture and can use this LoRA training path, but the suite maintainers have not completed an inference or training run with its full weights. Treat it as community-tested support and report results or incompatibilities.
|
||||
|
||||
It does **not** currently support:
|
||||
|
||||
- Local 1.7B training
|
||||
@@ -26,29 +23,12 @@ It does **not** currently support:
|
||||
|
||||
Current ComfyUI flow:
|
||||
|
||||
1. `🎞️ Training Clip Staging`
|
||||
1. `🎞️ MOSS Clip Staging`
|
||||
2. `🧾 MOSS Dataset Rows`
|
||||
3. `📦 MOSS Dataset Prep`
|
||||
4. `🎛️ MOSS Training Config`
|
||||
5. `🎓 Model Training`
|
||||
|
||||
If clips and transcripts are already prepared on disk, you can skip the first two
|
||||
nodes. Set `dataset_source` on `📦 MOSS Dataset Prep` to a folder containing
|
||||
same-name audio and text pairs:
|
||||
|
||||
```text
|
||||
my_dataset/
|
||||
├── clip001.wav
|
||||
├── clip001.txt
|
||||
├── clip002.flac
|
||||
└── clip002.txt
|
||||
```
|
||||
|
||||
Each `.txt` file must contain the transcript spoken in its matching audio file.
|
||||
Folder scanning supports WAV, FLAC, MP3, OGG, and M4A. Subfolders are ignored
|
||||
unless `recursive_folder_scan` is enabled. Existing JSONL manifest paths continue
|
||||
to work unchanged.
|
||||
|
||||
## The Important Fields
|
||||
|
||||
### `text_lines`
|
||||
@@ -249,7 +229,7 @@ If you do not have a separate validation manifest:
|
||||
|
||||
If you want the least confusing starting point:
|
||||
|
||||
- use `🎞️ Training Clip Staging`
|
||||
- use `🎞️ MOSS Clip Staging`
|
||||
- use `🧾 MOSS Dataset Rows`
|
||||
- fill only `text_lines`
|
||||
- leave `reference_clip_lines` blank
|
||||
|
||||
@@ -63,9 +63,6 @@ Step Audio EditX examples:
|
||||
|
||||
```text
|
||||
<Laughter>
|
||||
<giggle>
|
||||
<clears throat:2>
|
||||
<Question-oh>
|
||||
<emotion:happy>
|
||||
<style:whisper:2>
|
||||
<restore:1@2>
|
||||
@@ -96,12 +93,6 @@ OmniVoice examples:
|
||||
|
||||
Use the dedicated inline tag controls in the sidebar when you do not want to type these by hand.
|
||||
|
||||
The Step sound menu includes the original sounds plus the expanded sounds for the `2026-01-23` checkpoint: `inhale`, `exhale`, `laugh`, `chuckle`, `clears throat`, `snort`, `giggle`, `cough`, `breath`, `Surprise-yo`, `Question-ah`, `Question-en`, `Question-yi`, and `Question-oh`. Select `Step-Audio-EditX-2026-01-23` in the engine node to use these sounds. The `2025-11-28` checkpoint supports the original vocabulary; see the [checkpoint guide](STEP_AUDIO_EDITX_CHECKPOINTS.md).
|
||||
|
||||
Validation, highlighting, and the tag-swap menu recognize these sounds. Names are case-insensitive, and `<clears_throat>` is accepted as an alias for `<clears throat>`. The insertion controls preserve the existing iteration syntax, such as `<giggle:2>`, and pipe combinations, such as `<giggle:2|emotion:happy>`. The emotion menu also includes `fear` and `humour`.
|
||||
|
||||
The editor checks syntax for the chosen tag engine but does not inspect the engine node's selected model checkpoint. A valid new sound still requires the January weights.
|
||||
|
||||
Important differences:
|
||||
|
||||
- `Step Audio EditX` tags are post-process controls
|
||||
|
||||
@@ -93,7 +93,7 @@ Manual tests:
|
||||
|
||||
Runtime plan:
|
||||
- Main Environment only
|
||||
- or Shared Runtime if dependency conflicts are expected
|
||||
- or Shared Runtime / Dedicated Runtime if dependency conflicts are expected
|
||||
```
|
||||
|
||||
If the scope is vague, do not code yet.
|
||||
|
||||
@@ -27,7 +27,7 @@ Downloads and dependencies:
|
||||
- Did you document dependency conflicts or install.py changes?
|
||||
- Did you test `--no-deps` on Main/T5 and simple patching before falling back to the shared T4 runtime?
|
||||
- Did you avoid creating another environment or downloading/reinstalling Torch or Transformers without explicit maintainer approval?
|
||||
- If runtime isolation is needed, did you document the default mode and the reason in YAML/README?
|
||||
- If runtime isolation is recommended, does the engine expose the standard runtime selector, default to the recommended mode, route every choice honestly, and document why?
|
||||
|
||||
Audio format:
|
||||
- Did you verify the model native sample rate?
|
||||
@@ -42,7 +42,7 @@ Generated audio cache:
|
||||
|
||||
Text features:
|
||||
- Did you wire character tags if this node supports text generation?
|
||||
- Did you handle narrator fallback?
|
||||
- Does a missing character use narrator fallback while an existing but invalid voice (for example, missing required transcript) raises a clear error?
|
||||
- Did you wire pause tags?
|
||||
- Did you wire parameter switching for real native parameters?
|
||||
- Did you avoid adding fake unsupported parameters?
|
||||
|
||||
@@ -26,6 +26,7 @@
|
||||
|
||||
### Voice References
|
||||
- **Reference text requirement**: F5-TTS and Step Audio EditX REQUIRE `prompt_text` (transcript), ChatterBox/VibeVoice/Higgs don't
|
||||
- **Missing vs invalid character voice**: A character with no saved voice may fall back to narrator; a discovered/selected voice missing an engine-required transcript is invalid and MUST raise a clear error instead of silently using narrator
|
||||
- **Narrator voice mapping**: Map narrator from TTS Text node input by saving `audio_tensor` to temp file with `reference_text`
|
||||
- **Voice discovery**: Use `get_character_mapping()` not `discover_voices_for_engine()` (latter doesn't exist)
|
||||
|
||||
|
||||
@@ -135,18 +135,13 @@ See the [Sound Effects Guide](SOUND_EFFECTS_GUIDE.md) for pauses, crossfades, lo
|
||||
| `top_p` | `topp` | float | 0.0-1.0 | Nucleus sampling probability |
|
||||
| `inference_steps` | `steps` | int | 1-100 | Number of inference steps |
|
||||
|
||||
#### IndexTTS 2 / 2.5
|
||||
#### IndexTTS-2
|
||||
| Parameter | Alias | Type | Range | Description |
|
||||
|-----------|-------|------|-------|-------------|
|
||||
| `cfg` | — | float | 0.0-20.0 | CFG strength |
|
||||
| `top_p` | `topp` | float | 0.0-1.0 | Nucleus sampling probability |
|
||||
| `top_k` | `topk` | int | 1-100 | Top-k sampling |
|
||||
| `emotion_alpha` | — | float | 0.0-1.0 | Shared audio/vector/text emotion intensity |
|
||||
| `duration_factor` | `dur_factor` | float | 0.5-2.0 | Official IndexTTS-2.5 internal feature-duration scaling; 0.5 shorter/faster, 2.0 longer/slower |
|
||||
|
||||
`duration_factor` is a 2.5-only upstream parameter. It uses nearest-neighbor scaling inside the semantic length regulator after speech codes are generated. It is not natural prosody planning, exact-seconds targeting, waveform playback-speed control, or an inference-performance control. IndexTTS continues to use the suite's ordinary final timing modes in TTS SRT.
|
||||
|
||||
Switching the engine node between IndexTTS-2 and IndexTTS-2.5 invalidates the cached Text/SRT processor and model identity. `language`, `duration_factor`, and `text_normalization` also participate in the generated-audio cache identity, so changing a supported 2.5 generation parameter cannot return audio produced with the previous setting.
|
||||
| `emotion_alpha` | — | float | 0.0-2.0 | Shared audio/vector/text emotion intensity |
|
||||
|
||||
IndexTTS-2 also supports inline emotion controls. Named unsigned values replace
|
||||
that dimension; explicitly signed values adjust the connected vector:
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
# Step Audio EditX checkpoints
|
||||
|
||||
Choose the checkpoint in the existing **model_path** dropdown of **⚙️ Step Audio EditX Engine**. No widget was added or reordered, so saved device, precision, quantization, generation, and runtime values keep their positions.
|
||||
|
||||
| Model choice | Checkpoint | Download revision |
|
||||
|---|---|---|
|
||||
| `Step-Audio-EditX-2026-01-23` | Updated weights and expanded sound tags; default for new engine nodes | `5fe2f8a05c2353301ad47d3c1747b262115da138` |
|
||||
| `Step-Audio-EditX-2025-11-28` | Legacy weights | `7f3de603ae46c96dff6f06f47b1d5a45aabd34fe` |
|
||||
| `Step-Audio-EditX` | Compatibility choice for an existing unversioned installation | Existing files are retained; a missing installation downloads the January checkpoint |
|
||||
| `local:...` | Explicitly selected local model | Existing files are retained |
|
||||
|
||||
Dates identify the weight uploads. The January release was announced on January 29; its pinned revision also includes subsequent configuration fixes. These are versions of the same 3B model architecture, not official “v1/v2” names.
|
||||
|
||||
Sources: [official model](https://huggingface.co/stepfun-ai/Step-Audio-EditX), [November snapshot](https://huggingface.co/stepfun-ai/Step-Audio-EditX/tree/7f3de603ae46c96dff6f06f47b1d5a45aabd34fe), [upstream release notes](https://github.com/stepfun-ai/Step-Audio-EditX#readme).
|
||||
|
||||
## Existing workflows and storage
|
||||
|
||||
- Existing `Step-Audio-EditX` and `local:...` selections remain valid. The suite does not replace their existing weights, rename their folders, or infer their checkpoint. The existing compatibility normalization of `config.json` still applies.
|
||||
- Dated models download lazily into separate folders under `ComfyUI/models/TTS/step_audio_editx/`, respecting `extra_model_paths.yaml`.
|
||||
- Selecting both checkpoints stores both sets of model files. The second LLM checkpoint requires approximately another 7 GB.
|
||||
- Model and edit-result caches distinguish the selected checkpoint.
|
||||
- The shared Transformers 4 runtime remains recommended. Selecting the updated standard checkpoint does not switch to vLLM or to the separate upstream AWQ checkpoint. The suite's `int4` option remains bitsandbytes NF4.
|
||||
|
||||
## Sound tags
|
||||
|
||||
Use the **2026-01-23** checkpoint for expanded sound tags. The November checkpoint is available to preserve older behavior; selecting it does not add the new learned sounds.
|
||||
|
||||
Existing spellings remain supported:
|
||||
|
||||
`<Breathing>`, `<Laughter>`, `<Sigh>`, `<Uhm>`, `<Surprise-oh>`, `<Surprise-ah>`, `<Surprise-wa>`, `<Confirmation-en>`, `<Question-ei>`, `<Dissatisfaction-hnn>`.
|
||||
|
||||
The January vocabulary also exposes:
|
||||
|
||||
`<inhale>`, `<exhale>`, `<laugh>`, `<chuckle>`, `<clears throat>`, `<snort>`, `<giggle>`, `<cough>`, `<breath>`, `<Surprise-yo>`, `<Question-ah>`, `<Question-en>`, `<Question-yi>`, `<Question-oh>`.
|
||||
|
||||
Input is case-insensitive. `<clears_throat>` is an alias for `<clears throat>`; both become `[clears throat]` in the model instruction. Expanded tags support the existing iteration and pipe syntax, for example:
|
||||
|
||||
```text
|
||||
[Alice] That was funny <giggle:2>.
|
||||
[Bob] Let me explain <clears_throat:1|style:serious>.
|
||||
```
|
||||
|
||||
For ChatterBox v2/v3, use an explicit iteration to request Step editing: `<giggle:1>`. Bare `<giggle>`, `<inhale>`, `<exhale>`, and `<cough>` retain their ChatterBox-native meaning. CosyVoice's native sound tags remain native as well.
|
||||
|
||||
## Audio Editor and other engines
|
||||
|
||||
Connect the dated **Step Audio EditX Engine** output to **🎨 Step Audio EditX - Audio Editor** to choose a checkpoint explicitly. In the Audio Editor's transcript, use bare sound tags such as `<giggle>` or `<clears throat>`; its **n_edit_iterations** input controls the number of passes.
|
||||
|
||||
Without a connected engine, the Audio Editor prefers an installed January checkpoint, then an existing unversioned installation. If neither exists, it downloads the dated January checkpoint. This same fallback applies to automatic Step editing of other engines' audio. An existing unversioned installation may contain older weights; use the manual Audio Editor with a dated engine for explicit control.
|
||||
|
||||
See [the inline tag guide](INLINE_EDIT_TAGS_USER_GUIDE.md) for position, iteration, restoration, language, and duration behavior.
|
||||
@@ -0,0 +1,122 @@
|
||||
# TADA Engine Guide
|
||||
|
||||
TADA is Hume AI's zero-shot text-to-speech model. TTS Audio Suite exposes it through the unified **Text to Speech** and **SRT** nodes, including character switching, per-segment language and generation parameters, pause tags, caching, interruption, and Clear VRAM.
|
||||
|
||||
This integration deliberately describes TADA's expressive capability as **reference-conditioned expression transfer**. The model can carry voice, delivery, timing, and emotion present in the prompt audio into generated speech, but it does not provide a native emotion label, intensity slider, embedding, or proven text-instruction emotion control. It is not direct emotion control.
|
||||
|
||||
## Supported models and languages
|
||||
|
||||
| Model | Languages | Model weights | Intended use |
|
||||
|---|---|---:|---|
|
||||
| `HumeAI/tada-1b` | English | ~3.9GB | Lighter English-only option |
|
||||
| `HumeAI/tada-3b-ml` | English, Arabic, Chinese, German, Spanish, French, Italian, Japanese, Polish, Portuguese | ~8.9GB | Multilingual voice cloning |
|
||||
|
||||
The upstream language code for Chinese is `ch`; the suite presents it as Chinese/`zh` and maps it internally. TADA generates mono audio at 24kHz. Reference audio is resampled to 24kHz when needed.
|
||||
|
||||
## First-use requirements
|
||||
|
||||
TADA runs only in the suite's **shared Transformers 4 runtime**. The integration installs `hume-tada==0.1.9` there and does not modify the main ComfyUI Transformers 5 environment. TADA's current configuration code is incompatible with that main environment.
|
||||
|
||||
Before first use, select TADA in an **⚙️ TADA Engine** node and run a workflow. The suite downloads the selected TADA model, shared codec encoder/decoder, the aligner for the selected language, and tokenizer files into `ComfyUI/models/TTS/tada/`.
|
||||
|
||||
Downloads are selective. Choosing the 1B model does not download the 3B model, and changing language downloads only that language's codec aligner. A selected codec set is approximately 2.15GB in addition to the model checkpoint.
|
||||
|
||||
### Automatic download
|
||||
|
||||
No Hugging Face login is required for the normal automatic path. The suite downloads the three tokenizer files from the ungated `onnx-community/Llama-3.2-1B` redistribution, plus its `LICENSE.txt` and `USE_POLICY.md`. Those runtime files are byte-identical to Meta's official `meta-llama/Llama-3.2-1B` tokenizer and are checked against the official Git blob hashes before use.
|
||||
|
||||
The tokenizer remains governed by the Llama 3.2 Community License and Acceptable Use Policy. TADA's use of this component is **Built with Llama**. The full Llama model weights are not downloaded.
|
||||
|
||||
### Manual/offline installation
|
||||
|
||||
Users accustomed to managing ComfyUI models manually can copy the exact tokenizer and model files into the following layout. Only these files are required at runtime:
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/tada/
|
||||
├── TADA-1B/
|
||||
│ ├── config.json
|
||||
│ ├── generation_config.json
|
||||
│ └── model.safetensors
|
||||
├── TADA-3B-ML/
|
||||
│ ├── config.json
|
||||
│ ├── generation_config.json
|
||||
│ ├── model.safetensors.index.json
|
||||
│ ├── model-00001-of-00002.safetensors
|
||||
│ └── model-00002-of-00002.safetensors
|
||||
├── tada-codec/
|
||||
│ ├── encoder/config.json
|
||||
│ ├── encoder/model.safetensors
|
||||
│ ├── decoder/config.json
|
||||
│ ├── decoder/model.safetensors
|
||||
│ ├── aligner/config.json # English
|
||||
│ ├── aligner/model.safetensors
|
||||
│ ├── aligner-<language>/config.json # selected non-English language
|
||||
│ ├── aligner-<language>/model.safetensors
|
||||
│ └── wav2vec2-large/config.json
|
||||
└── llama-3.2-1b-tokenizer/
|
||||
├── tokenizer.json
|
||||
├── tokenizer_config.json
|
||||
└── special_tokens_map.json
|
||||
```
|
||||
|
||||
Install only the selected model directory. For non-English generation, `<language>` is one of `ar`, `ch`, `de`, `es`, `fr`, `it`, `ja`, `pl`, or `pt`. Existing complete files are detected and are not downloaded again. The full Llama model weights and Wav2Vec weights are not required.
|
||||
|
||||
## Reference voice requirements
|
||||
|
||||
Every generation in this integration requires:
|
||||
|
||||
- a clean reference speech clip; and
|
||||
- an exact transcript of the speech in that clip.
|
||||
|
||||
The transcript must match what is spoken, including word order. A mismatched transcript weakens alignment and cloning quality. TADA's official notebook can invoke an English Parakeet ASR model when text is omitted, but the suite does not do that: it would add a hidden ~4.25GB English-only dependency and would be unsuitable for multilingual references. Supply the exact text explicitly or save it beside a character voice as `.reference.txt`/`.txt`.
|
||||
|
||||
Expression comes from the reference recording. Use a prompt that actually contains the desired energy, pace, emotion, and speaking style. There is no emotion dropdown because adding one would be a fake control unsupported by the model API.
|
||||
|
||||
## Engine controls
|
||||
|
||||
Choose the model variant, device, language, and dtype on the engine node. `auto` is the default for device and dtype, and English is the default language. The engine node and unified TTS nodes expose these generation controls:
|
||||
|
||||
| Control | Suite default | Effect |
|
||||
|---|---:|---|
|
||||
| `seed` | unified-node value | Reproducible stochastic generation |
|
||||
| `acoustic_cfg_scale` | `1.6` | Acoustic classifier-free guidance strength |
|
||||
| `duration_cfg_scale` | `1.0` | Duration guidance strength |
|
||||
| `cfg_schedule` | `cosine` | Guidance schedule: `constant`, `linear`, or `cosine` |
|
||||
| `num_flow_matching_steps` | `10` | More ODE steps trade speed for refinement |
|
||||
| `noise_temperature` | `0.9` | Initial flow-matching noise scale |
|
||||
| `time_schedule` | `logsnr` | ODE schedule: `uniform`, `cosine`, or `logsnr` |
|
||||
| `negative_condition_source` | `negative_step_output` | Source used for negative CFG conditioning |
|
||||
| `speed_up_factor` | `0.0` (disabled) | Native two-pass duration scaling; values above 0 add a second generation pass, and values above 1 speak faster |
|
||||
| `num_transition_steps` | `5` | Prompt-to-generated-speech transition overlap |
|
||||
|
||||
Defaults follow the official implementation. Change one parameter at a time: CFG, noise, flow steps, and transition handling interact, and higher values are not automatically better. `speed_up_factor` is not a cheap playback-rate effect; it reruns generation with scaled predicted durations.
|
||||
|
||||
Per-segment tags can override supported parameters and language in unified Text or SRT prompts. The 1B model rejects non-English language selection instead of silently generating with the wrong model.
|
||||
|
||||
## Long text and SRT
|
||||
|
||||
TADA has no native long-form or streaming mode in this integration. The unified Text node uses the suite's sentence-aware chunking and combines the resulting 24kHz clips. The SRT processor generates each subtitle independently and uses the suite's timing modes and overlap fallback.
|
||||
|
||||
The official model was trained on short clips (up to roughly 30 seconds), so long uninterrupted generations can drift. Prefer normal chunking or subtitle-sized segments. Native batch inference, streaming, speech continuation, random-voice generation, text-sampling controls, multiple-candidate scoring, and speaker verification are not exposed in the first integration. Text sampling is irrelevant to normal suite TTS because the exact target text is supplied.
|
||||
|
||||
## Model layout
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/tada/
|
||||
├── TADA-1B/ # downloaded only if selected
|
||||
├── TADA-3B-ML/ # downloaded only if selected
|
||||
├── tada-codec/
|
||||
│ ├── encoder/
|
||||
│ ├── decoder/
|
||||
│ ├── aligner[-<language>]/ # selected language only
|
||||
│ └── wav2vec2-large/ # aligner config only, not model weights
|
||||
└── llama-3.2-1b-tokenizer/ # tokenizer files only
|
||||
```
|
||||
|
||||
## Licensing and upstream sources
|
||||
|
||||
- Official implementation: [HumeAI/tada](https://github.com/HumeAI/tada) — MIT code license.
|
||||
- Model checkpoints: [HumeAI/tada-1b](https://huggingface.co/HumeAI/tada-1b), [HumeAI/tada-3b-ml](https://huggingface.co/HumeAI/tada-3b-ml), and [HumeAI/tada-codec](https://huggingface.co/HumeAI/tada-codec).
|
||||
- Model weights and tokenizer are subject to the Meta Llama 3.2 Community License. Commercial use is conditional on those terms. Applications distributing or presenting the model must follow the license's attribution requirements, including the required **Built with Llama** notice where applicable.
|
||||
|
||||
TTS Audio Suite does not replace or relax upstream license terms. Review them before redistributing models or generated products.
|
||||
@@ -11,33 +11,6 @@ This document tracks updates applied to our bundled IndexTTS-2 code from the ups
|
||||
|
||||
---
|
||||
|
||||
## 2026-08-11: IndexTTS-2.5 Version Integration
|
||||
|
||||
**Official sources:** `index-tts/index-tts` commit `b5ea881bec284b72f0b1cc04e0a724ff0c6b93e9`; model snapshot `ba2480d9f7f629eb18f6acaebb357679d9ba88a4`
|
||||
|
||||
### Changes applied
|
||||
|
||||
- Added IndexTTS-2.5 as a selectable version of the existing `index_tts` engine.
|
||||
- Bundled the official 25 Hz semantic codec, multilingual tokenizer, Japanese G2P, and NeMo normalization bridge.
|
||||
- Preserved suite dual-source audio plus vector/text emotion blending.
|
||||
- Added Chinese, English, Japanese, Spanish, and Arabic conditioning.
|
||||
- Added the official 2.5-only `duration_factor`, documented honestly as nearest-neighbor internal semantic-feature scaling rather than natural prosody or exact-duration planning.
|
||||
- Deliberately excluded IndexTTS-2.5 from TTS SRT's native-duration option; the suite-owned exact-seconds extrapolation was removed after source and listening review.
|
||||
- Kept legacy IndexTTS-2 checkpoints, FP16 loading, MaskGCT, workflows, and node identity intact.
|
||||
- Pinned the audited Hugging Face model revision and retained the main Transformers 5 environment.
|
||||
- Added model-aware Text/SRT processor and audio-cache identities so switching 2.0/2.5 or a 2.5 generation parameter cannot reuse stale output.
|
||||
- Documented the suite's manual finding that 2.5 is not a universal cloning-quality upgrade: 2.0 may retain speaker resemblance better under strong different-speaker emotion transfer.
|
||||
|
||||
### Validation status
|
||||
|
||||
- [x] Python compilation
|
||||
- [x] Bundled backend import under `TTS_SUITE_TEST_VENV_PYTHON`
|
||||
- [x] Full checkpoint download and live ComfyUI generation
|
||||
- [x] Manual audio-quality review of the official duration factor and 2.0/2.5 speaker resemblance
|
||||
- [x] Live 2.5 → 2.0 model switching after processor-cache invalidation fix
|
||||
|
||||
---
|
||||
|
||||
## 2025-09-18: Major Update - Cache & Emotion Improvements
|
||||
|
||||
**Reference commit range:** `8336824..64cb31a` (September 11 → September 18, 2025)
|
||||
|
||||
@@ -56,7 +56,7 @@ python -c "import triton; print(triton.__version__)"
|
||||
|
||||
**Main environment**: no special configuration beyond a working compiler toolchain.
|
||||
|
||||
**Shared Runtime isolation**:
|
||||
**Shared / Dedicated Runtime isolation**:
|
||||
- TTS Audio Suite now tries to detect Visual Studio automatically and inject a proper MSVC build environment into the isolated worker
|
||||
- This is meant to preserve `torch.compile` parity with old direct Transformers 4 environments
|
||||
- It still depends on the Build Tools actually being installed
|
||||
@@ -112,7 +112,7 @@ pip install --upgrade torchvision --index-url https://download.pytorch.org/whl/c
|
||||
**Solution**:
|
||||
1. Install or repair Visual Studio Build Tools with the C++ workload
|
||||
2. Restart ComfyUI so the new environment is visible
|
||||
3. If using Shared Runtime, let TTS Audio Suite auto-detect the toolchain
|
||||
3. If using Shared/Dedicated Runtime, let TTS Audio Suite auto-detect the toolchain
|
||||
|
||||
If detection still fails, your machine state is broken, not just the Python package stack.
|
||||
|
||||
|
||||
@@ -67,43 +67,7 @@ class ChatterBoxEngineAdapter:
|
||||
"""
|
||||
self.node.load_tts_model(device, language)
|
||||
|
||||
def create_segment_cache(self, character="narrator", *, cache_model_name=None,
|
||||
cache_probe=False, **params):
|
||||
"""Share cache identity between generation and pre-load probes."""
|
||||
if not params.get("enable_audio_cache", True):
|
||||
return None
|
||||
from utils.audio.cache import create_cache_function
|
||||
|
||||
language = cache_model_name or params.get("current_language", params.get("model", "English"))
|
||||
device = params.get("device", "auto")
|
||||
model_source = params.get("model_source")
|
||||
if not model_source:
|
||||
# Do not borrow the source of a different currently loaded language.
|
||||
sources = getattr(self, "_cache_model_sources", {})
|
||||
if cache_probe:
|
||||
model_source = sources.get((language, device))
|
||||
if model_source is None:
|
||||
return None
|
||||
else:
|
||||
if hasattr(self.node, "model_manager"):
|
||||
model_source = self.node.model_manager.get_model_source("tts")
|
||||
model_source = model_source or "unknown"
|
||||
sources[(language, device)] = model_source
|
||||
self._cache_model_sources = sources
|
||||
|
||||
audio_component = params.get("stable_audio_component", "main_reference")
|
||||
if character != "narrator":
|
||||
audio_component = f"char_file_{character}"
|
||||
return create_cache_function(
|
||||
engine_type="chatterbox", character=character,
|
||||
exaggeration=params.get("exaggeration", 1.0),
|
||||
temperature=params.get("temperature", 0.8),
|
||||
cfg_weight=params.get("cfg_weight", 1.0),
|
||||
seed=params.get("seed", 0), audio_component=audio_component,
|
||||
model_source=model_source, device=device, language=language,
|
||||
)
|
||||
|
||||
def generate_segment_audio(self, text: str, char_audio: str,
|
||||
def generate_segment_audio(self, text: str, char_audio: str,
|
||||
character: str = "narrator", **params) -> torch.Tensor:
|
||||
"""
|
||||
Generate ChatterBox audio for a text segment.
|
||||
@@ -124,7 +88,34 @@ class ChatterBoxEngineAdapter:
|
||||
seed = params.get("seed", 0)
|
||||
enable_cache = params.get("enable_audio_cache", True)
|
||||
|
||||
cache_fn = self.create_segment_cache(character=character, **params)
|
||||
# Create cache function if caching is enabled
|
||||
cache_fn = None
|
||||
if enable_cache:
|
||||
from utils.audio.cache import create_cache_function
|
||||
|
||||
# Get current language/model for cache key
|
||||
current_language = params.get("current_language", params.get("model", "English"))
|
||||
audio_component = params.get("stable_audio_component", "main_reference")
|
||||
if character != "narrator":
|
||||
audio_component = f"char_file_{character}"
|
||||
|
||||
# Get model source
|
||||
model_source = params.get("model_source")
|
||||
if not model_source and hasattr(self.node, 'model_manager'):
|
||||
model_source = self.node.model_manager.get_model_source("tts")
|
||||
|
||||
cache_fn = create_cache_function(
|
||||
engine_type="chatterbox",
|
||||
character=character,
|
||||
exaggeration=exaggeration,
|
||||
temperature=temperature,
|
||||
cfg_weight=cfg_weight,
|
||||
seed=seed,
|
||||
audio_component=audio_component,
|
||||
model_source=model_source or "unknown",
|
||||
device=params.get("device", "auto"),
|
||||
language=current_language
|
||||
)
|
||||
|
||||
# Handle caching externally for consistency with F5-TTS
|
||||
if cache_fn:
|
||||
@@ -151,7 +142,8 @@ class ChatterBoxEngineAdapter:
|
||||
enable_pause_tags=True,
|
||||
character=character,
|
||||
seed=seed,
|
||||
enable_cache=enable_cache,
|
||||
enable_cache=True, # Enable internal caching for pause tag processing
|
||||
crash_protection_template=params.get("crash_protection_template", "hmm ,, {seg} hmm ,,"),
|
||||
stable_audio_component=params.get("stable_audio_component", "main_reference")
|
||||
)
|
||||
# Don't use external cache for pause tag segments
|
||||
@@ -168,6 +160,7 @@ class ChatterBoxEngineAdapter:
|
||||
character=character,
|
||||
seed=seed,
|
||||
enable_cache=False, # Disable internal caching since we handle it externally
|
||||
crash_protection_template=params.get("crash_protection_template", "hmm ,, {seg} hmm ,,"),
|
||||
stable_audio_component=params.get("stable_audio_component", "main_reference")
|
||||
)
|
||||
|
||||
@@ -407,4 +400,4 @@ class ChatterBoxEngineAdapter:
|
||||
else:
|
||||
num_samples = audio_tensor.numel()
|
||||
|
||||
return num_samples / 44100 # ChatterBox sample rate
|
||||
return num_samples / 44100 # ChatterBox sample rate
|
||||
@@ -110,6 +110,7 @@ class ChatterBoxStreamingAdapter(StreamingEngineAdapter):
|
||||
"seed": kwargs.get("seed", 42),
|
||||
"enable_chunking": False, # Don't chunk in streaming - already handled
|
||||
"enable_audio_cache": kwargs.get("enable_audio_cache", True),
|
||||
"crash_protection_template": kwargs.get("crash_protection_template", "hmm ,, {seg} hmm ,,"),
|
||||
"device": kwargs.get("device", "auto"),
|
||||
"reference_audio": kwargs.get("reference_audio", None)
|
||||
}
|
||||
@@ -299,6 +300,7 @@ class ChatterBoxStreamingAdapter(StreamingEngineAdapter):
|
||||
cfg_weight = kwargs.get('cfg_weight', 0.5)
|
||||
seed = kwargs.get('seed', 42)
|
||||
enable_cache = kwargs.get('enable_audio_cache', True)
|
||||
crash_protection = kwargs.get('crash_protection_template', 'hmm ,, {seg} hmm ,,')
|
||||
|
||||
# Check if node has generation method
|
||||
if hasattr(self.node, '_generate_tts_with_pause_tags'):
|
||||
@@ -319,6 +321,7 @@ class ChatterBoxStreamingAdapter(StreamingEngineAdapter):
|
||||
text, voice_path, exaggeration, temperature, cfg_weight,
|
||||
language, True, character=character, seed=seed,
|
||||
enable_cache=enable_cache,
|
||||
crash_protection_template=crash_protection,
|
||||
stable_audio_component=stable_audio_component
|
||||
)
|
||||
elif hasattr(self.node, 'tts_model') and self.node.tts_model:
|
||||
|
||||
@@ -26,36 +26,11 @@ class DramaBoxEngineAdapter:
|
||||
self.audio_cache = get_audio_cache()
|
||||
self._last_config: Optional[ModelLoadConfig] = None
|
||||
self._load_signature = None
|
||||
self._lora_signature = None
|
||||
self.last_generation_status: Dict[str, Any] = {"near_silent": False}
|
||||
|
||||
def update_config(self, new_config: Dict[str, Any]):
|
||||
self.config = new_config.copy() if new_config else {}
|
||||
|
||||
@staticmethod
|
||||
def _lora_revision(path: Any) -> str:
|
||||
"""Return a cheap cache token that changes when a managed adapter is replaced."""
|
||||
value = str(path or "").strip()
|
||||
if not value:
|
||||
return ""
|
||||
try:
|
||||
candidate = os.path.abspath(os.path.expanduser(value))
|
||||
if os.path.isfile(candidate):
|
||||
stat = os.stat(candidate)
|
||||
return f"{candidate}:{stat.st_size}:{stat.st_mtime_ns}"
|
||||
if os.path.isdir(candidate):
|
||||
entries = []
|
||||
for item in os.listdir(candidate):
|
||||
if not item.endswith(".safetensors"):
|
||||
continue
|
||||
item_path = os.path.join(candidate, item)
|
||||
stat = os.stat(item_path)
|
||||
entries.append(f"{item}:{stat.st_size}:{stat.st_mtime_ns}")
|
||||
return f"{candidate}|{'|'.join(sorted(entries))}"
|
||||
except OSError:
|
||||
pass
|
||||
return value
|
||||
|
||||
@classmethod
|
||||
def _warn_if_near_silent(
|
||||
cls,
|
||||
@@ -101,7 +76,6 @@ class DramaBoxEngineAdapter:
|
||||
}
|
||||
|
||||
def _build_load_signature(self) -> Tuple[Any, ...]:
|
||||
"""Identity of the expensive base runtime, excluding live LoRA state."""
|
||||
return (
|
||||
self.config.get("model_name", "DramaBox"),
|
||||
self.config.get("device", "auto"),
|
||||
@@ -111,50 +85,35 @@ class DramaBoxEngineAdapter:
|
||||
bool(self.config.get("compile_model", False)),
|
||||
)
|
||||
|
||||
def _build_lora_signature(self) -> Tuple[Any, ...]:
|
||||
path = self.config.get("lora_path", "")
|
||||
return (
|
||||
str(path or "").strip(),
|
||||
self._lora_revision(path),
|
||||
float(self.config.get("lora_strength", 1.0)),
|
||||
)
|
||||
|
||||
def _ensure_model_loaded(self):
|
||||
signature = self._build_load_signature()
|
||||
if signature == self._load_signature and self._last_config is not None:
|
||||
return
|
||||
|
||||
self._last_config = ModelLoadConfig(
|
||||
engine_name="dramabox",
|
||||
model_type="tts",
|
||||
model_name=self.config.get("model_name", "DramaBox"),
|
||||
device=self.config.get("device", "auto"),
|
||||
additional_params={
|
||||
"precision": self.config.get("precision", "auto"),
|
||||
"memory_mode": self.config.get("memory_mode", "fast"),
|
||||
"transformer_quantization": self.config.get(
|
||||
"transformer_quantization", "none"
|
||||
),
|
||||
"compile_model": bool(self.config.get("compile_model", False)),
|
||||
},
|
||||
)
|
||||
from utils.models.unified_model_interface import unified_model_interface
|
||||
|
||||
if signature != self._load_signature or self._last_config is None:
|
||||
self._last_config = ModelLoadConfig(
|
||||
engine_name="dramabox",
|
||||
model_type="tts",
|
||||
model_name=self.config.get("model_name", "DramaBox"),
|
||||
device=self.config.get("device", "auto"),
|
||||
additional_params={
|
||||
"precision": self.config.get("precision", "auto"),
|
||||
"memory_mode": self.config.get("memory_mode", "fast"),
|
||||
"transformer_quantization": self.config.get(
|
||||
"transformer_quantization", "none"
|
||||
),
|
||||
"compile_model": bool(self.config.get("compile_model", False)),
|
||||
},
|
||||
)
|
||||
self._load_signature = signature
|
||||
self._lora_signature = None
|
||||
|
||||
engine = unified_model_interface.load_model(self._last_config)
|
||||
lora_signature = self._build_lora_signature()
|
||||
if lora_signature != self._lora_signature:
|
||||
lora_path, lora_revision, lora_strength = lora_signature
|
||||
engine.set_lora(
|
||||
lora_path=lora_path,
|
||||
strength=lora_strength,
|
||||
revision=lora_revision,
|
||||
)
|
||||
self._lora_signature = lora_signature
|
||||
return engine
|
||||
unified_model_interface.load_model(self._last_config)
|
||||
self._load_signature = signature
|
||||
|
||||
def _get_engine(self):
|
||||
return self._ensure_model_loaded()
|
||||
self._ensure_model_loaded()
|
||||
from utils.models.unified_model_interface import unified_model_interface
|
||||
|
||||
return unified_model_interface.load_model(self._last_config)
|
||||
|
||||
def _extract_voice_reference(
|
||||
self, voice_ref: Optional[Dict[str, Any]]
|
||||
@@ -236,9 +195,6 @@ class DramaBoxEngineAdapter:
|
||||
),
|
||||
memory_mode=self.config.get("memory_mode", "fast"),
|
||||
compile_model=bool(self.config.get("compile_model", False)),
|
||||
lora_path=self.config.get("lora_path", ""),
|
||||
lora_strength=float(self.config.get("lora_strength", 1.0)),
|
||||
lora_revision=self._lora_revision(self.config.get("lora_path", "")),
|
||||
seed=int(seed),
|
||||
character=character_name or "narrator",
|
||||
)
|
||||
|
||||
@@ -73,30 +73,7 @@ class F5TTSEngineAdapter:
|
||||
|
||||
self.node.load_f5tts_model(model_name, device)
|
||||
|
||||
def create_segment_cache(self, character="narrator", char_text=None, *,
|
||||
cache_model_name=None, cache_probe=False, **params):
|
||||
"""Share cache identity between generation and pre-load probes."""
|
||||
if not params.get("enable_audio_cache", True):
|
||||
return None
|
||||
from utils.audio.cache import create_cache_function
|
||||
|
||||
audio_component = params.get("stable_audio_component", "main_reference")
|
||||
if character != "narrator":
|
||||
audio_component = f"char_file_{character}"
|
||||
model_name = cache_model_name or getattr(
|
||||
self.node, "current_model_name", params.get("model", "F5TTS_Base")
|
||||
)
|
||||
return create_cache_function(
|
||||
engine_type="f5tts", character=character, model_name=model_name,
|
||||
device=params.get("device", "auto"), audio_component=audio_component,
|
||||
ref_text=char_text, temperature=params.get("temperature", 0.8),
|
||||
speed=params.get("speed", 1.0), target_rms=params.get("target_rms", 0.1),
|
||||
cross_fade_duration=params.get("cross_fade_duration", 0.15),
|
||||
nfe_step=max(1, min(params.get("nfe_step", 32), 71)),
|
||||
cfg_strength=params.get("cfg_strength", 2.0), seed=params.get("seed", 0),
|
||||
)
|
||||
|
||||
def generate_segment_audio(self, text: str, char_audio: str, char_text: str,
|
||||
def generate_segment_audio(self, text: str, char_audio: str, char_text: str,
|
||||
character: str = "narrator", **params) -> torch.Tensor:
|
||||
"""
|
||||
Generate F5-TTS audio for a text segment.
|
||||
@@ -126,7 +103,34 @@ class F5TTSEngineAdapter:
|
||||
if safe_nfe_step != nfe_step:
|
||||
print(f"⚠️ F5-TTS: Clamped nfe_step from {nfe_step} to {safe_nfe_step} to prevent ODE solver issues")
|
||||
|
||||
cache_fn = self.create_segment_cache(character=character, char_text=char_text, **params)
|
||||
# Create cache function if caching is enabled
|
||||
cache_fn = None
|
||||
if enable_cache:
|
||||
from utils.audio.cache import create_cache_function
|
||||
|
||||
# Get audio component for cache key
|
||||
audio_component = params.get("stable_audio_component", "main_reference")
|
||||
if character != "narrator":
|
||||
audio_component = f"char_file_{character}"
|
||||
|
||||
# Get current model name for cache key
|
||||
current_model = getattr(self.node, 'current_model_name', params.get("model", "F5TTS_Base"))
|
||||
|
||||
cache_fn = create_cache_function(
|
||||
engine_type="f5tts",
|
||||
character=character,
|
||||
model_name=current_model,
|
||||
device=params.get("device", "auto"),
|
||||
audio_component=audio_component,
|
||||
ref_text=char_text,
|
||||
temperature=temperature,
|
||||
speed=speed,
|
||||
target_rms=target_rms,
|
||||
cross_fade_duration=cross_fade_duration,
|
||||
nfe_step=safe_nfe_step,
|
||||
cfg_strength=cfg_strength,
|
||||
seed=seed
|
||||
)
|
||||
|
||||
# Generate audio using F5-TTS with pause tag support
|
||||
return self.node.generate_f5tts_with_pause_tags(
|
||||
@@ -185,4 +189,4 @@ class F5TTSEngineAdapter:
|
||||
else:
|
||||
num_samples = audio_tensor.numel()
|
||||
|
||||
return num_samples / 24000 # F5-TTS sample rate
|
||||
return num_samples / 24000 # F5-TTS sample rate
|
||||
@@ -113,12 +113,9 @@ class IndexTTSAdapter:
|
||||
top_k: int = 30,
|
||||
length_penalty: float = 0.0,
|
||||
num_beams: int = 3,
|
||||
repetition_penalty: float = 10.0,
|
||||
max_mel_tokens: int = 1500,
|
||||
language: str = "English",
|
||||
duration_factor: float = 1.0,
|
||||
text_normalization: bool = True,
|
||||
# Streaming parameters
|
||||
repetition_penalty: float = 10.0,
|
||||
max_mel_tokens: int = 1500,
|
||||
# Streaming parameters
|
||||
stream_return: bool = False,
|
||||
more_segment_before: int = 0,
|
||||
**kwargs) -> torch.Tensor:
|
||||
@@ -142,10 +139,7 @@ class IndexTTSAdapter:
|
||||
length_penalty: Length penalty for beam search
|
||||
num_beams: Number of beams for beam search
|
||||
repetition_penalty: Repetition penalty
|
||||
max_mel_tokens: Maximum mel tokens to generate
|
||||
language: IndexTTS-2.5 language code/name
|
||||
duration_factor: Official 2.5 internal feature-duration multiplier
|
||||
text_normalization: Enable multilingual text normalization
|
||||
max_mel_tokens: Maximum mel tokens to generate
|
||||
**kwargs: Additional parameters
|
||||
|
||||
Returns:
|
||||
@@ -161,31 +155,9 @@ class IndexTTSAdapter:
|
||||
# Parse character switching tags with emotion support
|
||||
processed_segments = self._process_character_tags_with_emotions(text)
|
||||
|
||||
if len(processed_segments) > 1:
|
||||
# Multi-segment character switching - process each segment separately
|
||||
return self._generate_multi_character_segments(
|
||||
processed_segments, speaker_audio, emotion_audio,
|
||||
emotion_alpha=emotion_alpha,
|
||||
emotion_vector=emotion_vector,
|
||||
use_emotion_text=use_emotion_text,
|
||||
emotion_text=emotion_text,
|
||||
use_random=use_random,
|
||||
interval_silence=interval_silence,
|
||||
max_text_tokens_per_segment=max_text_tokens_per_segment,
|
||||
temperature=temperature,
|
||||
top_p=top_p,
|
||||
top_k=top_k,
|
||||
length_penalty=length_penalty,
|
||||
num_beams=num_beams,
|
||||
repetition_penalty=repetition_penalty,
|
||||
max_mel_tokens=max_mel_tokens,
|
||||
language=language,
|
||||
duration_factor=duration_factor,
|
||||
text_normalization=text_normalization,
|
||||
stream_return=stream_return,
|
||||
more_segment_before=more_segment_before,
|
||||
**kwargs,
|
||||
)
|
||||
if len(processed_segments) > 1:
|
||||
# Multi-segment character switching - process each segment separately
|
||||
return self._generate_multi_character_segments(processed_segments, speaker_audio, emotion_audio, **kwargs)
|
||||
elif processed_segments:
|
||||
# Single character segment
|
||||
first_segment = processed_segments[0]
|
||||
@@ -260,12 +232,9 @@ class IndexTTSAdapter:
|
||||
length_penalty=length_penalty,
|
||||
repetition_penalty=repetition_penalty,
|
||||
max_mel_tokens=max_mel_tokens,
|
||||
max_text_tokens_per_segment=max_text_tokens_per_segment,
|
||||
interval_silence=interval_silence,
|
||||
language=language,
|
||||
duration_factor=duration_factor,
|
||||
text_normalization=text_normalization,
|
||||
stream_return=stream_return,
|
||||
max_text_tokens_per_segment=max_text_tokens_per_segment,
|
||||
interval_silence=interval_silence,
|
||||
stream_return=stream_return,
|
||||
more_segment_before=more_segment_before,
|
||||
**kwargs # Include seed and other kwargs in cache key
|
||||
)
|
||||
@@ -329,12 +298,9 @@ class IndexTTSAdapter:
|
||||
top_k=top_k,
|
||||
length_penalty=length_penalty,
|
||||
num_beams=num_beams,
|
||||
repetition_penalty=repetition_penalty,
|
||||
max_mel_tokens=max_mel_tokens,
|
||||
language=language,
|
||||
duration_factor=duration_factor,
|
||||
text_normalization=text_normalization,
|
||||
**engine_kwargs
|
||||
repetition_penalty=repetition_penalty,
|
||||
max_mel_tokens=max_mel_tokens,
|
||||
**engine_kwargs
|
||||
)
|
||||
except torch.OutOfMemoryError as e:
|
||||
# Analyze audio after OOM to provide helpful feedback
|
||||
@@ -414,7 +380,7 @@ class IndexTTSAdapter:
|
||||
Returns:
|
||||
Combined audio tensor [1, samples] at 22050 Hz
|
||||
"""
|
||||
audio_segments = []
|
||||
audio_segments = []
|
||||
|
||||
# Get character mapping for all unique characters
|
||||
unique_characters = set()
|
||||
@@ -441,10 +407,7 @@ class IndexTTSAdapter:
|
||||
for segment in segments:
|
||||
character_name = segment.get('character', 'narrator')
|
||||
segment_text = segment.get('text', '').strip()
|
||||
emotion_ref = segment.get('emotion')
|
||||
segment_kwargs = dict(kwargs)
|
||||
if segment.get('language'):
|
||||
segment_kwargs['language'] = segment['language']
|
||||
emotion_ref = segment.get('emotion')
|
||||
|
||||
if not segment_text:
|
||||
continue
|
||||
@@ -470,9 +433,9 @@ class IndexTTSAdapter:
|
||||
# Generate cache key for this segment
|
||||
segment_cache_key = self._generate_cache_key(
|
||||
text=segment_text,
|
||||
speaker_audio=speaker_audio,
|
||||
emotion_audio=emotion_audio,
|
||||
**segment_kwargs
|
||||
speaker_audio=speaker_audio,
|
||||
emotion_audio=emotion_audio,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
# Check cache first
|
||||
@@ -485,9 +448,9 @@ class IndexTTSAdapter:
|
||||
try:
|
||||
segment_audio = self.engine.generate(
|
||||
text=segment_text,
|
||||
speaker_audio=speaker_audio,
|
||||
emotion_audio=emotion_audio,
|
||||
**segment_kwargs
|
||||
speaker_audio=speaker_audio,
|
||||
emotion_audio=emotion_audio,
|
||||
**kwargs
|
||||
)
|
||||
except torch.OutOfMemoryError as e:
|
||||
# Analyze audio after OOM in multi-character segments
|
||||
@@ -511,18 +474,9 @@ class IndexTTSAdapter:
|
||||
# Return silence if no segments generated
|
||||
return torch.zeros(1, 22050, dtype=torch.float32)
|
||||
|
||||
def _generate_cache_key(self, **params) -> str:
|
||||
"""Generate cache key for IndexTTS-2."""
|
||||
model_identity = {}
|
||||
if self.engine is not None:
|
||||
model_identity = {
|
||||
"model_name": getattr(self.engine, "model_name", None),
|
||||
"model_version": getattr(self.engine, "model_version", None),
|
||||
"model_path": getattr(self.engine, "model_dir", None),
|
||||
}
|
||||
return self.audio_cache.generate_cache_key(
|
||||
'index_tts', **model_identity, **params
|
||||
)
|
||||
def _generate_cache_key(self, **params) -> str:
|
||||
"""Generate cache key for IndexTTS-2."""
|
||||
return self.audio_cache.generate_cache_key('index_tts', **params)
|
||||
|
||||
def _analyze_audio_after_oom(self, speaker_audio: str, emotion_audio: str, max_mel_tokens: int) -> str:
|
||||
"""
|
||||
@@ -639,6 +593,10 @@ class IndexTTSAdapter:
|
||||
|
||||
def unload(self):
|
||||
"""Unload the engine to free memory."""
|
||||
if self.engine:
|
||||
self.engine.unload()
|
||||
self.engine = None
|
||||
if self.engine:
|
||||
self.engine.unload()
|
||||
self.engine = None
|
||||
|
||||
def __del__(self):
|
||||
"""Cleanup on deletion."""
|
||||
self.unload()
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
"""Adapter between TTS Audio Suite processors and the isolated TADA runtime."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
from typing import Any, Dict, Optional, Tuple
|
||||
|
||||
import torch
|
||||
|
||||
current_dir = os.path.dirname(__file__)
|
||||
engines_dir = os.path.dirname(current_dir)
|
||||
project_root = os.path.dirname(engines_dir)
|
||||
if project_root not in sys.path:
|
||||
sys.path.insert(0, project_root)
|
||||
|
||||
from engines.tada.languages import format_tada_language_display, validate_tada_model_language
|
||||
from engines.tada.tada_downloader import TadaDownloader
|
||||
from utils.audio.audio_hash import generate_stable_audio_component
|
||||
from utils.audio.cache import get_audio_cache
|
||||
from utils.device import resolve_torch_device
|
||||
from utils.models.factory_config import ModelLoadConfig, RUNTIME_MODE_SHARED
|
||||
from utils.voice.reference import effective_voice_audio
|
||||
|
||||
|
||||
class TadaEngineAdapter:
|
||||
"""Resolve assets, load the shared runtime proxy, and cache TADA waveforms."""
|
||||
|
||||
SAMPLE_RATE = 24000
|
||||
|
||||
def __init__(self, config: Optional[Dict[str, Any]] = None):
|
||||
self.config = config.copy() if config else {}
|
||||
self.audio_cache = get_audio_cache()
|
||||
self.downloader = TadaDownloader()
|
||||
self._last_config: Optional[ModelLoadConfig] = None
|
||||
self._load_signature: Optional[Tuple[Any, ...]] = None
|
||||
self._ensured_assets: Dict[Tuple[str, str], Dict[str, str]] = {}
|
||||
|
||||
def update_config(self, new_config: Dict[str, Any]) -> None:
|
||||
self.config = new_config.copy() if new_config else {}
|
||||
|
||||
def _model_identifier(self) -> str:
|
||||
return str(
|
||||
self.config.get("model_variant")
|
||||
or self.config.get("model_path")
|
||||
or self.config.get("model_name")
|
||||
or "TADA-1B"
|
||||
)
|
||||
|
||||
def _ensure_assets(self, language: str) -> Dict[str, str]:
|
||||
identifier = self._model_identifier()
|
||||
display_language = format_tada_language_display(language)
|
||||
key = (identifier, display_language)
|
||||
assets = self._ensured_assets.get(key)
|
||||
if assets is None:
|
||||
assets = self.downloader.resolve_model_assets(identifier, display_language)
|
||||
self._ensured_assets[key] = assets
|
||||
return assets
|
||||
|
||||
def _build_load_signature(self, assets: Dict[str, str]) -> Tuple[Any, ...]:
|
||||
resolved_device = resolve_torch_device(self.config.get("device", "auto"))
|
||||
return (
|
||||
assets["model_name"],
|
||||
os.path.abspath(assets["model_path"]),
|
||||
os.path.abspath(assets["codec_path"]),
|
||||
os.path.abspath(assets["tokenizer_path"]),
|
||||
resolved_device,
|
||||
self.config.get("dtype", "auto"),
|
||||
self.config.get("attn_implementation", "sdpa"),
|
||||
bool(self.config.get("use_torch_compile", False)),
|
||||
int(self.config.get("prompt_cache_size", 8)),
|
||||
self.config.get("runtime_mode", RUNTIME_MODE_SHARED),
|
||||
self.config.get("runtime_profile", "vibevoice_transformers4_shared"),
|
||||
)
|
||||
|
||||
def _ensure_model_loaded(self, language: str) -> None:
|
||||
from utils.models.unified_model_interface import unified_model_interface
|
||||
|
||||
assets = self._ensure_assets(language)
|
||||
signature = self._build_load_signature(assets)
|
||||
if signature == self._load_signature and self._last_config is not None:
|
||||
return
|
||||
|
||||
resolved_device = resolve_torch_device(self.config.get("device", "auto"))
|
||||
load_config = ModelLoadConfig(
|
||||
engine_name="tada",
|
||||
model_type="tts",
|
||||
model_name=assets["model_name"],
|
||||
model_path=assets["model_path"],
|
||||
device=resolved_device,
|
||||
additional_params={
|
||||
"codec_path": assets["codec_path"],
|
||||
"tokenizer_path": assets["tokenizer_path"],
|
||||
"dtype": self.config.get("dtype", "auto"),
|
||||
"attn_implementation": self.config.get("attn_implementation", "sdpa"),
|
||||
"use_torch_compile": bool(self.config.get("use_torch_compile", False)),
|
||||
"prompt_cache_size": int(self.config.get("prompt_cache_size", 8)),
|
||||
},
|
||||
runtime_mode=self.config.get("runtime_mode", RUNTIME_MODE_SHARED),
|
||||
runtime_profile=self.config.get(
|
||||
"runtime_profile", "vibevoice_transformers4_shared"
|
||||
),
|
||||
)
|
||||
self._last_config = load_config
|
||||
unified_model_interface.load_model(load_config)
|
||||
self._load_signature = signature
|
||||
|
||||
def _get_engine(self):
|
||||
if self._last_config is None:
|
||||
raise RuntimeError("TADA model was requested before its assets were resolved")
|
||||
from utils.models.unified_model_interface import unified_model_interface
|
||||
|
||||
return unified_model_interface.load_model(self._last_config)
|
||||
|
||||
@staticmethod
|
||||
def _reference_text(voice_ref: Optional[Dict[str, Any]]) -> str:
|
||||
if not isinstance(voice_ref, dict):
|
||||
return ""
|
||||
return str(
|
||||
voice_ref.get("reference_text")
|
||||
or voice_ref.get("prompt_text")
|
||||
or voice_ref.get("text")
|
||||
or ""
|
||||
).strip()
|
||||
|
||||
@staticmethod
|
||||
def _audio_component(reference_audio: Any, voice_ref: Dict[str, Any]) -> str:
|
||||
if isinstance(reference_audio, str):
|
||||
return generate_stable_audio_component(audio_file_path=reference_audio)
|
||||
if isinstance(reference_audio, dict) and "waveform" in reference_audio:
|
||||
return generate_stable_audio_component(reference_audio=reference_audio)
|
||||
if torch.is_tensor(reference_audio):
|
||||
return generate_stable_audio_component(
|
||||
reference_audio={
|
||||
"waveform": reference_audio,
|
||||
"sample_rate": int(voice_ref.get("sample_rate", TadaEngineAdapter.SAMPLE_RATE)),
|
||||
}
|
||||
)
|
||||
if isinstance(reference_audio, (tuple, list)) and len(reference_audio) == 2:
|
||||
waveform, sample_rate = reference_audio
|
||||
if torch.is_tensor(waveform):
|
||||
return generate_stable_audio_component(
|
||||
reference_audio={"waveform": waveform, "sample_rate": int(sample_rate)}
|
||||
)
|
||||
raise TypeError(f"Unsupported TADA reference audio type: {type(reference_audio)}")
|
||||
|
||||
def generate_single(
|
||||
self,
|
||||
text: str,
|
||||
voice_ref: Optional[Dict[str, Any]],
|
||||
seed: int = 0,
|
||||
enable_audio_cache: bool = True,
|
||||
character_name: Optional[str] = None,
|
||||
) -> torch.Tensor:
|
||||
stripped = str(text or "").strip()
|
||||
if not stripped:
|
||||
return torch.zeros(1, 0, dtype=torch.float32)
|
||||
if not isinstance(voice_ref, dict):
|
||||
raise ValueError("TADA requires reference audio and its exact transcript")
|
||||
|
||||
reference_audio = effective_voice_audio(voice_ref)
|
||||
reference_text = self._reference_text(voice_ref)
|
||||
if reference_audio is None:
|
||||
raise ValueError("TADA requires reference audio; random-voice mode is not exposed")
|
||||
if not reference_text:
|
||||
raise ValueError(
|
||||
"TADA requires the exact reference transcript. Automatic Parakeet ASR is "
|
||||
"intentionally disabled to prevent a hidden 4.25 GB download."
|
||||
)
|
||||
|
||||
model_variant = self._model_identifier()
|
||||
language = format_tada_language_display(self.config.get("language", "English"))
|
||||
validate_tada_model_language(model_variant, language)
|
||||
speed_up_factor = float(self.config.get("speed_up_factor", 0.0) or 0.0)
|
||||
native_speed = speed_up_factor if speed_up_factor > 0 else None
|
||||
audio_component = self._audio_component(reference_audio, voice_ref)
|
||||
|
||||
generation_params = {
|
||||
"model_variant": model_variant,
|
||||
"decoder_source": "official_tada_codec_v1",
|
||||
"language": language,
|
||||
"reference_text": reference_text,
|
||||
"audio_component": audio_component,
|
||||
"seed": int(seed or 0),
|
||||
"acoustic_cfg_scale": float(self.config.get("acoustic_cfg_scale", 1.6)),
|
||||
"duration_cfg_scale": float(self.config.get("duration_cfg_scale", 1.0)),
|
||||
"noise_temperature": float(self.config.get("noise_temperature", 0.9)),
|
||||
"num_flow_matching_steps": int(self.config.get("num_flow_matching_steps", 10)),
|
||||
"cfg_schedule": self.config.get("cfg_schedule", "cosine"),
|
||||
"time_schedule": self.config.get("time_schedule", "logsnr"),
|
||||
"negative_condition_source": self.config.get(
|
||||
"negative_condition_source", "negative_step_output"
|
||||
),
|
||||
"speed_up_factor": native_speed,
|
||||
"num_transition_steps": int(self.config.get("num_transition_steps", 5)),
|
||||
"device": self.config.get("device", "auto"),
|
||||
"dtype": self.config.get("dtype", "auto"),
|
||||
"runtime_mode": self.config.get("runtime_mode", RUNTIME_MODE_SHARED),
|
||||
"runtime_profile": self.config.get(
|
||||
"runtime_profile", "vibevoice_transformers4_shared"
|
||||
),
|
||||
"character": character_name or "narrator",
|
||||
}
|
||||
|
||||
cache_key = None
|
||||
if enable_audio_cache:
|
||||
cache_key = self.audio_cache.generate_cache_key(
|
||||
"tada", text=stripped, **generation_params
|
||||
)
|
||||
cached = self.audio_cache.get_cached_audio(cache_key)
|
||||
if cached:
|
||||
print(
|
||||
f"💾 Using cached TADA audio for '{character_name or 'narrator'}': "
|
||||
f"'{stripped[:30]}...'"
|
||||
)
|
||||
return cached[0]
|
||||
|
||||
self._ensure_model_loaded(language)
|
||||
engine = self._get_engine()
|
||||
audio, sample_rate = engine.generate_speech(
|
||||
text=stripped,
|
||||
reference_audio=reference_audio,
|
||||
reference_text=reference_text,
|
||||
seed=generation_params["seed"],
|
||||
language=language,
|
||||
acoustic_cfg_scale=generation_params["acoustic_cfg_scale"],
|
||||
duration_cfg_scale=generation_params["duration_cfg_scale"],
|
||||
cfg_schedule=generation_params["cfg_schedule"],
|
||||
time_schedule=generation_params["time_schedule"],
|
||||
num_flow_matching_steps=generation_params["num_flow_matching_steps"],
|
||||
noise_temperature=generation_params["noise_temperature"],
|
||||
speed_up_factor=native_speed,
|
||||
num_transition_steps=generation_params["num_transition_steps"],
|
||||
negative_condition_source=generation_params["negative_condition_source"],
|
||||
)
|
||||
if int(sample_rate) != self.SAMPLE_RATE:
|
||||
raise RuntimeError(
|
||||
f"TADA returned {sample_rate} Hz audio; expected {self.SAMPLE_RATE} Hz"
|
||||
)
|
||||
if not isinstance(audio, torch.Tensor):
|
||||
audio = torch.as_tensor(audio, dtype=torch.float32)
|
||||
audio = audio.detach().cpu().float()
|
||||
if audio.dim() == 1:
|
||||
audio = audio.unsqueeze(0)
|
||||
elif audio.dim() != 2:
|
||||
raise RuntimeError(f"TADA returned an invalid audio shape: {tuple(audio.shape)}")
|
||||
if audio.shape[-1] == 0:
|
||||
raise RuntimeError("TADA returned empty audio")
|
||||
|
||||
if enable_audio_cache and cache_key:
|
||||
duration = self.audio_cache._calculate_duration(audio, "tada")
|
||||
self.audio_cache.cache_audio(cache_key, audio, duration)
|
||||
return audio
|
||||
|
||||
|
||||
__all__ = ["TadaEngineAdapter"]
|
||||
@@ -74,10 +74,7 @@ class MaskedDiffWithXvec(torch.nn.Module):
|
||||
|
||||
# concat text and prompt_text
|
||||
mask = (~make_pad_mask(token_len)).float().unsqueeze(-1).to(device)
|
||||
# TTS Audio Suite patch: bound speech-token indices to prevent CUDA embedding asserts.
|
||||
token = self.input_embedding(
|
||||
torch.clamp(token, min=0, max=self.input_embedding.num_embeddings - 1)
|
||||
) * mask
|
||||
token = self.input_embedding(torch.clamp(token, min=0)) * mask
|
||||
|
||||
# text encode
|
||||
h, h_lengths = self.encoder(token, token_len)
|
||||
@@ -127,10 +124,7 @@ class MaskedDiffWithXvec(torch.nn.Module):
|
||||
token_len1, token_len2 = prompt_token.shape[1], token.shape[1]
|
||||
token, token_len = torch.concat([prompt_token, token], dim=1), prompt_token_len + token_len
|
||||
mask = (~make_pad_mask(token_len)).unsqueeze(-1).to(embedding)
|
||||
# TTS Audio Suite patch: bound speech-token indices to prevent CUDA embedding asserts.
|
||||
token = self.input_embedding(
|
||||
torch.clamp(token, min=0, max=self.input_embedding.num_embeddings - 1)
|
||||
) * mask
|
||||
token = self.input_embedding(torch.clamp(token, min=0)) * mask
|
||||
|
||||
# text encode
|
||||
h, h_lengths = self.encoder(token, token_len)
|
||||
@@ -221,10 +215,7 @@ class CausalMaskedDiffWithXvec(torch.nn.Module):
|
||||
# concat text and prompt_text
|
||||
token, token_len = torch.concat([prompt_token, token], dim=1), prompt_token_len + token_len
|
||||
mask = (~make_pad_mask(token_len)).unsqueeze(-1).to(embedding)
|
||||
# TTS Audio Suite patch: bound speech-token indices to prevent CUDA embedding asserts.
|
||||
token = self.input_embedding(
|
||||
torch.clamp(token, min=0, max=self.input_embedding.num_embeddings - 1)
|
||||
) * mask
|
||||
token = self.input_embedding(torch.clamp(token, min=0)) * mask
|
||||
|
||||
# text encode
|
||||
h, h_lengths = self.encoder(token, token_len)
|
||||
|
||||
@@ -1,5 +1,3 @@
|
||||
import logging
|
||||
|
||||
from .s3tokenizer import (
|
||||
S3_SR,
|
||||
S3_HOP,
|
||||
@@ -29,20 +27,4 @@ def drop_invalid_tokens(x):
|
||||
e = None
|
||||
|
||||
x = x[s: e]
|
||||
|
||||
# T3's speech head spans `speech_tokens_dict_size` (8194) while S3Gen's
|
||||
# input embedding only has SPEECH_VOCAB_SIZE (6561) rows, so a sampled id
|
||||
# above the vocab that is not exactly SoS/EoS reaches the embedding gather
|
||||
# and trips a device-side "index out of bounds" assert (which surfaces
|
||||
# later, at the next CUDA sync, with a misleading traceback).
|
||||
invalid = x >= SPEECH_VOCAB_SIZE
|
||||
if invalid.any():
|
||||
logging.warning(
|
||||
"Dropping %d out-of-vocabulary speech token(s) (max id %d, vocab %d)",
|
||||
int(invalid.sum()), int(x.max()), SPEECH_VOCAB_SIZE,
|
||||
)
|
||||
x = x[~invalid]
|
||||
# TTS Audio Suite patch: do not pass an empty speech stream to S3Gen.
|
||||
if x.numel() == 0:
|
||||
raise RuntimeError("ChatterBox generated no valid speech tokens. Try another seed or text.")
|
||||
return x
|
||||
|
||||
@@ -16,6 +16,7 @@ from .modules.cond_enc import T3CondEnc, T3Cond
|
||||
from .modules.t3_config import T3Config
|
||||
from .llama_configs import LLAMA_CONFIGS
|
||||
from .inference.t3_hf_backend import T3HuggingfaceBackend
|
||||
from .inference.alignment_stream_analyzer import AlignmentStreamAnalyzer
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -276,9 +277,13 @@ class T3(nn.Module):
|
||||
self.tfmr.use_cache = True
|
||||
self.tfmr.return_dict = True
|
||||
|
||||
# TTS Audio Suite patch: alignment analysis is disabled in the backend, so do
|
||||
# not install its persistent attention hook/forward wrapper on every inference.
|
||||
alignment_stream_analyzer = None
|
||||
alignment_stream_analyzer = AlignmentStreamAnalyzer(
|
||||
self.tfmr,
|
||||
None,
|
||||
text_tokens_slice=(len_cond, len_cond + text_tokens.size(-1)),
|
||||
alignment_layer_idx=9,
|
||||
eos_idx=self.hp.stop_speech_token,
|
||||
)
|
||||
|
||||
# Create backend with updated settings
|
||||
patched_model = T3HuggingfaceBackend(
|
||||
|
||||
@@ -268,7 +268,7 @@ class ChatterboxTTS:
|
||||
if custom_path.suffix == ".safetensors":
|
||||
return load_file(custom_path, device=device)
|
||||
if custom_path.suffix == ".pt":
|
||||
return torch.load(custom_path, map_location=device, weights_only=True)
|
||||
return torch.load(custom_path, map_location=device)
|
||||
|
||||
safetensors_path = ckpt_dir / f"{base_name}.safetensors"
|
||||
pt_path = ckpt_dir / f"{base_name}.pt"
|
||||
@@ -276,7 +276,7 @@ class ChatterboxTTS:
|
||||
if safetensors_path.exists():
|
||||
return load_file(safetensors_path, device=device)
|
||||
elif pt_path.exists():
|
||||
return torch.load(pt_path, map_location=device, weights_only=True)
|
||||
return torch.load(pt_path, map_location=device)
|
||||
elif not required:
|
||||
return None
|
||||
else:
|
||||
@@ -299,7 +299,7 @@ class ChatterboxTTS:
|
||||
return load_file(safetensors_path, device=device)
|
||||
elif pt_path.exists():
|
||||
print(f"📁 Loading {base_name} from local English model: {pt_path}")
|
||||
return torch.load(pt_path, map_location=device, weights_only=True)
|
||||
return torch.load(pt_path, map_location=device)
|
||||
|
||||
# Download English model if not available locally
|
||||
print(f"📦 Downloading English model components for incomplete language model...")
|
||||
@@ -315,7 +315,7 @@ class ChatterboxTTS:
|
||||
return load_file(safetensors_path, device=device)
|
||||
elif pt_path.exists():
|
||||
print(f"📁 Loading {base_name} from downloaded English model: {pt_path}")
|
||||
return torch.load(pt_path, map_location=device, weights_only=True)
|
||||
return torch.load(pt_path, map_location=device)
|
||||
|
||||
raise FileNotFoundError(f"Could not load {base_name} from English fallback")
|
||||
|
||||
@@ -452,7 +452,7 @@ class ChatterboxTTS:
|
||||
warnings.simplefilter("ignore")
|
||||
|
||||
print(f"📦 Loading checkpoint: {unified_model_path.name}")
|
||||
checkpoint = torch.load(unified_model_path, map_location=device, weights_only=True)
|
||||
checkpoint = torch.load(unified_model_path, map_location=device)
|
||||
|
||||
# Extract model configuration
|
||||
model_config = checkpoint.get('model_config', {})
|
||||
|
||||
@@ -98,7 +98,7 @@ class ChatterboxVC:
|
||||
if safetensors_path.exists():
|
||||
return load_file(safetensors_path, device=device)
|
||||
elif pt_path.exists():
|
||||
return torch.load(pt_path, map_location=device, weights_only=True)
|
||||
return torch.load(pt_path, map_location=device)
|
||||
else:
|
||||
raise FileNotFoundError(f"Neither {base_name}.safetensors nor {base_name}.pt found in {ckpt_dir}")
|
||||
|
||||
@@ -107,7 +107,7 @@ class ChatterboxVC:
|
||||
|
||||
ref_dict = None
|
||||
if (builtin_voice := ckpt_dir / "conds.pt").exists():
|
||||
states = torch.load(builtin_voice, map_location=device, weights_only=True)
|
||||
states = torch.load(builtin_voice, map_location=device)
|
||||
ref_dict = states['gen']
|
||||
|
||||
s3gen = S3Gen()
|
||||
|
||||
@@ -1,5 +1,3 @@
|
||||
import logging
|
||||
|
||||
from .s3tokenizer import (
|
||||
S3_SR,
|
||||
S3_HOP,
|
||||
@@ -29,20 +27,4 @@ def drop_invalid_tokens(x):
|
||||
e = None
|
||||
|
||||
x = x[s: e]
|
||||
|
||||
# T3's speech head spans `speech_tokens_dict_size` (8194) while S3Gen's
|
||||
# input embedding only has SPEECH_VOCAB_SIZE (6561) rows, so a sampled id
|
||||
# above the vocab that is not exactly SoS/EoS reaches the embedding gather
|
||||
# and trips a device-side "index out of bounds" assert (which surfaces
|
||||
# later, at the next CUDA sync, with a misleading traceback).
|
||||
invalid = x >= SPEECH_VOCAB_SIZE
|
||||
if invalid.any():
|
||||
logging.warning(
|
||||
"Dropping %d out-of-vocabulary speech token(s) (max id %d, vocab %d)",
|
||||
int(invalid.sum()), int(x.max()), SPEECH_VOCAB_SIZE,
|
||||
)
|
||||
x = x[~invalid]
|
||||
# TTS Audio Suite patch: do not pass an empty speech stream to S3Gen.
|
||||
if x.numel() == 0:
|
||||
raise RuntimeError("ChatterBox generated no valid speech tokens. Try another seed or text.")
|
||||
return x
|
||||
|
||||
+2
-13
@@ -60,10 +60,6 @@ class AlignmentStreamAnalyzer:
|
||||
# using it for all layers slows things down too much. We can apply it to just one layer
|
||||
# by intercepting the kwargs and adding a forward hook (credit: jrm)
|
||||
self.last_aligned_attns = []
|
||||
# TTS Audio Suite patch: retain hook handles so each inference can remove its
|
||||
# analyzer hooks instead of accumulating three more on the shared model.
|
||||
self._hook_handles = []
|
||||
self._original_output_attentions = getattr(getattr(tfmr, "config", None), "output_attentions", None)
|
||||
for i, (layer_idx, head_idx) in enumerate(LLAMA_ALIGNED_HEADS):
|
||||
self.last_aligned_attns += [None]
|
||||
self._add_attention_spy(tfmr, i, layer_idx, head_idx)
|
||||
@@ -85,18 +81,11 @@ class AlignmentStreamAnalyzer:
|
||||
|
||||
target_layer = tfmr.layers[layer_idx].self_attn
|
||||
# Register hook and store the handle
|
||||
self._hook_handles.append(target_layer.register_forward_hook(attention_forward_hook))
|
||||
target_layer.register_forward_hook(attention_forward_hook)
|
||||
if hasattr(tfmr, 'config') and hasattr(tfmr.config, 'output_attentions'):
|
||||
self.original_output_attentions = tfmr.config.output_attentions
|
||||
tfmr.config.output_attentions = True
|
||||
|
||||
def close(self, tfmr):
|
||||
"""Remove inference-scoped hooks and restore the transformer setting."""
|
||||
for hook_handle in self._hook_handles:
|
||||
hook_handle.remove()
|
||||
self._hook_handles.clear()
|
||||
if self._original_output_attentions is not None:
|
||||
tfmr.config.output_attentions = self._original_output_attentions
|
||||
|
||||
def step(self, logits, next_token=None):
|
||||
"""
|
||||
Emits an AlignmentAnalysisResult into the output queue, and potentially modifies the logits to force an EOS.
|
||||
|
||||
@@ -268,12 +268,6 @@ class T3(nn.Module):
|
||||
# TODO? synchronize the expensive compile function
|
||||
# with self.compile_lock:
|
||||
if not self.compiled:
|
||||
# TTS Audio Suite patch: clean up hooks left by an interrupted prior inference.
|
||||
previous_backend = getattr(self, "patched_model", None)
|
||||
previous_analyzer = getattr(previous_backend, "alignment_stream_analyzer", None)
|
||||
if previous_analyzer is not None:
|
||||
previous_analyzer.close(self.tfmr)
|
||||
|
||||
# Default to None for English models, only create for multilingual
|
||||
alignment_stream_analyzer = None
|
||||
if self.hp.is_multilingual and use_alignment_analyzer:
|
||||
@@ -408,7 +402,4 @@ class T3(nn.Module):
|
||||
|
||||
# Concatenate all predicted tokens along the sequence dimension.
|
||||
predicted_tokens = torch.cat(predicted, dim=1) # shape: (B, num_tokens)
|
||||
if self.patched_model.alignment_stream_analyzer is not None:
|
||||
self.patched_model.alignment_stream_analyzer.close(self.tfmr)
|
||||
self.patched_model.alignment_stream_analyzer = None
|
||||
return predicted_tokens
|
||||
|
||||
@@ -432,7 +432,7 @@ class ChatterboxOfficial23LangTTS:
|
||||
warnings.simplefilter("ignore")
|
||||
|
||||
print(f"📦 Loading checkpoint: {unified_model_path.name}")
|
||||
checkpoint = torch.load(unified_model_path, map_location=actual_device, weights_only=True)
|
||||
checkpoint = torch.load(unified_model_path, map_location=actual_device)
|
||||
|
||||
# Extract model configuration
|
||||
model_config = checkpoint.get('model_config', {})
|
||||
|
||||
@@ -114,7 +114,7 @@ class ChatterboxVC:
|
||||
if safetensors_path.exists():
|
||||
return load_file(safetensors_path, device=device)
|
||||
elif pt_path.exists():
|
||||
return torch.load(pt_path, map_location=device, weights_only=True)
|
||||
return torch.load(pt_path, map_location=device)
|
||||
else:
|
||||
raise FileNotFoundError(f"Neither {base_name}.safetensors nor {base_name}.pt found in {ckpt_dir}")
|
||||
|
||||
@@ -123,7 +123,7 @@ class ChatterboxVC:
|
||||
|
||||
ref_dict = None
|
||||
if (builtin_voice := ckpt_dir / "conds.pt").exists():
|
||||
states = torch.load(builtin_voice, weights_only=True)
|
||||
states = torch.load(builtin_voice)
|
||||
ref_dict = states['gen']
|
||||
|
||||
s3gen = S3Gen()
|
||||
|
||||
@@ -73,8 +73,7 @@ def main():
|
||||
assert num == len(path_list)
|
||||
for path in path_list:
|
||||
print('Processing {}'.format(path))
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
states = torch.load(path, map_location=torch.device('cpu'), weights_only=True)
|
||||
states = torch.load(path, map_location=torch.device('cpu'))
|
||||
for k in states.keys():
|
||||
if k not in ['step', 'epoch']:
|
||||
if k not in avg.keys():
|
||||
|
||||
@@ -134,8 +134,7 @@ def main():
|
||||
start_step, start_epoch = 0, -1
|
||||
if args.checkpoint is not None:
|
||||
if os.path.exists(args.checkpoint):
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
state_dict = torch.load(args.checkpoint, map_location='cpu', weights_only=True)
|
||||
state_dict = torch.load(args.checkpoint, map_location='cpu')
|
||||
model.load_state_dict(state_dict, strict=False)
|
||||
if 'step' in state_dict:
|
||||
start_step = state_dict['step']
|
||||
@@ -162,8 +161,7 @@ def main():
|
||||
# DPO related
|
||||
if args.dpo is True:
|
||||
ref_model = deepcopy(configs[args.model])
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
state_dict = torch.load(args.ref_model, map_location='cpu', weights_only=True)
|
||||
state_dict = torch.load(args.ref_model, map_location='cpu')
|
||||
ref_model.load_state_dict(state_dict, strict=False)
|
||||
dpo_loss = DPOLoss(beta=0.01, label_smoothing=0.0, ipo=False)
|
||||
# NOTE maybe it is not needed to wrap ref_model as ddp because its parameter is not updated
|
||||
|
||||
@@ -47,8 +47,7 @@ class CosyVoiceFrontEnd:
|
||||
providers=["CUDAExecutionProvider" if torch.cuda.is_available() else
|
||||
"CPUExecutionProvider"])
|
||||
if os.path.exists(spk2info):
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
self.spk2info = torch.load(spk2info, map_location=self.device, weights_only=True)
|
||||
self.spk2info = torch.load(spk2info, map_location=self.device)
|
||||
else:
|
||||
self.spk2info = {}
|
||||
self.allowed_special = allowed_special
|
||||
|
||||
@@ -64,8 +64,7 @@ class CosyVoiceModel:
|
||||
self.hift_cache_dict = {}
|
||||
|
||||
def load(self, llm_model, flow_model, hift_model):
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
self.llm.load_state_dict(torch.load(llm_model, map_location=self.device, weights_only=True), strict=True)
|
||||
self.llm.load_state_dict(torch.load(llm_model, map_location=self.device), strict=True)
|
||||
self.llm.to(self.device).eval()
|
||||
# FIX: Bundled code fix - Ensure all nested modules are moved to device
|
||||
# Issue: Some nested layers (embed_tokens, etc.) weren't being moved with .to(device)
|
||||
@@ -73,16 +72,14 @@ class CosyVoiceModel:
|
||||
for module in self.llm.modules():
|
||||
module.to(self.device)
|
||||
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
self.flow.load_state_dict(torch.load(flow_model, map_location=self.device, weights_only=True), strict=True)
|
||||
self.flow.load_state_dict(torch.load(flow_model, map_location=self.device), strict=True)
|
||||
self.flow.to(self.device).eval()
|
||||
# FIX: Bundled code fix - Ensure all nested modules are moved to device
|
||||
for module in self.flow.modules():
|
||||
module.to(self.device)
|
||||
|
||||
# in case hift_model is a hifigan model
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
hift_state_dict = {k.replace('generator.', ''): v for k, v in torch.load(hift_model, map_location=self.device, weights_only=True).items()}
|
||||
hift_state_dict = {k.replace('generator.', ''): v for k, v in torch.load(hift_model, map_location=self.device).items()}
|
||||
self.hift.load_state_dict(hift_state_dict, strict=True)
|
||||
self.hift.to(self.device).eval()
|
||||
# FIX: Bundled code fix - Ensure all nested modules are moved to device
|
||||
|
||||
@@ -84,8 +84,7 @@ def assert_required_models_available(args):
|
||||
def load_hifigan(checkpoint_path, device):
|
||||
h = AttrDict(v1)
|
||||
hifigan = HiFiGAN(h).to(device)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
hifigan.load_state_dict(torch.load(checkpoint_path, map_location=device, weights_only=True)["generator"])
|
||||
hifigan.load_state_dict(torch.load(checkpoint_path, map_location=device)["generator"])
|
||||
_ = hifigan.eval()
|
||||
hifigan.remove_weight_norm()
|
||||
return hifigan
|
||||
|
||||
@@ -41,8 +41,7 @@ def get_padding(kernel_size, dilation=1):
|
||||
def load_checkpoint(filepath, device):
|
||||
assert os.path.isfile(filepath)
|
||||
print(f"Loading '{filepath}'")
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint_dict = torch.load(filepath, map_location=device, weights_only=True)
|
||||
checkpoint_dict = torch.load(filepath, map_location=device)
|
||||
print("Complete.")
|
||||
return checkpoint_dict
|
||||
|
||||
|
||||
@@ -15,8 +15,6 @@ from utils.hf_download_logging import quiet_hf_download_logs
|
||||
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
|
||||
|
||||
|
||||
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
|
||||
|
||||
class DotsTTSDownloader:
|
||||
"""Resolve and download official dots.tts model folders."""
|
||||
|
||||
@@ -61,7 +59,6 @@ class DotsTTSDownloader:
|
||||
self.base_path = base_path
|
||||
os.makedirs(self.base_path, exist_ok=True)
|
||||
|
||||
@validate_model_paths
|
||||
def resolve_model_path(self, model_identifier: str) -> str:
|
||||
"""Resolve local: names, absolute paths, or known official model identifiers."""
|
||||
if not model_identifier:
|
||||
@@ -115,7 +112,7 @@ class DotsTTSDownloader:
|
||||
with quiet_hf_download_logs():
|
||||
snapshot_download(
|
||||
repo_id=repo_id,
|
||||
local_dir=allowed_path(model_dir),
|
||||
local_dir=model_dir,
|
||||
local_dir_use_symlinks=False,
|
||||
resume_download=True,
|
||||
force_download=force,
|
||||
|
||||
@@ -4,7 +4,6 @@ Dots TTS engine wrapper.
|
||||
Wraps the official dots.tts runtime with ComfyUI-friendly model lifecycle hooks.
|
||||
"""
|
||||
|
||||
from utils.security.data_loading import RestrictedLoadModule, load_dependency_checkpoint
|
||||
import os
|
||||
import sys
|
||||
import types
|
||||
@@ -109,9 +108,6 @@ class DotsTTSEngine:
|
||||
try:
|
||||
with self._text_normalizer_compat():
|
||||
from dots_tts.runtime import DotsTtsRuntime
|
||||
from dots_tts.models.dots_tts import core
|
||||
if not isinstance(core.torch, RestrictedLoadModule):
|
||||
core.torch = RestrictedLoadModule(core.torch, load_dependency_checkpoint)
|
||||
except Exception as e:
|
||||
for module_name, module in stale_modules:
|
||||
sys.modules.setdefault(module_name, module)
|
||||
|
||||
@@ -9,8 +9,6 @@ from utils.downloads.unified_downloader import unified_downloader
|
||||
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
|
||||
|
||||
|
||||
from utils.security.path_access import validate_model_paths, allowed_path
|
||||
|
||||
class DramaBoxDownloader:
|
||||
"""Resolve DramaBox checkpoints without using Hugging Face cache storage."""
|
||||
|
||||
@@ -67,7 +65,6 @@ class DramaBoxDownloader:
|
||||
models.insert(0, local_name)
|
||||
return models
|
||||
|
||||
@validate_model_paths
|
||||
def resolve_model_path(self, model_identifier: str = MODEL_NAME) -> Dict[str, str]:
|
||||
model_identifier = model_identifier or self.MODEL_NAME
|
||||
if os.path.isabs(model_identifier) and os.path.isdir(model_identifier):
|
||||
|
||||
@@ -28,8 +28,6 @@ class DramaBoxEngine:
|
||||
memory_mode: str = "fast",
|
||||
transformer_quantization: str = "none",
|
||||
compile_model: bool = False,
|
||||
lora_path: str = "",
|
||||
lora_strength: float = 1.0,
|
||||
):
|
||||
self.model_name = model_name
|
||||
self.device = resolve_torch_device(device)
|
||||
@@ -38,8 +36,6 @@ class DramaBoxEngine:
|
||||
self.memory_mode = str(memory_mode)
|
||||
self.transformer_quantization = str(transformer_quantization)
|
||||
self.compile_model = bool(compile_model)
|
||||
self.lora_path = str(lora_path or "").strip()
|
||||
self.lora_strength = float(lora_strength)
|
||||
self._server = None
|
||||
self._server_module = None
|
||||
|
||||
@@ -108,8 +104,6 @@ class DramaBoxEngine:
|
||||
bnb_4bit=True,
|
||||
memory_mode=self.memory_mode,
|
||||
transformer_quantization=self.transformer_quantization,
|
||||
lora_path=self.lora_path,
|
||||
lora_strength=self.lora_strength,
|
||||
)
|
||||
print("✅ DramaBox runtime ready")
|
||||
|
||||
@@ -175,17 +169,6 @@ class DramaBoxEngine:
|
||||
"sample_rate": int(sample_rate),
|
||||
}
|
||||
|
||||
def set_lora(self, lora_path: str = "", strength: float = 1.0, revision: str = ""):
|
||||
"""Update the live adapter without rebuilding the base DramaBox runtime."""
|
||||
self.lora_path = str(lora_path or "").strip()
|
||||
self.lora_strength = float(strength)
|
||||
if self._server is not None:
|
||||
self._server.configure_lora(
|
||||
self.lora_path,
|
||||
self.lora_strength,
|
||||
revision=str(revision or ""),
|
||||
)
|
||||
|
||||
def parameters(self) -> Iterator[torch.nn.Parameter]:
|
||||
"""Expose loaded submodule parameters for ComfyUI memory accounting."""
|
||||
if self._server is None:
|
||||
|
||||
@@ -1,5 +0,0 @@
|
||||
"""DramaBox LoRA dataset and training integration."""
|
||||
|
||||
from .handler import DramaBoxTrainingHandler
|
||||
|
||||
__all__ = ["DramaBoxTrainingHandler"]
|
||||
@@ -1,452 +0,0 @@
|
||||
"""Dataset normalization for the official DramaBox IC-LoRA trainer.
|
||||
|
||||
The upstream preprocessor accepts JSONL and TSV, but the upstream training
|
||||
loop builds its speaker map from ``~``-delimited index rows. This module keeps
|
||||
that conversion in the suite so a manifest that is valid for preprocessing is
|
||||
also valid for training.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from utils.security.path_access import resolve_input_path, allowed_path, child_path
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import wave
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, List, Optional, Tuple
|
||||
|
||||
import folder_paths
|
||||
|
||||
|
||||
AUDIO_EXTENSIONS = {".wav", ".flac", ".mp3", ".ogg", ".m4a", ".aac"}
|
||||
PREPROCESSED_SAMPLE_PATTERN = re.compile(r"sample_(\d+)\.pt$")
|
||||
|
||||
|
||||
def slugify(value: Any) -> str:
|
||||
safe = "".join(
|
||||
ch if ch.isalnum() or ch in ("-", "_") else "_"
|
||||
for ch in str(value or "").strip()
|
||||
)
|
||||
safe = safe.strip("_")
|
||||
return safe or "dramabox_lora"
|
||||
|
||||
|
||||
def get_dramabox_training_root() -> str:
|
||||
root = os.path.join(
|
||||
folder_paths.get_output_directory(), "tts_audio_suite_training", "dramabox"
|
||||
)
|
||||
os.makedirs(root, exist_ok=True)
|
||||
return root
|
||||
|
||||
|
||||
def _resolve_source_path(value: str) -> Path:
|
||||
path = Path(resolve_input_path(value, datasets=True, models=False))
|
||||
if not (path.is_file()):
|
||||
raise FileNotFoundError(f"Dataset source not found: {value}")
|
||||
return path
|
||||
|
||||
|
||||
def _resolve_audio_path(raw_path: Any, *, source_path: Path, audio_dir: str) -> Path:
|
||||
value = os.path.expanduser(str(raw_path or "").strip())
|
||||
if not value:
|
||||
raise ValueError("Dataset row is missing audio_filepath/audio_path")
|
||||
|
||||
candidates: List[Path] = []
|
||||
if os.path.isabs(value):
|
||||
candidates.append(Path(value))
|
||||
else:
|
||||
if audio_dir:
|
||||
candidates.append(Path(os.path.expanduser(audio_dir)) / value)
|
||||
candidates.append(source_path.parent / value)
|
||||
candidates.append(Path(value))
|
||||
|
||||
for candidate in candidates:
|
||||
if candidate.is_file():
|
||||
return Path(allowed_path(candidate, models=False))
|
||||
raise FileNotFoundError(f"DramaBox audio file not found: {raw_path}")
|
||||
|
||||
|
||||
def _clean_text(value: Any) -> str:
|
||||
return re.sub(r"\s+", " ", str(value or "").replace("\x00", "")).strip()
|
||||
|
||||
|
||||
def _speaker_value(row: Dict[str, Any], default: str = "speaker_1") -> str:
|
||||
value = (
|
||||
row.get("speaker")
|
||||
or row.get("speaker_id")
|
||||
or row.get("voice")
|
||||
or row.get("character")
|
||||
or default
|
||||
)
|
||||
return _clean_text(value).replace("~", "_") or default
|
||||
|
||||
|
||||
def _language_value(row: Dict[str, Any]) -> str:
|
||||
return _clean_text(row.get("language") or row.get("lang") or "en").replace("~", "_") or "en"
|
||||
|
||||
|
||||
def _coerce_float(value: Any, default: float = 0.0) -> float:
|
||||
try:
|
||||
parsed = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return float(default)
|
||||
return parsed if parsed > 0 else float(default)
|
||||
|
||||
|
||||
def _probe_audio(path: Path) -> Tuple[int, int, float]:
|
||||
"""Return sample rate, frame count, and duration without loading audio."""
|
||||
try:
|
||||
import torchaudio
|
||||
|
||||
info = torchaudio.info(str(path))
|
||||
sample_rate = int(getattr(info, "sample_rate", 0) or 0)
|
||||
frames = int(getattr(info, "num_frames", 0) or 0)
|
||||
if sample_rate > 0 and frames > 0:
|
||||
return sample_rate, frames, frames / sample_rate
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if path.suffix.lower() == ".wav":
|
||||
with wave.open(str(path), "rb") as handle:
|
||||
sample_rate = int(handle.getframerate())
|
||||
frames = int(handle.getnframes())
|
||||
if sample_rate > 0 and frames > 0:
|
||||
return sample_rate, frames, frames / sample_rate
|
||||
|
||||
raise RuntimeError(
|
||||
f"Could not inspect audio duration for '{path}'. Add a positive duration "
|
||||
"field to the manifest or install a Torchaudio-compatible decoder."
|
||||
)
|
||||
|
||||
|
||||
def _parse_manifest(source_path: Path, audio_dir: str) -> Iterable[Dict[str, Any]]:
|
||||
text = source_path.read_text(encoding="utf-8-sig")
|
||||
stripped = text.lstrip()
|
||||
if stripped.startswith("["):
|
||||
raw_rows = json.loads(text)
|
||||
else:
|
||||
raw_rows = [json.loads(line) for line in text.splitlines() if line.strip()]
|
||||
for row in raw_rows:
|
||||
if not isinstance(row, dict):
|
||||
continue
|
||||
yield {
|
||||
"audio": _resolve_audio_path(
|
||||
row.get("audio_filepath", row.get("audio_path", row.get("audio"))),
|
||||
source_path=source_path,
|
||||
audio_dir=audio_dir,
|
||||
),
|
||||
"text": _clean_text(row.get("text", row.get("transcript", ""))),
|
||||
"duration": _coerce_float(row.get("duration")),
|
||||
"sample_rate": int(_coerce_float(row.get("sample_rate"))),
|
||||
"samples": int(_coerce_float(row.get("samples", row.get("num_frames")))),
|
||||
"speaker": _speaker_value(row),
|
||||
"language": _language_value(row),
|
||||
}
|
||||
|
||||
|
||||
def _parse_tsv(source_path: Path, audio_dir: str) -> Iterable[Dict[str, Any]]:
|
||||
with source_path.open("r", encoding="utf-8-sig", newline="") as handle:
|
||||
for row_number, row in enumerate(csv.reader(handle, delimiter="\t"), start=1):
|
||||
if len(row) < 2:
|
||||
continue
|
||||
yield {
|
||||
"audio": _resolve_audio_path(row[0], source_path=source_path, audio_dir=audio_dir),
|
||||
"text": _clean_text(row[1]),
|
||||
"duration": _coerce_float(row[2]) if len(row) > 2 else 0.0,
|
||||
"sample_rate": 0,
|
||||
"samples": 0,
|
||||
"speaker": _clean_text(row[3]).replace("~", "_") if len(row) > 3 else "speaker_1",
|
||||
"language": _clean_text(row[4]).replace("~", "_") if len(row) > 4 else "en",
|
||||
"row_number": row_number,
|
||||
}
|
||||
|
||||
|
||||
def _parse_gemini(source_path: Path, audio_dir: str) -> Iterable[Dict[str, Any]]:
|
||||
for line in source_path.read_text(encoding="utf-8-sig").splitlines():
|
||||
parts = line.strip().split("~")
|
||||
if len(parts) < 8:
|
||||
continue
|
||||
file_id, speaker, language = parts[:3]
|
||||
sample_rate = int(_coerce_float(parts[3], 24000))
|
||||
samples = int(_coerce_float(parts[4]))
|
||||
duration = _coerce_float(parts[5])
|
||||
text = _clean_text(parts[-1])
|
||||
yield {
|
||||
"audio": _resolve_audio_path(file_id, source_path=source_path, audio_dir=audio_dir),
|
||||
"text": text,
|
||||
"duration": duration,
|
||||
"sample_rate": sample_rate,
|
||||
"samples": samples,
|
||||
"speaker": _clean_text(speaker).replace("~", "_") or "speaker_1",
|
||||
"language": _clean_text(language).replace("~", "_") or "en",
|
||||
}
|
||||
|
||||
|
||||
def _parse_libriheavy(source_path: Path, audio_dir: str) -> Iterable[Dict[str, Any]]:
|
||||
for line in source_path.read_text(encoding="utf-8-sig").splitlines():
|
||||
parts = line.strip().split("~")
|
||||
if len(parts) < 7:
|
||||
continue
|
||||
file_id, speaker, language = parts[:3]
|
||||
# Format: id~speaker~lang~samples~duration_ms~phonemes~text.
|
||||
sample_rate = 24000
|
||||
samples = int(_coerce_float(parts[3]))
|
||||
duration = _coerce_float(parts[4]) / 1000.0 if len(parts) >= 5 else 0.0
|
||||
yield {
|
||||
"audio": _resolve_audio_path(file_id, source_path=source_path, audio_dir=audio_dir),
|
||||
"text": _clean_text(parts[-1]),
|
||||
"duration": duration,
|
||||
"sample_rate": sample_rate,
|
||||
"samples": samples,
|
||||
"speaker": _clean_text(speaker).replace("~", "_") or "speaker_1",
|
||||
"language": _clean_text(language).replace("~", "_") or "en",
|
||||
}
|
||||
|
||||
|
||||
def _raw_rows(source_path: Path, dataset_type: str, audio_dir: str) -> Iterable[Dict[str, Any]]:
|
||||
parsers = {
|
||||
"manifest": _parse_manifest,
|
||||
"tsv": _parse_tsv,
|
||||
"gemini_synthetic": _parse_gemini,
|
||||
"libriheavy": _parse_libriheavy,
|
||||
}
|
||||
try:
|
||||
parser = parsers[str(dataset_type)]
|
||||
except KeyError as exc:
|
||||
raise ValueError(f"Unsupported DramaBox dataset type: {dataset_type}") from exc
|
||||
return parser(source_path, audio_dir)
|
||||
|
||||
|
||||
def _fingerprint(source_path: Path, *, dataset_type: str, audio_dir: str, min_duration: float, max_duration: float) -> str:
|
||||
stat = source_path.stat()
|
||||
raw = f"{source_path}|{stat.st_size}|{stat.st_mtime_ns}|{dataset_type}|{audio_dir}|{min_duration}|{max_duration}"
|
||||
return hashlib.sha256(raw.encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
|
||||
def _normalize_rows(
|
||||
source_path: Path,
|
||||
*,
|
||||
dataset_type: str,
|
||||
audio_dir: str,
|
||||
min_duration: float,
|
||||
max_duration: float,
|
||||
) -> List[Dict[str, Any]]:
|
||||
records: List[Dict[str, Any]] = []
|
||||
for row_index, row in enumerate(_raw_rows(source_path, dataset_type, audio_dir)):
|
||||
text = _clean_text(row.get("text"))
|
||||
if not text:
|
||||
continue
|
||||
|
||||
audio = Path(row["audio"]).resolve()
|
||||
sample_rate = int(row.get("sample_rate") or 0)
|
||||
samples = int(row.get("samples") or 0)
|
||||
duration = _coerce_float(row.get("duration"))
|
||||
if not sample_rate or not samples or not duration:
|
||||
try:
|
||||
probed_rate, probed_samples, probed_duration = _probe_audio(audio)
|
||||
sample_rate = sample_rate or probed_rate
|
||||
samples = samples or probed_samples
|
||||
duration = duration or probed_duration
|
||||
except RuntimeError:
|
||||
if duration <= 0:
|
||||
raise
|
||||
sample_rate = sample_rate or 24000
|
||||
samples = samples or max(1, round(duration * sample_rate))
|
||||
|
||||
if duration < float(min_duration) or duration > float(max_duration):
|
||||
continue
|
||||
records.append(
|
||||
{
|
||||
"id": f"sample_{row_index:06d}",
|
||||
"audio": str(audio),
|
||||
"text": text,
|
||||
"duration": float(duration),
|
||||
"sample_rate": int(sample_rate),
|
||||
"samples": int(samples),
|
||||
"speaker": _speaker_value(row),
|
||||
"language": _language_value(row),
|
||||
}
|
||||
)
|
||||
|
||||
if not records:
|
||||
raise ValueError(
|
||||
"DramaBox dataset preparation produced no usable rows. Check the audio paths, "
|
||||
"transcripts, and the min/max duration filters."
|
||||
)
|
||||
|
||||
speaker_counts: Dict[str, int] = {}
|
||||
for record in records:
|
||||
speaker_counts[record["speaker"]] = speaker_counts.get(record["speaker"], 0) + 1
|
||||
unusable = sorted(name for name, count in speaker_counts.items() if count < 2)
|
||||
if unusable:
|
||||
raise ValueError(
|
||||
"DramaBox LoRA training needs at least two clips per speaker so the official "
|
||||
f"trainer can choose a reference clip. Speakers with fewer than two clips: {', '.join(unusable)}."
|
||||
)
|
||||
return records
|
||||
|
||||
|
||||
def _write_index(records: List[Dict[str, Any]], index_path: Path) -> None:
|
||||
index_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with index_path.open("w", encoding="utf-8") as handle:
|
||||
for record in records:
|
||||
text = str(record["text"]).replace("\r", " ").replace("\n", " ")
|
||||
handle.write(
|
||||
"~".join(
|
||||
(
|
||||
str(Path(record["audio"]).resolve()),
|
||||
str(record["speaker"]),
|
||||
str(record["language"]),
|
||||
str(int(record["sample_rate"])),
|
||||
str(int(record["samples"])),
|
||||
f"{float(record['duration']):.6f}",
|
||||
"_",
|
||||
text,
|
||||
)
|
||||
)
|
||||
+ "\n"
|
||||
)
|
||||
|
||||
|
||||
def _preprocessed_indices(directory: Path) -> set[int]:
|
||||
indices: set[int] = set()
|
||||
if not directory.is_dir():
|
||||
return indices
|
||||
for path in directory.glob("sample_*.pt"):
|
||||
match = PREPROCESSED_SAMPLE_PATTERN.fullmatch(path.name)
|
||||
if match:
|
||||
indices.add(int(match.group(1)))
|
||||
return indices
|
||||
|
||||
|
||||
def validate_preprocessed_dataset(
|
||||
records: List[Dict[str, Any]],
|
||||
preprocessed_dir: str | Path,
|
||||
*,
|
||||
raise_on_missing: bool = False,
|
||||
) -> bool:
|
||||
"""Require matching text conditions and audio latents for every index row."""
|
||||
root = Path(preprocessed_dir)
|
||||
expected = set(range(len(records)))
|
||||
available = _preprocessed_indices(root / "conditions") & _preprocessed_indices(
|
||||
root / "audio_latents"
|
||||
)
|
||||
missing = sorted(expected - available)
|
||||
complete = bool(expected) and not missing
|
||||
if raise_on_missing and not complete:
|
||||
preview = ", ".join(str(index) for index in missing[:10]) or "all"
|
||||
suffix = "..." if len(missing) > 10 else ""
|
||||
raise RuntimeError(
|
||||
"DramaBox preprocessing did not produce matching condition/audio-latent "
|
||||
f"files for {len(missing) or len(expected)} sample(s) (indices: {preview}{suffix}). "
|
||||
"Fix the reported source-audio errors and run Dataset Prep again."
|
||||
)
|
||||
return complete
|
||||
|
||||
|
||||
def prepare_dramabox_dataset(
|
||||
shared_settings: Dict[str, Any],
|
||||
*,
|
||||
dataset_source: str,
|
||||
model_name: str,
|
||||
dataset_type: str = "manifest",
|
||||
audio_dir: str = "",
|
||||
min_duration: float = 2.0,
|
||||
max_duration: float = 20.0,
|
||||
reuse_existing: bool = True,
|
||||
preprocess_now: bool = True,
|
||||
dry_run: bool = False,
|
||||
) -> Dict[str, Any]:
|
||||
source_path = _resolve_source_path(dataset_source)
|
||||
fingerprint = _fingerprint(
|
||||
source_path,
|
||||
dataset_type=dataset_type,
|
||||
audio_dir=audio_dir,
|
||||
min_duration=min_duration,
|
||||
max_duration=max_duration,
|
||||
)
|
||||
safe_name = slugify(model_name)
|
||||
dataset_root = Path(get_dramabox_training_root()) / "datasets" / f"{safe_name}_{fingerprint}"
|
||||
index_path = dataset_root / "speaker_index.txt"
|
||||
metadata_path = dataset_root / "dataset.json"
|
||||
preprocessed_dir = dataset_root / "preprocessed"
|
||||
|
||||
if reuse_existing and metadata_path.is_file() and index_path.is_file():
|
||||
try:
|
||||
metadata = json.loads(metadata_path.read_text(encoding="utf-8"))
|
||||
records = metadata.get("records") or []
|
||||
except Exception:
|
||||
records = []
|
||||
else:
|
||||
records = []
|
||||
|
||||
if not records:
|
||||
records = _normalize_rows(
|
||||
source_path,
|
||||
dataset_type=dataset_type,
|
||||
audio_dir=audio_dir,
|
||||
min_duration=float(min_duration),
|
||||
max_duration=float(max_duration),
|
||||
)
|
||||
dataset_root.mkdir(parents=True, exist_ok=True)
|
||||
_write_index(records, index_path)
|
||||
metadata_path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"type": "dramabox_dataset",
|
||||
"source_path": str(source_path),
|
||||
"dataset_type": dataset_type,
|
||||
"audio_dir": audio_dir,
|
||||
"min_duration": float(min_duration),
|
||||
"max_duration": float(max_duration),
|
||||
"records": records,
|
||||
},
|
||||
indent=2,
|
||||
ensure_ascii=False,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
# Rewrite cached indexes as well so datasets prepared by older suite
|
||||
# builds migrate from synthetic sample ids to resolvable audio paths.
|
||||
_write_index(records, index_path)
|
||||
|
||||
dataset: Dict[str, Any] = {
|
||||
"type": "training_dataset",
|
||||
"engine_type": "dramabox",
|
||||
"training_mode": "audio_lora",
|
||||
"model_name": model_name,
|
||||
"dataset_type": dataset_type,
|
||||
"source_path": str(source_path),
|
||||
"index_path": str(index_path),
|
||||
"speaker_index": str(index_path),
|
||||
"data_dir": [str(preprocessed_dir)],
|
||||
"preprocessed_dir": str(preprocessed_dir),
|
||||
"min_duration": float(min_duration),
|
||||
"max_duration": float(max_duration),
|
||||
"records": records,
|
||||
"train_records": len(records),
|
||||
"speakers": sorted({str(record["speaker"]) for record in records}),
|
||||
"preprocessed": validate_preprocessed_dataset(records, preprocessed_dir),
|
||||
"dry_run": bool(dry_run),
|
||||
"shared_settings": dict(shared_settings or {}),
|
||||
}
|
||||
|
||||
if preprocess_now and not dry_run and not dataset["preprocessed"]:
|
||||
from .trainer import run_dramabox_preprocess
|
||||
|
||||
run_dramabox_preprocess(dataset, shared_settings, batch_size=8)
|
||||
dataset["preprocessed"] = True
|
||||
|
||||
return dataset
|
||||
|
||||
|
||||
__all__ = [
|
||||
"get_dramabox_training_root",
|
||||
"prepare_dramabox_dataset",
|
||||
"slugify",
|
||||
"validate_preprocessed_dataset",
|
||||
]
|
||||
@@ -1,82 +0,0 @@
|
||||
"""DramaBox backend for the unified model-training node."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict
|
||||
|
||||
from engines.training.base_handler import BaseTrainingHandler
|
||||
from engines.training.registry import register_training_handler
|
||||
|
||||
|
||||
class DramaBoxTrainingHandler(BaseTrainingHandler):
|
||||
engine_type = "dramabox"
|
||||
artifact_type = "lora_adapter"
|
||||
|
||||
def _shared_settings(self, tts_engine: Any) -> Dict[str, Any]:
|
||||
config = self.ensure_engine_type(tts_engine)
|
||||
return {
|
||||
"model_name": config.get("model_name", "DramaBox"),
|
||||
"device": str(config.get("device", "auto")),
|
||||
"precision": str(config.get("precision", "auto")),
|
||||
}
|
||||
|
||||
def build_default_training_config(self, tts_engine: Any) -> Dict[str, Any]:
|
||||
self._shared_settings(tts_engine)
|
||||
return {
|
||||
"type": "training_config",
|
||||
"engine_type": "dramabox",
|
||||
"training_mode": "audio_lora",
|
||||
"base_model": "dev",
|
||||
"steps": 10000,
|
||||
"learning_rate": 1e-4,
|
||||
"lr_scheduler": "cosine",
|
||||
"warmup_steps": 500,
|
||||
"batch_size": 1,
|
||||
"grad_accum": 4,
|
||||
"max_grad_norm": 1.0,
|
||||
"save_every": 500,
|
||||
"log_every": 10,
|
||||
"seed": 42,
|
||||
"lora_rank": 128,
|
||||
"lora_alpha": 128,
|
||||
"lora_dropout": 0.1,
|
||||
"ref_ratio": 0.3,
|
||||
"max_ref_tokens": 200,
|
||||
"text_dropout": 0.4,
|
||||
"preprocess_batch_size": 8,
|
||||
"validation_config": "",
|
||||
"validation_gpu": "",
|
||||
"dry_run": False,
|
||||
}
|
||||
|
||||
def prepare_dataset(self, tts_engine: Any, **kwargs) -> Dict[str, Any]:
|
||||
from .dataset import prepare_dramabox_dataset
|
||||
|
||||
return prepare_dramabox_dataset(self._shared_settings(tts_engine), **kwargs)
|
||||
|
||||
def train(
|
||||
self,
|
||||
tts_engine: Any,
|
||||
training_dataset: Dict[str, Any],
|
||||
training_config: Dict[str, Any],
|
||||
output_name: str = "",
|
||||
resume: bool = False,
|
||||
overwrite: bool = False,
|
||||
continue_from: Any = None,
|
||||
node_id: str = "",
|
||||
) -> Dict[str, Any]:
|
||||
from .trainer import run_dramabox_training_job
|
||||
|
||||
return run_dramabox_training_job(
|
||||
shared_settings=self._shared_settings(tts_engine),
|
||||
dataset_info=training_dataset,
|
||||
training_config=training_config,
|
||||
output_name=output_name,
|
||||
resume=resume,
|
||||
overwrite=overwrite,
|
||||
continue_from=continue_from,
|
||||
node_id=node_id,
|
||||
)
|
||||
|
||||
|
||||
register_training_handler("dramabox", DramaBoxTrainingHandler)
|
||||
@@ -1,687 +0,0 @@
|
||||
"""Process runner for the official DramaBox IC-LoRA trainer."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, Optional
|
||||
|
||||
import folder_paths
|
||||
|
||||
from engines.dramabox.dramabox_downloader import DramaBoxDownloader
|
||||
from engines.training.progress_io import write_json_progress_file
|
||||
from engines.training.progress_registry import (
|
||||
finalize_training_job,
|
||||
register_training_job,
|
||||
update_training_job,
|
||||
)
|
||||
|
||||
from .dataset import (
|
||||
get_dramabox_training_root,
|
||||
slugify,
|
||||
validate_preprocessed_dataset,
|
||||
)
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[3]
|
||||
VENDOR_ROOT = PROJECT_ROOT / "engines" / "dramabox" / "vendor"
|
||||
PREPROCESS_SCRIPT = VENDOR_ROOT / "src" / "preprocess.py"
|
||||
TRAIN_SCRIPT = VENDOR_ROOT / "src" / "train.py"
|
||||
|
||||
|
||||
def _write_progress(progress_file: str, *, status: str, phase: str, **updates: Any) -> None:
|
||||
payload: Dict[str, Any] = {}
|
||||
if progress_file and os.path.isfile(progress_file):
|
||||
try:
|
||||
with open(progress_file, "r", encoding="utf-8") as handle:
|
||||
existing = json.load(handle)
|
||||
if isinstance(existing, dict):
|
||||
payload.update(existing)
|
||||
except Exception:
|
||||
pass
|
||||
payload.update(updates)
|
||||
payload["status"] = status
|
||||
payload["phase"] = phase
|
||||
payload["updated_at"] = datetime.now().isoformat()
|
||||
if progress_file:
|
||||
write_json_progress_file(progress_file, payload, default=str)
|
||||
|
||||
|
||||
def _interrupt_requested() -> bool:
|
||||
try:
|
||||
import comfy.model_management as model_management
|
||||
except Exception:
|
||||
return False
|
||||
try:
|
||||
return bool(model_management.processing_interrupted())
|
||||
except Exception:
|
||||
return bool(getattr(model_management, "interrupt_processing", False))
|
||||
|
||||
|
||||
def _device_environment(shared_settings: Dict[str, Any]) -> Dict[str, str]:
|
||||
env = os.environ.copy()
|
||||
device = str(shared_settings.get("device", "auto") or "auto").strip().lower()
|
||||
if device.startswith("cpu"):
|
||||
# CPU mode is explicit. This also prevents a CUDA-enabled torch build
|
||||
# from silently taking the user's GPU during preprocessing.
|
||||
env["CUDA_VISIBLE_DEVICES"] = ""
|
||||
elif device.startswith("cuda:"):
|
||||
env["CUDA_VISIBLE_DEVICES"] = device.split(":", 1)[1]
|
||||
return env
|
||||
|
||||
|
||||
def _run_process(
|
||||
command: Iterable[str],
|
||||
*,
|
||||
cwd: Path,
|
||||
env: Dict[str, str],
|
||||
phase: str,
|
||||
progress_file: str = "",
|
||||
node_id: str = "",
|
||||
total_steps: int = 0,
|
||||
) -> None:
|
||||
command = [str(value) for value in command]
|
||||
print(f"🎓 DramaBox {phase} command: {' '.join(command)}")
|
||||
process = subprocess.Popen(
|
||||
command,
|
||||
cwd=str(cwd),
|
||||
env=env,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
bufsize=1,
|
||||
)
|
||||
tail: list[str] = []
|
||||
recent_loss_trace: list[Dict[str, Any]] = []
|
||||
best_loss: Optional[float] = None
|
||||
try:
|
||||
assert process.stdout is not None
|
||||
for raw_line in process.stdout:
|
||||
line = raw_line.rstrip()
|
||||
if line:
|
||||
telemetry_match = re.fullmatch(
|
||||
r"TTS_SUITE_PROGRESS\s+step=(\d+)\s+total=(\d+)", line
|
||||
)
|
||||
if telemetry_match is None:
|
||||
print(f"[DramaBox {phase}] {line}")
|
||||
tail.append(line)
|
||||
del tail[:-30]
|
||||
|
||||
if progress_file:
|
||||
match = telemetry_match or re.search(
|
||||
r"(?:Step|step)\s+(\d+)(?:/(\d+))?", line
|
||||
)
|
||||
if match:
|
||||
step = int(match.group(1))
|
||||
parsed_total = int(match.group(2) or total_steps or 0)
|
||||
overall_progress = (step / parsed_total) if parsed_total else 0.0
|
||||
progress_updates: Dict[str, Any] = {
|
||||
"step": step,
|
||||
"total_steps": parsed_total,
|
||||
"overall_progress": overall_progress,
|
||||
"latest_log": line,
|
||||
}
|
||||
loss_match = re.search(
|
||||
r"\bloss=([-+0-9.eE]+)", line, re.IGNORECASE
|
||||
)
|
||||
if loss_match:
|
||||
loss_value = float(loss_match.group(1))
|
||||
lr_match = re.search(
|
||||
r"\blr=([-+0-9.eE]+)", line, re.IGNORECASE
|
||||
)
|
||||
learning_rate = (
|
||||
float(lr_match.group(1)) if lr_match else None
|
||||
)
|
||||
recent_loss_trace.append(
|
||||
{"step": step, "total_loss": loss_value}
|
||||
)
|
||||
recent_loss_trace = recent_loss_trace[-120:]
|
||||
best_loss = (
|
||||
loss_value
|
||||
if best_loss is None
|
||||
else min(best_loss, loss_value)
|
||||
)
|
||||
progress_updates.update(
|
||||
latest_loss=loss_value,
|
||||
best_gen_loss=best_loss,
|
||||
recent_loss_trace=recent_loss_trace,
|
||||
current_metrics={
|
||||
"loss_gen_all": loss_value,
|
||||
"loss_disc_all": 0.0,
|
||||
"loss_mel": 0.0,
|
||||
"loss_kl": 0.0,
|
||||
"loss_fm": 0.0,
|
||||
"learning_rate": learning_rate,
|
||||
},
|
||||
)
|
||||
_write_progress(
|
||||
progress_file,
|
||||
status="running",
|
||||
phase=phase,
|
||||
**progress_updates,
|
||||
)
|
||||
update_training_job(
|
||||
node_id,
|
||||
status="running",
|
||||
phase=phase,
|
||||
**progress_updates,
|
||||
)
|
||||
elif "encoding:" in line.lower():
|
||||
match = re.search(r"(\d+)\s*/\s*(\d+)", line)
|
||||
if match:
|
||||
step = int(match.group(1))
|
||||
parsed_total = int(match.group(2))
|
||||
overall_progress = step / max(parsed_total, 1)
|
||||
_write_progress(
|
||||
progress_file,
|
||||
status="running",
|
||||
phase=phase,
|
||||
step=step,
|
||||
total_steps=parsed_total,
|
||||
overall_progress=overall_progress,
|
||||
latest_log=line,
|
||||
)
|
||||
update_training_job(
|
||||
node_id,
|
||||
status="running",
|
||||
phase=phase,
|
||||
step=step,
|
||||
total_steps=parsed_total,
|
||||
overall_progress=overall_progress,
|
||||
latest_log=line,
|
||||
)
|
||||
|
||||
if _interrupt_requested():
|
||||
process.terminate()
|
||||
try:
|
||||
process.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
process.kill()
|
||||
raise InterruptedError(f"DramaBox {phase} interrupted by user")
|
||||
|
||||
return_code = process.wait()
|
||||
except BaseException:
|
||||
if process.poll() is None:
|
||||
process.terminate()
|
||||
raise
|
||||
|
||||
if return_code != 0:
|
||||
details = "\n".join(tail[-10:])
|
||||
raise RuntimeError(
|
||||
f"DramaBox {phase} process failed with exit code {return_code}."
|
||||
+ (f"\nLast output:\n{details}" if details else "")
|
||||
)
|
||||
|
||||
|
||||
def _resolve_model_paths(shared_settings: Dict[str, Any]) -> Dict[str, str]:
|
||||
model_name = str(shared_settings.get("model_name", "DramaBox") or "DramaBox")
|
||||
return DramaBoxDownloader().resolve_model_path(model_name)
|
||||
|
||||
|
||||
def build_preprocess_command(
|
||||
dataset_info: Dict[str, Any],
|
||||
shared_settings: Dict[str, Any],
|
||||
*,
|
||||
batch_size: int = 8,
|
||||
skip_existing: bool = True,
|
||||
) -> list[str]:
|
||||
paths = _resolve_model_paths(shared_settings)
|
||||
command = [
|
||||
sys.executable,
|
||||
str(PREPROCESS_SCRIPT),
|
||||
"--dataset-type",
|
||||
"gemini_synthetic",
|
||||
"--index",
|
||||
str(dataset_info["index_path"]),
|
||||
"--output-dir",
|
||||
str(dataset_info["preprocessed_dir"]),
|
||||
"--checkpoint",
|
||||
paths["audio_components"],
|
||||
"--audio-only-ckpt",
|
||||
paths["audio_components"],
|
||||
"--gemma-root",
|
||||
paths["gemma_root"],
|
||||
"--max-duration",
|
||||
str(float(dataset_info.get("max_duration", 20.0))),
|
||||
"--min-duration",
|
||||
str(float(dataset_info.get("min_duration", 2.0))),
|
||||
"--batch-size",
|
||||
str(max(1, int(batch_size))),
|
||||
]
|
||||
if skip_existing:
|
||||
command.append("--skip-existing")
|
||||
return command
|
||||
|
||||
|
||||
def run_dramabox_preprocess(
|
||||
dataset_info: Dict[str, Any],
|
||||
shared_settings: Dict[str, Any],
|
||||
*,
|
||||
batch_size: int = 8,
|
||||
progress_file: str = "",
|
||||
node_id: str = "",
|
||||
) -> Dict[str, Any]:
|
||||
command = build_preprocess_command(
|
||||
dataset_info,
|
||||
shared_settings,
|
||||
batch_size=batch_size,
|
||||
skip_existing=True,
|
||||
)
|
||||
_run_process(
|
||||
command,
|
||||
cwd=VENDOR_ROOT,
|
||||
env=_device_environment(shared_settings),
|
||||
phase="preprocess",
|
||||
progress_file=progress_file,
|
||||
node_id=node_id,
|
||||
)
|
||||
validate_preprocessed_dataset(
|
||||
dataset_info.get("records") or [],
|
||||
dataset_info["preprocessed_dir"],
|
||||
raise_on_missing=True,
|
||||
)
|
||||
dataset_info["preprocessed"] = True
|
||||
return dataset_info
|
||||
|
||||
|
||||
def _resolve_validation_config(value: str) -> str:
|
||||
raw = os.path.expanduser(str(value or "").strip())
|
||||
if not raw:
|
||||
return ""
|
||||
candidates = [Path(raw)]
|
||||
if not os.path.isabs(raw):
|
||||
candidates.extend(
|
||||
(
|
||||
Path(folder_paths.get_input_directory()) / raw,
|
||||
VENDOR_ROOT / raw,
|
||||
)
|
||||
)
|
||||
for candidate in candidates:
|
||||
if candidate.is_file():
|
||||
return str(candidate.resolve())
|
||||
raise FileNotFoundError(f"DramaBox validation config not found: {value}")
|
||||
|
||||
|
||||
def _validation_gpu(training_device: str, requested_gpu: Any) -> str:
|
||||
value = str(requested_gpu or "").strip()
|
||||
if not value:
|
||||
raise ValueError(
|
||||
"DramaBox validation_config requires validation_gpu because official validation "
|
||||
"runs a second full model process. Reserve a GPU different from the training GPU."
|
||||
)
|
||||
if not value.isdigit():
|
||||
raise ValueError("DramaBox validation_gpu must be a non-negative CUDA device index")
|
||||
device = str(training_device or "auto").strip().lower()
|
||||
training_gpu = device.split(":", 1)[1] if device.startswith("cuda:") else "0"
|
||||
if value == training_gpu:
|
||||
raise ValueError(
|
||||
f"DramaBox validation_gpu ({value}) must differ from the training GPU ({training_gpu})"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def _resolve_continue_lora(continue_from: Any) -> str:
|
||||
if continue_from is None:
|
||||
return ""
|
||||
if isinstance(continue_from, str):
|
||||
value = os.path.abspath(os.path.expanduser(continue_from.strip()))
|
||||
elif isinstance(continue_from, dict):
|
||||
if str(continue_from.get("engine_type", "") or "").strip().lower() not in {"", "dramabox"}:
|
||||
raise ValueError("continue_from TRAINING_ARTIFACTS must come from a DramaBox training run")
|
||||
value = str(
|
||||
continue_from.get("lora_path")
|
||||
or continue_from.get("model_path")
|
||||
or (continue_from.get("lora_adapter") or {}).get("adapter_path", "")
|
||||
).strip()
|
||||
value = os.path.abspath(os.path.expanduser(value)) if value else ""
|
||||
else:
|
||||
raise ValueError("Unsupported DramaBox continue_from input")
|
||||
|
||||
if not value:
|
||||
return ""
|
||||
if os.path.isdir(value):
|
||||
candidates = sorted(Path(value).glob("lora_step_*.safetensors"))
|
||||
candidates += [Path(value) / "adapter_model.safetensors"]
|
||||
for candidate in reversed(candidates):
|
||||
if candidate.is_file():
|
||||
return str(candidate)
|
||||
raise FileNotFoundError(f"No DramaBox LoRA weights found in '{value}'")
|
||||
if not os.path.isfile(value):
|
||||
raise FileNotFoundError(f"DramaBox LoRA checkpoint not found: {value}")
|
||||
return value
|
||||
|
||||
|
||||
def _managed_lora_root() -> Path:
|
||||
try:
|
||||
from utils.models.extra_paths import get_all_tts_model_paths
|
||||
|
||||
for base_path in get_all_tts_model_paths("TTS"):
|
||||
root = Path(base_path) / "dramabox" / "loras"
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
return root
|
||||
except Exception:
|
||||
pass
|
||||
root = Path(folder_paths.models_dir) / "TTS" / "dramabox" / "loras"
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
return root
|
||||
|
||||
|
||||
def _next_managed_lora_dir(name: str, *, overwrite: bool) -> Path:
|
||||
target = _managed_lora_root() / slugify(name)
|
||||
if overwrite or not target.exists():
|
||||
return target
|
||||
counter = 2
|
||||
while True:
|
||||
candidate = target.parent / f"{target.name}_{counter}"
|
||||
if not candidate.exists():
|
||||
return candidate
|
||||
counter += 1
|
||||
|
||||
|
||||
def _latest_lora_file(output_dir: Path) -> Optional[Path]:
|
||||
candidates = sorted(
|
||||
output_dir.glob("lora_step_*.safetensors"),
|
||||
key=lambda path: int(re.search(r"(\d+)", path.stem).group(1))
|
||||
if re.search(r"(\d+)", path.stem)
|
||||
else -1,
|
||||
)
|
||||
if candidates:
|
||||
return candidates[-1]
|
||||
candidate = output_dir / "adapter_model.safetensors"
|
||||
return candidate if candidate.is_file() else None
|
||||
|
||||
|
||||
def _build_train_config(
|
||||
dataset_info: Dict[str, Any],
|
||||
shared_settings: Dict[str, Any],
|
||||
training_config: Dict[str, Any],
|
||||
*,
|
||||
output_dir: Path,
|
||||
continue_lora: str,
|
||||
resolve_paths: bool = True,
|
||||
) -> Dict[str, Any]:
|
||||
if shared_settings.get("model_paths"):
|
||||
paths = dict(shared_settings["model_paths"])
|
||||
elif resolve_paths:
|
||||
paths = _resolve_model_paths(shared_settings)
|
||||
else:
|
||||
paths = {
|
||||
"transformer": "<dramabox-transformer.safetensors>",
|
||||
"audio_components": "<dramabox-audio-components.safetensors>",
|
||||
}
|
||||
config: Dict[str, Any] = {
|
||||
"data_dir": [str(dataset_info["preprocessed_dir"])],
|
||||
"speaker_index": [str(dataset_info["index_path"])],
|
||||
"output_dir": str(output_dir),
|
||||
"checkpoint": paths["transformer"],
|
||||
"full_checkpoint": paths["audio_components"],
|
||||
"base_model": str(training_config.get("base_model", "dev")),
|
||||
"lora_rank": int(training_config.get("lora_rank", 128)),
|
||||
"lora_alpha": int(training_config.get("lora_alpha", 128)),
|
||||
"lora_dropout": float(training_config.get("lora_dropout", 0.1)),
|
||||
"ref_ratio": float(training_config.get("ref_ratio", 0.3)),
|
||||
"max_ref_tokens": int(training_config.get("max_ref_tokens", 200)),
|
||||
"text_dropout": float(training_config.get("text_dropout", 0.4)),
|
||||
"steps": int(training_config.get("steps", 10000)),
|
||||
"lr": float(training_config.get("learning_rate", 1e-4)),
|
||||
"lr_scheduler": str(training_config.get("lr_scheduler", "cosine")),
|
||||
"warmup_steps": int(training_config.get("warmup_steps", 500)),
|
||||
"batch_size": int(training_config.get("batch_size", 1)),
|
||||
"grad_accum": int(training_config.get("grad_accum", 4)),
|
||||
"max_grad_norm": float(training_config.get("max_grad_norm", 1.0)),
|
||||
"save_every": max(1, int(training_config.get("save_every", 500))),
|
||||
"log_every": int(training_config.get("log_every", 10)),
|
||||
"seed": int(training_config.get("seed", 42)),
|
||||
}
|
||||
if continue_lora:
|
||||
config["resume_lora"] = continue_lora
|
||||
validation_config = _resolve_validation_config(
|
||||
training_config.get("validation_config", "")
|
||||
)
|
||||
if validation_config:
|
||||
config["val_config"] = validation_config
|
||||
return config
|
||||
|
||||
|
||||
def _accelerate_command() -> list[str]:
|
||||
executable = shutil.which("accelerate")
|
||||
if executable:
|
||||
return [executable, "launch", "--num_processes", "1"]
|
||||
return [sys.executable, "-m", "accelerate.commands.launch", "--num_processes", "1"]
|
||||
|
||||
|
||||
def run_dramabox_training_job(
|
||||
shared_settings: Dict[str, Any],
|
||||
dataset_info: Dict[str, Any],
|
||||
training_config: Dict[str, Any],
|
||||
*,
|
||||
output_name: str = "",
|
||||
resume: bool = False,
|
||||
overwrite: bool = False,
|
||||
continue_from: Any = None,
|
||||
node_id: str = "",
|
||||
) -> Dict[str, Any]:
|
||||
if str(dataset_info.get("engine_type", "") or "").strip().lower() != "dramabox":
|
||||
raise ValueError("DramaBox training requires a DramaBox TRAINING_DATASET payload")
|
||||
if str(training_config.get("training_mode", "audio_lora") or "").strip().lower() != "audio_lora":
|
||||
raise ValueError("DramaBox training currently supports audio_lora mode only")
|
||||
if resume:
|
||||
raise RuntimeError(
|
||||
"DramaBox does not support exact optimizer-state resume. Use continue_from with a saved LoRA checkpoint for a warm start."
|
||||
)
|
||||
if str(shared_settings.get("device", "auto") or "auto").strip().lower().startswith("cpu") and not bool(
|
||||
training_config.get("dry_run", False)
|
||||
):
|
||||
raise RuntimeError(
|
||||
"DramaBox model training requires CUDA. Use dry_run for CPU-only validation; "
|
||||
"no model weights or CUDA process will be started in that mode."
|
||||
)
|
||||
requested_validation = str(
|
||||
training_config.get("validation_config", "") or ""
|
||||
).strip()
|
||||
if requested_validation:
|
||||
_resolve_validation_config(requested_validation)
|
||||
_validation_gpu(
|
||||
shared_settings.get("device", "auto"),
|
||||
training_config.get("validation_gpu", ""),
|
||||
)
|
||||
|
||||
safe_name = slugify(output_name or dataset_info.get("model_name") or "dramabox_lora")
|
||||
root = Path(get_dramabox_training_root()) / "jobs"
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
fingerprint = f"{safe_name}|{dataset_info.get('index_path')}|{training_config}"
|
||||
job_hash = __import__("hashlib").sha256(fingerprint.encode("utf-8")).hexdigest()[:12]
|
||||
job_dir = root / f"{safe_name}_{job_hash}"
|
||||
if job_dir.exists() and not overwrite:
|
||||
job_dir = root / f"{safe_name}_{job_hash}_{int(time.time())}"
|
||||
if overwrite and job_dir.exists():
|
||||
shutil.rmtree(job_dir)
|
||||
job_dir.mkdir(parents=True, exist_ok=True)
|
||||
train_output_dir = job_dir / "lora"
|
||||
progress_file = str(job_dir / "progress.json")
|
||||
managed_dir = _next_managed_lora_dir(safe_name, overwrite=overwrite)
|
||||
continue_lora = _resolve_continue_lora(continue_from)
|
||||
|
||||
register_training_job(
|
||||
node_id,
|
||||
engine_type="dramabox",
|
||||
progress_file=progress_file,
|
||||
job_dir=str(job_dir),
|
||||
model_name=safe_name,
|
||||
sample_rate="48k",
|
||||
total_epochs=1,
|
||||
)
|
||||
try:
|
||||
_write_progress(
|
||||
progress_file,
|
||||
status="starting",
|
||||
phase="setup",
|
||||
engine_type="dramabox",
|
||||
model_name=safe_name,
|
||||
dataset_records=int(dataset_info.get("train_records", 0)),
|
||||
speakers=dataset_info.get("speakers", []),
|
||||
started_at=time.time(),
|
||||
)
|
||||
|
||||
if not bool(dataset_info.get("preprocessed")):
|
||||
if bool(training_config.get("dry_run", False)):
|
||||
print("🧪 DramaBox dry-run: skipping GPU dataset preprocessing")
|
||||
else:
|
||||
_write_progress(progress_file, status="running", phase="preprocess")
|
||||
run_dramabox_preprocess(
|
||||
dataset_info,
|
||||
shared_settings,
|
||||
batch_size=int(training_config.get("preprocess_batch_size", 8)),
|
||||
progress_file=progress_file,
|
||||
node_id=node_id,
|
||||
)
|
||||
|
||||
train_config = _build_train_config(
|
||||
dataset_info,
|
||||
shared_settings,
|
||||
training_config,
|
||||
output_dir=train_output_dir,
|
||||
continue_lora=continue_lora,
|
||||
resolve_paths=not bool(training_config.get("dry_run", False)),
|
||||
)
|
||||
config_path = job_dir / "training_config.yaml"
|
||||
import yaml
|
||||
|
||||
config_path.write_text(yaml.safe_dump(train_config, sort_keys=False), encoding="utf-8")
|
||||
(job_dir / "resolved_training_config.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"dataset": dataset_info,
|
||||
"shared_settings": shared_settings,
|
||||
"training_config": training_config,
|
||||
"official_config": train_config,
|
||||
"continue_from": continue_lora,
|
||||
},
|
||||
indent=2,
|
||||
ensure_ascii=False,
|
||||
default=str,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
command = [*_accelerate_command(), str(TRAIN_SCRIPT), "--config", str(config_path)]
|
||||
if bool(training_config.get("dry_run", False)):
|
||||
summary = (
|
||||
f"DramaBox dry-run ready: {safe_name} | {dataset_info.get('train_records', 0)} rows | "
|
||||
f"official command prepared without loading CUDA or model weights"
|
||||
)
|
||||
_write_progress(
|
||||
progress_file,
|
||||
status="completed",
|
||||
phase="dry_run",
|
||||
overall_progress=1.0,
|
||||
summary=summary,
|
||||
command=command,
|
||||
)
|
||||
finalize_training_job(node_id, status="completed", summary=summary, dry_run=True)
|
||||
return {
|
||||
"type": "training_artifacts",
|
||||
"engine_type": "dramabox",
|
||||
"training_mode": "audio_lora",
|
||||
"dry_run": True,
|
||||
"job_dir": str(job_dir),
|
||||
"training_config": str(config_path),
|
||||
"summary": summary,
|
||||
"command": command,
|
||||
}
|
||||
|
||||
_write_progress(progress_file, status="running", phase="train", total_steps=int(train_config["steps"]))
|
||||
train_env = _device_environment(shared_settings)
|
||||
if train_config.get("val_config"):
|
||||
paths = _resolve_model_paths(shared_settings)
|
||||
train_env["LTX_CHECKPOINT"] = paths["transformer"]
|
||||
train_env["LTX_FULL_CHECKPOINT"] = paths["audio_components"]
|
||||
train_env["GEMMA_ROOT"] = paths["gemma_root"]
|
||||
train_env["TRAIN_VAL_GPU"] = _validation_gpu(
|
||||
shared_settings.get("device", "auto"),
|
||||
training_config.get("validation_gpu", ""),
|
||||
)
|
||||
_run_process(
|
||||
command,
|
||||
cwd=VENDOR_ROOT,
|
||||
env=train_env,
|
||||
phase="train",
|
||||
progress_file=progress_file,
|
||||
node_id=node_id,
|
||||
total_steps=int(train_config["steps"]),
|
||||
)
|
||||
|
||||
selected_lora = _latest_lora_file(train_output_dir)
|
||||
if selected_lora is None:
|
||||
raise RuntimeError(
|
||||
f"DramaBox training exited successfully but produced no LoRA file in '{train_output_dir}'."
|
||||
)
|
||||
if managed_dir.exists():
|
||||
shutil.rmtree(managed_dir)
|
||||
managed_dir.mkdir(parents=True, exist_ok=True)
|
||||
managed_lora = managed_dir / selected_lora.name
|
||||
shutil.copy2(selected_lora, managed_lora)
|
||||
if selected_lora.name != "adapter_model.safetensors":
|
||||
shutil.copy2(selected_lora, managed_dir / "adapter_model.safetensors")
|
||||
adapter_config = train_output_dir / "adapter_config.json"
|
||||
if adapter_config.is_file():
|
||||
shutil.copy2(adapter_config, managed_dir / adapter_config.name)
|
||||
shutil.copy2(config_path, managed_dir / "training_config.yaml")
|
||||
|
||||
summary = (
|
||||
f"DramaBox audio LoRA training complete: {safe_name} | "
|
||||
f"steps={train_config['steps']} | adapter={managed_lora}"
|
||||
)
|
||||
_write_progress(
|
||||
progress_file,
|
||||
status="completed",
|
||||
phase="done",
|
||||
overall_progress=1.0,
|
||||
output_adapter=str(managed_lora),
|
||||
output_dir=str(managed_dir),
|
||||
summary=summary,
|
||||
)
|
||||
finalize_training_job(
|
||||
node_id,
|
||||
status="completed",
|
||||
output_adapter=str(managed_lora),
|
||||
summary=summary,
|
||||
)
|
||||
return {
|
||||
"type": "training_artifacts",
|
||||
"engine_type": "dramabox",
|
||||
"training_mode": "audio_lora",
|
||||
"model_path": str(managed_dir),
|
||||
"lora_path": str(managed_lora),
|
||||
"job_dir": str(job_dir),
|
||||
"summary": summary,
|
||||
"lora_adapter": {
|
||||
"type": "dramabox_lora",
|
||||
"adapter_path": str(managed_lora),
|
||||
"adapter_dir": str(managed_dir),
|
||||
},
|
||||
}
|
||||
except InterruptedError as error:
|
||||
_write_progress(progress_file, status="cancelled", phase="cancelled", error=str(error))
|
||||
finalize_training_job(node_id, status="cancelled", error=str(error))
|
||||
raise
|
||||
except Exception as error:
|
||||
_write_progress(progress_file, status="error", phase="error", error=str(error))
|
||||
finalize_training_job(node_id, status="error", error=str(error))
|
||||
raise
|
||||
|
||||
|
||||
__all__ = [
|
||||
"build_preprocess_command",
|
||||
"run_dramabox_preprocess",
|
||||
"run_dramabox_training_job",
|
||||
]
|
||||
Vendored
+2
-25
@@ -1,4 +1,4 @@
|
||||
# Bundled DramaBox inference and training source
|
||||
# Bundled DramaBox inference source
|
||||
|
||||
This directory contains the inference-critical source copied unchanged from:
|
||||
|
||||
@@ -15,27 +15,4 @@ The bundled-code changes are marked inline:
|
||||
- `src/inference_server.py`: ComfyUI cancellation exceptions are allowed to
|
||||
propagate from progress callbacks instead of being swallowed; the official
|
||||
negative-prompt, FP8-cast, compile, and staged-memory controls are exposed to
|
||||
the suite wrapper; suite-managed PEFT LoRA loading is added for trained
|
||||
DramaBox audio adapters.
|
||||
- `src/validate.py`: validation accepts the suite's separately organized
|
||||
DramaBox transformer and audio-components checkpoints.
|
||||
- `src/preprocess.py`: suite-distributed pre-quantized Gemma checkpoints use
|
||||
the same bitsandbytes-aware prompt-encoder loader as DramaBox inference.
|
||||
- `src/train.py`: the batch collator lives at module scope so Windows
|
||||
spawn-based DataLoader workers can serialize it; lightweight per-step
|
||||
telemetry keeps the suite's training dashboard current between normal logs.
|
||||
- `ltx2/ltx_core/text_encoders/gemma/encoders/encoder_configurator.py`:
|
||||
supports both wrapped and direct SigLIP vision-tower layouts for the suite's
|
||||
newer Transformers runtime.
|
||||
|
||||
The official training entry points are also bundled at this pin:
|
||||
|
||||
- `src/preprocess.py`
|
||||
- `src/train.py`
|
||||
- `src/validate.py`
|
||||
- `configs/training_args.example.yaml`
|
||||
- `configs/val_config.example.yaml`
|
||||
|
||||
The suite invokes these scripts through the unified training backend. Apart
|
||||
from the documented compatibility patches, training behavior stays upstream;
|
||||
dataset normalization, job lifecycle, and UI wiring remain suite-side.
|
||||
the suite wrapper.
|
||||
|
||||
@@ -1,63 +0,0 @@
|
||||
# DramaBox IC-LoRA training config — values become the defaults for
|
||||
# `accelerate launch src/train.py --config configs/training_args.example.yaml`.
|
||||
# Any flag explicitly passed on the CLI overrides the YAML.
|
||||
|
||||
# ── Data ───────────────────────────────────────────────────────────────────
|
||||
# One entry per preprocessed dataset (output dirs from src/preprocess.py).
|
||||
data_dir:
|
||||
- /path/to/preprocessed_dataset_a/
|
||||
- /path/to/preprocessed_dataset_b/
|
||||
|
||||
# One index file per data_dir entry. Each line follows the format you fed to
|
||||
# preprocess.py — see README "Prepare your index file".
|
||||
speaker_index:
|
||||
- /path/to/preprocessed_dataset_a/index.txt
|
||||
- /path/to/preprocessed_dataset_b/index.txt
|
||||
|
||||
# Output directory for LoRA shards + logs (relative paths resolve against the
|
||||
# repo root).
|
||||
output_dir: tts_iclora_v1
|
||||
|
||||
# ── Base model ─────────────────────────────────────────────────────────────
|
||||
# Train your LoRA on top of DramaBox itself (recommended) — the trimmed audio
|
||||
# components are enough; no need to ship the raw LTX-2.3 base.
|
||||
checkpoint: dramabox-dit-v1.safetensors
|
||||
full_checkpoint: dramabox-audio-components.safetensors
|
||||
base_model: dev # 'dev' = ShiftedLogitNormal sampler; 'distilled' = DistilledTimestepSampler
|
||||
|
||||
# ── LoRA hyperparams (rank == alpha → scale = 1.0) ─────────────────────────
|
||||
lora_rank: 128
|
||||
lora_alpha: 128
|
||||
lora_dropout: 0.1 # ~0.1 helps regularize on small datasets
|
||||
|
||||
# Resume an existing LoRA — step number parsed from the filename
|
||||
# (e.g. lora_step_05000.safetensors → starts at step 5000).
|
||||
# resume_lora: tts_iclora_v0/lora_step_05000.safetensors
|
||||
|
||||
# ── Voice-cloning reference tokens ─────────────────────────────────────────
|
||||
ref_ratio: 0.3 # fraction of training samples that get a ref-token tail
|
||||
max_ref_tokens: 200 # cap on appended ref tokens after patchification
|
||||
|
||||
# CFG training: probability of zeroing the text condition (forces reliance on
|
||||
# the voice ref / unconditional path).
|
||||
text_dropout: 0.4
|
||||
|
||||
# ── Schedule ───────────────────────────────────────────────────────────────
|
||||
# Cosine + 1e-4 = from-scratch fine-tune.
|
||||
# Constant + 1e-5 = polish on top of an existing LoRA (use with `resume_lora`).
|
||||
steps: 10000
|
||||
lr: 1.0e-04
|
||||
lr_scheduler: cosine
|
||||
warmup_steps: 500
|
||||
|
||||
batch_size: 1
|
||||
grad_accum: 4
|
||||
max_grad_norm: 1.0
|
||||
|
||||
save_every: 500
|
||||
log_every: 50
|
||||
seed: 53
|
||||
|
||||
# Optional per-save-step validation pass. Generates a sample for every speaker
|
||||
# in the val_config so you can A/B listen during training.
|
||||
# val_config: configs/val_config.example.yaml
|
||||
@@ -1,25 +0,0 @@
|
||||
# Validation prompts run by src/validate.py at every --save-every checkpoint.
|
||||
# Each entry produces one .wav under <output_dir>/val_step_<N>/<name>.wav.
|
||||
#
|
||||
# Fields:
|
||||
# name — short tag used as the output filename
|
||||
# prompt — full DramaBox-style scene prompt
|
||||
# reference — (optional) absolute path to a 10+ s voice reference clip;
|
||||
# omit for prompt-only generation
|
||||
|
||||
speakers:
|
||||
- name: villain_growl
|
||||
prompt: 'A shadowy villain speaks with cold menace, "You have entered my domain, mortal." He chuckles darkly, "Such arrogance will be your undoing."'
|
||||
reference: /path/to/voice_refs/male_villain.wav
|
||||
|
||||
- name: tender_whisper
|
||||
prompt: 'A woman speaks tenderly, "It has been a long day, my love." She whispers, "Close your eyes. I am right here."'
|
||||
reference: /path/to/voice_refs/female_warm.wav
|
||||
|
||||
- name: catgirl_giggle
|
||||
prompt: 'A playful girl already mid-giggle, "Hehehe, oh my gosh you should see your face!" She gasps, "Oh my, hehe, I cannot stop!"'
|
||||
# No `reference:` here — pure prompt-driven generation.
|
||||
|
||||
- name: announcer_smug
|
||||
prompt: 'A confident announcer speaks proudly, "And now, the moment you have all been waiting for." He chuckles knowingly, "Heheh."'
|
||||
reference: /path/to/voice_refs/male_announcer.wav
|
||||
+5
-31
@@ -154,48 +154,22 @@ VIDEO_ONLY_EMBEDDINGS_PROCESSOR_KEY_OPS = (
|
||||
|
||||
def create_and_populate(module: GemmaTextEncoder) -> GemmaTextEncoder:
|
||||
model = module.model
|
||||
vision_tower = model.model.vision_tower
|
||||
# TTS Audio Suite patch: Transformers 5 exposes SiglipVisionModel
|
||||
# directly, while the upstream-pinned layout wraps it in `.vision_model`.
|
||||
v_model = getattr(vision_tower, "vision_model", vision_tower)
|
||||
v_model = model.model.vision_tower.vision_model
|
||||
l_model = model.model.language_model
|
||||
|
||||
config = model.config.text_config
|
||||
dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
|
||||
if hasattr(config, "rope_local_base_freq"):
|
||||
base = config.rope_local_base_freq
|
||||
rope_type = config.rope_scaling["rope_type"]
|
||||
rope_kwargs = {}
|
||||
else:
|
||||
# TTS Audio Suite patch: Transformers 5 migrates Gemma 3's local and
|
||||
# full-attention RoPE settings into named `rope_parameters` entries.
|
||||
rope_parameters = config.rope_parameters
|
||||
base = rope_parameters["sliding_attention"]["rope_theta"]
|
||||
rope_type = rope_parameters["full_attention"]["rope_type"]
|
||||
rope_kwargs = {"layer_type": "full_attention"}
|
||||
base = config.rope_local_base_freq
|
||||
local_rope_freqs = 1.0 / (base ** (torch.arange(0, dim, 2, dtype=torch.int64).to(dtype=torch.float) / dim))
|
||||
inv_freqs, _ = ROPE_INIT_FUNCTIONS[rope_type](config, **rope_kwargs)
|
||||
inv_freqs, _ = ROPE_INIT_FUNCTIONS[config.rope_scaling["rope_type"]](config)
|
||||
|
||||
positions_length = len(v_model.embeddings.position_ids[0])
|
||||
position_ids = torch.arange(positions_length, dtype=torch.long, device="cpu").unsqueeze(0)
|
||||
v_model.embeddings.register_buffer("position_ids", position_ids)
|
||||
embed_scale = torch.tensor(model.config.text_config.hidden_size**0.5, device="cpu")
|
||||
l_model.embed_tokens.register_buffer("embed_scale", embed_scale)
|
||||
if hasattr(l_model, "rotary_emb_local"):
|
||||
l_model.rotary_emb_local.register_buffer("inv_freq", local_rope_freqs)
|
||||
l_model.rotary_emb.register_buffer("inv_freq", inv_freqs)
|
||||
else:
|
||||
# TTS Audio Suite patch: Transformers 5 consolidates both attention
|
||||
# variants into one rotary module with separately named buffers.
|
||||
rotary_emb = l_model.rotary_emb
|
||||
rotary_emb.register_buffer("sliding_attention_inv_freq", local_rope_freqs)
|
||||
rotary_emb.register_buffer(
|
||||
"sliding_attention_original_inv_freq", local_rope_freqs.clone()
|
||||
)
|
||||
rotary_emb.register_buffer("full_attention_inv_freq", inv_freqs)
|
||||
rotary_emb.register_buffer(
|
||||
"full_attention_original_inv_freq", inv_freqs.clone()
|
||||
)
|
||||
l_model.rotary_emb_local.register_buffer("inv_freq", local_rope_freqs)
|
||||
l_model.rotary_emb.register_buffer("inv_freq", inv_freqs)
|
||||
|
||||
return module
|
||||
|
||||
|
||||
+1
-266
@@ -125,8 +125,7 @@ def auto_rescale_for_cfg(cfg: float) -> float:
|
||||
class TTSServer:
|
||||
def __init__(self, checkpoint=None, full_checkpoint=None, gemma_root=None,
|
||||
device="cuda", dtype="bf16", compile_model=True, bnb_4bit=True,
|
||||
memory_mode="fast", transformer_quantization="none",
|
||||
lora_path="", lora_strength=1.0):
|
||||
memory_mode="fast", transformer_quantization="none"):
|
||||
MODELS = APP_DIR / "models"
|
||||
self.checkpoint = checkpoint or str(MODELS / "ltx-2.3-22b-dev-audio-only-v13-merged.safetensors")
|
||||
self.full_checkpoint = full_checkpoint or os.environ.get(
|
||||
@@ -141,14 +140,6 @@ class TTSServer:
|
||||
self.bnb_4bit = bnb_4bit
|
||||
self.memory_mode = str(memory_mode)
|
||||
self.transformer_quantization = str(transformer_quantization)
|
||||
# TTS Audio Suite patch: accept a trained DramaBox audio LoRA at
|
||||
# runtime so training artifacts can be used without a second CLI.
|
||||
self.lora_path = str(lora_path or "").strip()
|
||||
self.lora_strength = float(lora_strength)
|
||||
self._active_lora_revision = ""
|
||||
self._active_lora_file = ""
|
||||
self._applied_lora_strength = 0.0
|
||||
self._unmerged_lora_weight_scale = 1.0
|
||||
if self.memory_mode not in {"fast", "staged", "sequential"}:
|
||||
raise ValueError(f"Unknown DramaBox memory mode: {self.memory_mode}")
|
||||
if self.transformer_quantization not in {"none", "fp8_cast"}:
|
||||
@@ -271,8 +262,6 @@ class TTSServer:
|
||||
self._velocity_model = builder.build(
|
||||
device=self.device, dtype=build_dtype
|
||||
).to(self.device).eval()
|
||||
if self.lora_path and self.lora_strength != 0.0:
|
||||
self.configure_lora(self.lora_path, self.lora_strength)
|
||||
n_params = sum(p.numel() for p in self._velocity_model.parameters()) / 1e9
|
||||
vram_gb = sum(p.numel() * p.element_size() for p in self._velocity_model.parameters()) / 1e9
|
||||
logging.info(f" Transformer: {time.time()-t0:.1f}s ({n_params:.1f}B params, {vram_gb:.1f}GB VRAM, {self.dtype})")
|
||||
@@ -298,260 +287,6 @@ class TTSServer:
|
||||
)
|
||||
logging.info(f" AudioDecoder (warm): {time.time()-t0:.1f}s")
|
||||
|
||||
@staticmethod
|
||||
def _resolve_lora_file(lora_path: str) -> Path:
|
||||
path = Path(os.path.expanduser(str(lora_path or "").strip()))
|
||||
if path.is_dir():
|
||||
candidates = sorted(path.glob("lora_step_*.safetensors"))
|
||||
candidates += [path / "adapter_model.safetensors"]
|
||||
for candidate in reversed(candidates):
|
||||
if candidate.is_file():
|
||||
return candidate
|
||||
if path.is_file():
|
||||
return path
|
||||
raise FileNotFoundError(f"DramaBox LoRA file not found: {lora_path}")
|
||||
|
||||
# TTS Audio Suite patch: keep PEFT state attached and reversibly merge it
|
||||
# so strength changes avoid both a base reload and per-step LoRA matmuls.
|
||||
@staticmethod
|
||||
def _set_lora_strength(model, strength: float) -> None:
|
||||
"""Re-merge the live adapter at a new strength without reloading the base."""
|
||||
try:
|
||||
from peft.tuners.lora.layer import LoraLayer
|
||||
except ImportError as exc:
|
||||
raise RuntimeError("DramaBox LoRA inference requires peft.") from exc
|
||||
|
||||
updated = 0
|
||||
if any(
|
||||
isinstance(module, LoraLayer) and bool(module.merged)
|
||||
for module in model.modules()
|
||||
):
|
||||
model.unmerge_adapter()
|
||||
|
||||
for module in model.modules():
|
||||
if isinstance(module, LoraLayer) and "default" in module.lora_A:
|
||||
module.set_scale("default", float(strength))
|
||||
updated += 1
|
||||
if updated <= 0:
|
||||
raise RuntimeError("DramaBox LoRA modules are missing from the live model.")
|
||||
if hasattr(model, "set_adapter"):
|
||||
model.set_adapter("default")
|
||||
if float(strength) == 0.0:
|
||||
model.disable_adapter_layers()
|
||||
else:
|
||||
model.enable_adapter_layers()
|
||||
model.merge_adapter(adapter_names=["default"])
|
||||
|
||||
@staticmethod
|
||||
def _set_unmerged_lora_strength(
|
||||
model, strength: float, current_weight_scale: float
|
||||
) -> float:
|
||||
"""Scale a BF16 PEFT branch over an immutable FP8 base in place."""
|
||||
try:
|
||||
from peft.tuners.lora.layer import LoraLayer
|
||||
except ImportError as exc:
|
||||
raise RuntimeError("DramaBox LoRA inference requires peft.") from exc
|
||||
|
||||
if float(strength) == 0.0:
|
||||
model.disable_adapter_layers()
|
||||
return float(current_weight_scale)
|
||||
|
||||
model.enable_adapter_layers()
|
||||
if hasattr(model, "set_adapter"):
|
||||
model.set_adapter("default")
|
||||
ratio = float(strength) / float(current_weight_scale)
|
||||
updated = 0
|
||||
for module in model.modules():
|
||||
if isinstance(module, LoraLayer) and "default" in module.lora_A:
|
||||
if ratio != 1.0:
|
||||
with torch.no_grad():
|
||||
module.lora_B["default"].weight.mul_(ratio)
|
||||
updated += 1
|
||||
if updated <= 0:
|
||||
raise RuntimeError("DramaBox LoRA modules are missing from the live model.")
|
||||
return float(strength)
|
||||
|
||||
def _prepare_unmerged_lora(self, model) -> None:
|
||||
"""Keep PEFT matrices in the activation dtype used above FP8 storage."""
|
||||
from peft.tuners.lora.layer import LoraLayer
|
||||
|
||||
for module in model.modules():
|
||||
if isinstance(module, LoraLayer) and "default" in module.lora_A:
|
||||
module.lora_A["default"].to(device=self.device, dtype=self.dtype)
|
||||
module.lora_B["default"].to(device=self.device, dtype=self.dtype)
|
||||
|
||||
# TTS Audio Suite patch: replace only the live adapter modules while
|
||||
# preserving the already-loaded official DramaBox transformer weights.
|
||||
@classmethod
|
||||
def _attach_lora(cls, model, lora_path: str, strength: float):
|
||||
"""Attach or replace the official PEFT-compatible audio LoRA in place."""
|
||||
try:
|
||||
from peft import LoraConfig, PeftModel, get_peft_model
|
||||
from safetensors.torch import load_file
|
||||
except ImportError as exc:
|
||||
raise RuntimeError(
|
||||
"DramaBox LoRA inference requires peft and safetensors."
|
||||
) from exc
|
||||
|
||||
lora_file = cls._resolve_lora_file(lora_path)
|
||||
adapter_config_path = lora_file.parent / "adapter_config.json"
|
||||
rank = 128
|
||||
alpha = 128
|
||||
if adapter_config_path.is_file():
|
||||
try:
|
||||
metadata = json.loads(adapter_config_path.read_text(encoding="utf-8"))
|
||||
rank = int(metadata.get("r", rank))
|
||||
alpha = int(metadata.get("lora_alpha", alpha))
|
||||
except Exception as exc:
|
||||
logging.warning("Could not read DramaBox LoRA adapter_config.json: %s", exc)
|
||||
|
||||
lora_state = load_file(str(lora_file))
|
||||
# Standalone upstream checkpoints may omit adapter_config.json. Infer
|
||||
# the rank from the first LoRA-A tensor so those files remain usable.
|
||||
if not adapter_config_path.is_file():
|
||||
for key, value in lora_state.items():
|
||||
if ".lora_A." in key or key.endswith(".lora_A.weight"):
|
||||
rank = int(value.shape[0])
|
||||
alpha = rank
|
||||
break
|
||||
lora_config = LoraConfig(
|
||||
r=rank,
|
||||
lora_alpha=alpha,
|
||||
lora_dropout=0.0,
|
||||
bias="none",
|
||||
target_modules=[
|
||||
"audio_attn1.to_k",
|
||||
"audio_attn1.to_q",
|
||||
"audio_attn1.to_v",
|
||||
"audio_attn1.to_out.0",
|
||||
"audio_attn2.to_k",
|
||||
"audio_attn2.to_q",
|
||||
"audio_attn2.to_v",
|
||||
"audio_attn2.to_out.0",
|
||||
"audio_ff.net.0.proj",
|
||||
"audio_ff.net.2",
|
||||
],
|
||||
)
|
||||
if isinstance(model, PeftModel):
|
||||
if any(
|
||||
hasattr(module, "merged") and bool(module.merged)
|
||||
for module in model.modules()
|
||||
):
|
||||
model.unmerge_adapter()
|
||||
if "default" in model.peft_config:
|
||||
model.delete_adapter("default")
|
||||
model.add_adapter("default", lora_config)
|
||||
adapted = model
|
||||
else:
|
||||
adapted = get_peft_model(model, lora_config)
|
||||
mapped = {}
|
||||
is_peft_format = any("base_model.model." in key for key in lora_state)
|
||||
is_original_format = any("diffusion_model." in key for key in lora_state)
|
||||
compiled_blocks = any("._orig_mod." in key for key in adapted.state_dict())
|
||||
for key, value in lora_state.items():
|
||||
if is_peft_format:
|
||||
new_key = key
|
||||
elif is_original_format:
|
||||
new_key = key.replace("diffusion_model.", "base_model.model.")
|
||||
else:
|
||||
continue
|
||||
new_key = new_key.replace(".lora_A.weight", ".lora_A.default.weight")
|
||||
new_key = new_key.replace(".lora_B.weight", ".lora_B.default.weight")
|
||||
if compiled_blocks and "._orig_mod." not in new_key:
|
||||
# TTS Audio Suite patch: torch.compile wraps every official
|
||||
# transformer block in OptimizedModule and inserts `_orig_mod`
|
||||
# into its state-dict path before PEFT attaches the adapter.
|
||||
new_key = re.sub(
|
||||
r"(transformer_blocks\.\d+)\.",
|
||||
r"\1._orig_mod.",
|
||||
new_key,
|
||||
count=1,
|
||||
)
|
||||
mapped[new_key] = value
|
||||
if not mapped:
|
||||
raise RuntimeError(
|
||||
f"DramaBox LoRA '{lora_file}' is not in a recognized PEFT/ID-LoRA format."
|
||||
)
|
||||
|
||||
missing, unexpected = adapted.load_state_dict(mapped, strict=False)
|
||||
loaded = len(mapped) - len(unexpected)
|
||||
if loaded <= 0:
|
||||
raise RuntimeError(
|
||||
f"DramaBox LoRA '{lora_file}' did not match the audio transformer modules."
|
||||
)
|
||||
logging.info(
|
||||
"DramaBox LoRA loaded: %s (%d tensors, strength %.2f)",
|
||||
lora_file,
|
||||
loaded,
|
||||
float(strength),
|
||||
)
|
||||
return adapted.eval(), str(lora_file.resolve())
|
||||
|
||||
# TTS Audio Suite patch: split mutable adapter identity from the expensive
|
||||
# base-model cache identity used by the suite's ComfyUI model wrapper.
|
||||
def configure_lora(self, lora_path: str, strength: float, revision: str = "") -> None:
|
||||
"""Hot-swap a DramaBox adapter or update only its runtime strength."""
|
||||
path = str(lora_path or "").strip()
|
||||
strength = float(strength)
|
||||
revision = str(revision or "")
|
||||
use_unmerged_fp8 = self.transformer_quantization == "fp8_cast"
|
||||
|
||||
if not path:
|
||||
if self._active_lora_file:
|
||||
if use_unmerged_fp8:
|
||||
self._unmerged_lora_weight_scale = self._set_unmerged_lora_strength(
|
||||
self._velocity_model, 0.0, self._unmerged_lora_weight_scale
|
||||
)
|
||||
else:
|
||||
self._set_lora_strength(self._velocity_model, 0.0)
|
||||
logging.info("DramaBox LoRA disabled without reloading the base model")
|
||||
self.lora_path = ""
|
||||
self.lora_strength = strength
|
||||
self._applied_lora_strength = 0.0
|
||||
return
|
||||
|
||||
lora_file = str(self._resolve_lora_file(path).resolve())
|
||||
same_adapter = (
|
||||
lora_file == self._active_lora_file
|
||||
and revision == self._active_lora_revision
|
||||
)
|
||||
if same_adapter:
|
||||
if strength != self._applied_lora_strength:
|
||||
if use_unmerged_fp8:
|
||||
self._unmerged_lora_weight_scale = self._set_unmerged_lora_strength(
|
||||
self._velocity_model,
|
||||
strength,
|
||||
self._unmerged_lora_weight_scale,
|
||||
)
|
||||
else:
|
||||
self._set_lora_strength(self._velocity_model, strength)
|
||||
logging.info(
|
||||
"DramaBox LoRA strength updated in place: %.2f", strength
|
||||
)
|
||||
else:
|
||||
self._velocity_model, lora_file = self._attach_lora(
|
||||
self._velocity_model, path, strength
|
||||
)
|
||||
self._active_lora_file = lora_file
|
||||
self._active_lora_revision = revision
|
||||
if use_unmerged_fp8:
|
||||
self._prepare_unmerged_lora(self._velocity_model)
|
||||
self._unmerged_lora_weight_scale = 1.0
|
||||
self._unmerged_lora_weight_scale = self._set_unmerged_lora_strength(
|
||||
self._velocity_model,
|
||||
strength,
|
||||
self._unmerged_lora_weight_scale,
|
||||
)
|
||||
logging.info(
|
||||
"DramaBox FP8 base: using an unmerged BF16 LoRA branch"
|
||||
)
|
||||
else:
|
||||
self._set_lora_strength(self._velocity_model, strength)
|
||||
|
||||
self.lora_path = path
|
||||
self.lora_strength = strength
|
||||
self._applied_lora_strength = strength
|
||||
|
||||
def _move_velocity_model(self, target: torch.device) -> None:
|
||||
"""Move the persistent DiT between CUDA and RAM for staged inference."""
|
||||
target = torch.device(target)
|
||||
|
||||
-384
@@ -1,384 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Preprocess TTS datasets for LTX-2.3 audio-only LoRA fine-tuning.
|
||||
|
||||
Takes paired (audio, transcript) data and produces the format expected by
|
||||
the LTX trainer:
|
||||
.precomputed/
|
||||
├── latents/sample_N.pt # Dummy video latents (minimal)
|
||||
├── conditions/sample_N.pt # Text embeddings from Gemma
|
||||
└── audio_latents/sample_N.pt # Audio VAE-encoded latents
|
||||
|
||||
Supports multiple dataset formats:
|
||||
- gemini_synthetic: index.txt with ~-separated fields (id~speaker~lang~sr~samples~dur~phonemes~text)
|
||||
- libriheavy: index_ft.txt with ~-separated fields (id~speaker~lang~samples~dur~phonemes~text)
|
||||
- manifest: JSON/JSONL with {"audio_filepath": ..., "text": ...}
|
||||
- tsv: TSV file with audio_path<TAB>text columns
|
||||
|
||||
Usage:
|
||||
python preprocess_tts_data.py \
|
||||
--dataset-type gemini_synthetic \
|
||||
--index /path/to/dataset/index.txt \
|
||||
--audio-dir /path/to/dataset/wavs \
|
||||
--output-dir /path/to/output/tts_training_data \
|
||||
--max-samples 10000 \
|
||||
--max-duration 20.0 \
|
||||
--min-duration 3.0
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
import torchaudio
|
||||
|
||||
REPO_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "ltx2"))
|
||||
# ltx-pipelines on path via ltx2/
|
||||
|
||||
MODEL_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
GEMMA_DIR = os.environ.get("GEMMA_DIR", "gemma-3-12b-it-qat-q4_0-unquantized")
|
||||
|
||||
|
||||
def parse_args():
|
||||
p = argparse.ArgumentParser(description="Preprocess TTS data for LTX-2.3 fine-tuning")
|
||||
p.add_argument("--dataset-type", required=True,
|
||||
choices=["gemini_synthetic", "libriheavy", "manifest", "tsv"],
|
||||
help="Dataset format type")
|
||||
p.add_argument("--index", required=True, help="Path to index/manifest file")
|
||||
p.add_argument("--audio-dir", default=None,
|
||||
help="Base directory for audio files (if paths in index are relative)")
|
||||
p.add_argument("--output-dir", required=True, help="Output directory for preprocessed data")
|
||||
p.add_argument("--checkpoint", default=os.path.join(MODEL_DIR, "ltx-2.3-22b-distilled.safetensors"))
|
||||
p.add_argument("--gemma-root", default=GEMMA_DIR)
|
||||
p.add_argument("--max-samples", type=int, default=0, help="Max samples to process (0=all)")
|
||||
p.add_argument("--max-duration", type=float, default=20.0, help="Max audio duration in seconds")
|
||||
p.add_argument("--min-duration", type=float, default=2.0, help="Min audio duration in seconds")
|
||||
p.add_argument("--batch-size", type=int, default=8, help="Batch size for text encoding")
|
||||
p.add_argument("--skip-existing", action="store_true", help="Skip already processed samples")
|
||||
p.add_argument("--audio-only-ckpt", default=None,
|
||||
help="Audio-only checkpoint for VAE encoding (optional, uses full ckpt if not set)")
|
||||
p.add_argument("--shard", type=int, default=0, help="Shard index (for parallel processing)")
|
||||
p.add_argument("--num-shards", type=int, default=1, help="Total number of shards")
|
||||
p.add_argument("--gpu", type=int, default=None, help="GPU device index to use")
|
||||
return p.parse_args()
|
||||
|
||||
|
||||
def parse_gemini_synthetic(index_path: str, audio_dir: str | None) -> list[dict]:
|
||||
"""Parse gemini_synthetic format: id~speaker~lang~sr~samples~dur~phonemes~text"""
|
||||
samples = []
|
||||
with open(index_path) as f:
|
||||
for line in f:
|
||||
parts = line.strip().split("~")
|
||||
if len(parts) < 7:
|
||||
continue
|
||||
file_id = parts[0]
|
||||
text = parts[-1] # Last field is always the text
|
||||
sr = int(parts[3])
|
||||
n_samples = int(parts[4])
|
||||
duration = n_samples / sr
|
||||
|
||||
# Find audio file
|
||||
if audio_dir:
|
||||
# Try common extensions
|
||||
for ext in [".flac", ".wav", ".mp3"]:
|
||||
audio_path = os.path.join(audio_dir, file_id + ext)
|
||||
if os.path.exists(audio_path):
|
||||
break
|
||||
else:
|
||||
continue
|
||||
else:
|
||||
audio_path = file_id
|
||||
|
||||
samples.append({
|
||||
"id": file_id,
|
||||
"audio_path": audio_path,
|
||||
"text": text,
|
||||
"duration": duration,
|
||||
})
|
||||
return samples
|
||||
|
||||
|
||||
def parse_libriheavy(index_path: str, audio_dir: str | None) -> list[dict]:
|
||||
"""Parse libriheavy format: id~speaker~lang~samples~dur~phonemes~text"""
|
||||
samples = []
|
||||
with open(index_path) as f:
|
||||
for line in f:
|
||||
parts = line.strip().split("~")
|
||||
if len(parts) < 7:
|
||||
continue
|
||||
file_id = parts[0]
|
||||
text = parts[-1]
|
||||
n_samples = int(parts[3])
|
||||
duration = int(parts[4]) / 1000.0 # milliseconds to seconds
|
||||
|
||||
if audio_dir:
|
||||
for ext in [".flac", ".wav", ".mp3"]:
|
||||
audio_path = os.path.join(audio_dir, file_id + ext)
|
||||
if os.path.exists(audio_path):
|
||||
break
|
||||
else:
|
||||
continue
|
||||
else:
|
||||
audio_path = file_id
|
||||
|
||||
samples.append({
|
||||
"id": file_id,
|
||||
"audio_path": audio_path,
|
||||
"text": text,
|
||||
"duration": duration,
|
||||
})
|
||||
return samples
|
||||
|
||||
|
||||
def parse_manifest(index_path: str, audio_dir: str | None) -> list[dict]:
|
||||
"""Parse JSON/JSONL manifest with audio_filepath and text fields."""
|
||||
samples = []
|
||||
with open(index_path) as f:
|
||||
for line in f:
|
||||
entry = json.loads(line.strip())
|
||||
audio_path = entry.get("audio_filepath", entry.get("audio_path", ""))
|
||||
text = entry.get("text", entry.get("transcript", ""))
|
||||
duration = entry.get("duration", 0.0)
|
||||
|
||||
if audio_dir and not os.path.isabs(audio_path):
|
||||
audio_path = os.path.join(audio_dir, audio_path)
|
||||
|
||||
if os.path.exists(audio_path) and text:
|
||||
samples.append({
|
||||
"id": Path(audio_path).stem,
|
||||
"audio_path": audio_path,
|
||||
"text": text,
|
||||
"duration": duration,
|
||||
})
|
||||
return samples
|
||||
|
||||
|
||||
def parse_tsv(index_path: str, audio_dir: str | None) -> list[dict]:
|
||||
"""Parse TSV file with audio_path<TAB>text."""
|
||||
samples = []
|
||||
with open(index_path) as f:
|
||||
for line in f:
|
||||
parts = line.strip().split("\t")
|
||||
if len(parts) < 2:
|
||||
continue
|
||||
audio_path, text = parts[0], parts[1]
|
||||
if audio_dir and not os.path.isabs(audio_path):
|
||||
audio_path = os.path.join(audio_dir, audio_path)
|
||||
if os.path.exists(audio_path):
|
||||
samples.append({
|
||||
"id": Path(audio_path).stem,
|
||||
"audio_path": audio_path,
|
||||
"text": text,
|
||||
"duration": 0.0,
|
||||
})
|
||||
return samples
|
||||
|
||||
|
||||
PARSERS = {
|
||||
"gemini_synthetic": parse_gemini_synthetic,
|
||||
"libriheavy": parse_libriheavy,
|
||||
"manifest": parse_manifest,
|
||||
"tsv": parse_tsv,
|
||||
}
|
||||
|
||||
|
||||
@torch.inference_mode()
|
||||
def main():
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
args = parse_args()
|
||||
|
||||
from ltx_core.model.audio_vae import encode_audio as vae_encode_audio
|
||||
from ltx_core.types import Audio
|
||||
from ltx_pipelines.utils.blocks import AudioConditioner, PromptEncoder
|
||||
from ltx_pipelines.utils.media_io import decode_audio_from_file
|
||||
from ltx_trainer.model_loader import load_text_encoder, load_embeddings_processor
|
||||
|
||||
if args.gpu is not None:
|
||||
os.environ["CUDA_VISIBLE_DEVICES"] = str(args.gpu)
|
||||
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
||||
dtype = torch.bfloat16
|
||||
|
||||
# Create output directories
|
||||
out = Path(args.output_dir)
|
||||
(out / "latents").mkdir(parents=True, exist_ok=True)
|
||||
(out / "conditions").mkdir(parents=True, exist_ok=True)
|
||||
(out / "audio_latents").mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Parse dataset
|
||||
logging.info(f"Parsing {args.dataset_type} dataset from {args.index}...")
|
||||
samples = PARSERS[args.dataset_type](args.index, args.audio_dir)
|
||||
logging.info(f"Found {len(samples)} samples")
|
||||
|
||||
# Filter by duration
|
||||
before = len(samples)
|
||||
samples = [s for s in samples if args.min_duration <= s["duration"] <= args.max_duration]
|
||||
logging.info(f"After duration filter [{args.min_duration}s, {args.max_duration}s]: {len(samples)} (dropped {before - len(samples)})")
|
||||
|
||||
if args.max_samples > 0:
|
||||
samples = samples[:args.max_samples]
|
||||
logging.info(f"Limiting to {len(samples)} samples")
|
||||
|
||||
# Assign global indices before sharding
|
||||
for i, s in enumerate(samples):
|
||||
s["global_idx"] = i
|
||||
|
||||
# Shard the data for parallel processing
|
||||
if args.num_shards > 1:
|
||||
total = len(samples)
|
||||
samples = samples[args.shard::args.num_shards]
|
||||
logging.info(f"Shard {args.shard}/{args.num_shards}: {len(samples)} samples (of {total} total)")
|
||||
|
||||
# ── Step 1: Encode text with Gemma (Blocks 1+2 only) ──
|
||||
# The trainer runs Block 3 (embeddings processor/connectors) during training,
|
||||
# so we only precompute Blocks 1+2 here (Gemma LLM + feature extractor).
|
||||
logging.info("Loading text encoder (Gemma + feature extractor)...")
|
||||
gemma_config_path = Path(args.gemma_root) / "config.json"
|
||||
gemma_config = {}
|
||||
if gemma_config_path.is_file():
|
||||
try:
|
||||
gemma_config = json.loads(gemma_config_path.read_text(encoding="utf-8"))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
prompt_encoder = None
|
||||
if "quantization_config" in gemma_config:
|
||||
# TTS Audio Suite patch: the suite distributes a pre-quantized BNB
|
||||
# Gemma checkpoint. The official tensor builder treats its packed
|
||||
# weights as dense matrices, producing thousands of shape mismatches.
|
||||
# Reuse the inference loader that already understands this format.
|
||||
logging.info("Loading pre-quantized Gemma through the BNB prompt encoder...")
|
||||
prompt_encoder = PromptEncoder(
|
||||
checkpoint_path=args.checkpoint,
|
||||
gemma_root=args.gemma_root,
|
||||
dtype=dtype,
|
||||
device=device,
|
||||
warm=True,
|
||||
use_bnb_4bit=True,
|
||||
audio_only=True,
|
||||
)
|
||||
text_encoder = prompt_encoder._warm_text_encoder
|
||||
embeddings_processor = prompt_encoder._warm_embeddings_processor
|
||||
text_encoder.feature_extractor = embeddings_processor.feature_extractor
|
||||
prompt_encoder._warm_text_encoder = None
|
||||
prompt_encoder._warm_embeddings_processor = None
|
||||
else:
|
||||
text_encoder = load_text_encoder(args.gemma_root, device=device, dtype=dtype)
|
||||
|
||||
# Load feature extractor on CPU first to save GPU memory, then move to device
|
||||
logging.info("Loading feature extractor (on CPU first to save GPU memory)...")
|
||||
emb_proc = load_embeddings_processor(args.checkpoint, device="cpu", dtype=dtype)
|
||||
text_encoder.feature_extractor = emb_proc.feature_extractor.to(device)
|
||||
del emb_proc
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
logging.info("Encoding text prompts (Blocks 1+2: Gemma + feature extractor)...")
|
||||
for i, sample in enumerate(samples):
|
||||
gidx = sample["global_idx"]
|
||||
cond_path = out / "conditions" / f"sample_{gidx:06d}.pt"
|
||||
if args.skip_existing and cond_path.exists():
|
||||
continue
|
||||
|
||||
text = sample["text"]
|
||||
# Run Blocks 1+2: Gemma LLM → feature extractor
|
||||
hidden_states, attention_mask = text_encoder.encode(text)
|
||||
video_feats, audio_feats = text_encoder.feature_extractor(
|
||||
hidden_states, attention_mask, "left"
|
||||
)
|
||||
|
||||
torch.save({
|
||||
"video_prompt_embeds": video_feats.squeeze(0).cpu(),
|
||||
"audio_prompt_embeds": audio_feats.squeeze(0).cpu() if audio_feats is not None else video_feats.squeeze(0).cpu(),
|
||||
"prompt_attention_mask": attention_mask.squeeze(0).bool().cpu(),
|
||||
}, cond_path)
|
||||
|
||||
if i % 100 == 0:
|
||||
logging.info(f" Text encoding: {i}/{len(samples)}")
|
||||
|
||||
del text_encoder
|
||||
if prompt_encoder is not None:
|
||||
del embeddings_processor
|
||||
del prompt_encoder
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
# ── Step 2: Encode audio with Audio VAE ──
|
||||
ckpt_for_vae = args.audio_only_ckpt or args.checkpoint
|
||||
logging.info(f"Loading audio VAE from {ckpt_for_vae}...")
|
||||
|
||||
ac = AudioConditioner(checkpoint_path=ckpt_for_vae, dtype=dtype, device=device)
|
||||
|
||||
logging.info("Encoding audio samples...")
|
||||
for idx, sample in enumerate(samples):
|
||||
gidx = sample["global_idx"]
|
||||
audio_path = out / "audio_latents" / f"sample_{gidx:06d}.pt"
|
||||
if args.skip_existing and audio_path.exists():
|
||||
continue
|
||||
|
||||
try:
|
||||
# Load audio
|
||||
voice = decode_audio_from_file(sample["audio_path"], device, 0.0, args.max_duration)
|
||||
if voice is None:
|
||||
logging.warning(f" Skipping {sample['id']}: no audio")
|
||||
continue
|
||||
|
||||
w = voice.waveform
|
||||
if w.dim() == 2:
|
||||
if w.shape[0] == 1:
|
||||
w = w.repeat(2, 1)
|
||||
w = w.unsqueeze(0)
|
||||
elif w.dim() == 3 and w.shape[1] == 1:
|
||||
w = w.repeat(1, 2, 1)
|
||||
voice = Audio(waveform=w, sampling_rate=voice.sampling_rate)
|
||||
|
||||
# Encode through Audio VAE
|
||||
audio_latent = ac(lambda enc: vae_encode_audio(voice, enc, None))
|
||||
|
||||
# Save audio latent
|
||||
torch.save({
|
||||
"latents": audio_latent.squeeze(0).cpu(), # [C=8, T, F=16]
|
||||
"sample_rate": 16000,
|
||||
}, audio_path)
|
||||
|
||||
except Exception as e:
|
||||
logging.warning(f" Skipping {sample['id']}: {e}")
|
||||
continue
|
||||
|
||||
if idx % 100 == 0:
|
||||
logging.info(f" Audio encoding: {idx}/{len(samples)}")
|
||||
|
||||
del ac
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
# ── Step 3: Create dummy video latents ──
|
||||
logging.info("Creating dummy video latents...")
|
||||
# Minimal video: 1 frame, 64x64 = 2x2 in latent space
|
||||
dummy_video = {
|
||||
"latents": torch.zeros(128, 1, 2, 2),
|
||||
"num_frames": 1,
|
||||
"height": 2,
|
||||
"width": 2,
|
||||
"fps": 24.0,
|
||||
}
|
||||
for idx, sample in enumerate(samples):
|
||||
gidx = sample["global_idx"]
|
||||
latent_path = out / "latents" / f"sample_{gidx:06d}.pt"
|
||||
if args.skip_existing and latent_path.exists():
|
||||
continue
|
||||
torch.save(dummy_video, latent_path)
|
||||
|
||||
# ── Summary ──
|
||||
n_audio = len(list((out / "audio_latents").glob("*.pt")))
|
||||
n_cond = len(list((out / "conditions").glob("*.pt")))
|
||||
n_lat = len(list((out / "latents").glob("*.pt")))
|
||||
logging.info(f"\nDone! Output: {args.output_dir}")
|
||||
logging.info(f" audio_latents: {n_audio} files")
|
||||
logging.info(f" conditions: {n_cond} files")
|
||||
logging.info(f" latents: {n_lat} files")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Vendored
-902
@@ -1,902 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Audio-Only IC-LoRA Training for Voice Cloning on LTX-2.3.
|
||||
|
||||
Uses the IC-LoRA pattern: reference audio tokens are APPENDED to the end of
|
||||
the target sequence using AudioConditionByReferenceLatent. Loss is computed
|
||||
only on target tokens; reference tokens remain clean (denoise_mask=0).
|
||||
|
||||
This follows the official video-to-video IC-LoRA strategy closely, but adapted
|
||||
for the audio-only modality path.
|
||||
|
||||
Usage (single GPU):
|
||||
CUDA_VISIBLE_DEVICES=0 python train_audio_iclora.py --data-dir ... --speaker-index ...
|
||||
|
||||
Usage (multi-GPU with accelerate):
|
||||
CUDA_VISIBLE_DEVICES=4,5,6,7 accelerate launch --num_processes=4 train_audio_iclora.py ...
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import random
|
||||
import shutil
|
||||
import sys
|
||||
import time
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
from torch.utils.data import DataLoader, Dataset
|
||||
|
||||
REPO_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "ltx2"))
|
||||
# ltx-pipelines already on path via ltx2/
|
||||
|
||||
MODEL_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
# Import audio conditioning item from our module
|
||||
sys.path.insert(0, MODEL_DIR)
|
||||
from audio_conditioning import AudioConditionByReferenceLatent
|
||||
|
||||
|
||||
# ─── Timestep Sampling ───
|
||||
|
||||
class DistilledTimestepSampler:
|
||||
"""Sample timesteps from the distilled sigma schedule.
|
||||
|
||||
The distilled model was trained to denoise at these specific sigma values.
|
||||
We sample uniformly from the intervals between consecutive sigmas,
|
||||
matching the distribution the model actually operates on.
|
||||
"""
|
||||
|
||||
# Distilled 8-step sigma values (boundaries of denoising intervals)
|
||||
SIGMAS = [1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875, 0.0]
|
||||
|
||||
def __init__(self, jitter: float = 0.02):
|
||||
self.jitter = jitter
|
||||
|
||||
def sample(self, batch_size: int, seq_length: int = None, device: torch.device = None) -> torch.Tensor:
|
||||
n_intervals = len(self.SIGMAS) - 1
|
||||
interval_idx = torch.randint(0, n_intervals, (batch_size,), device=device)
|
||||
t = torch.rand(batch_size, device=device)
|
||||
sigma_high = torch.tensor([self.SIGMAS[i] for i in interval_idx], device=device)
|
||||
sigma_low = torch.tensor([self.SIGMAS[i + 1] for i in interval_idx], device=device)
|
||||
sigma = sigma_low + t * (sigma_high - sigma_low)
|
||||
return sigma.clamp(0.01, 0.99)
|
||||
|
||||
|
||||
class ShiftedLogitNormalTimestepSampler:
|
||||
"""Shifted logit-normal distribution, shift depends on sequence length."""
|
||||
|
||||
def __init__(self, std: float = 1.0, eps: float = 1e-3, uniform_prob: float = 0.1):
|
||||
self.std = std
|
||||
self.eps = eps
|
||||
self.uniform_prob = uniform_prob
|
||||
self.normal_999_percentile = 3.0902 * std
|
||||
self.normal_005_percentile = -2.5758 * std
|
||||
|
||||
def sample(self, batch_size: int, seq_length: int, device: torch.device = None) -> torch.Tensor:
|
||||
mu = self._get_shift(seq_length)
|
||||
normal = torch.randn(batch_size, device=device) * self.std + mu
|
||||
logitnormal = torch.sigmoid(normal)
|
||||
|
||||
p999 = torch.sigmoid(torch.tensor(mu + self.normal_999_percentile, device=device))
|
||||
p005 = torch.sigmoid(torch.tensor(mu + self.normal_005_percentile, device=device))
|
||||
stretched = (logitnormal - p005) / (p999 - p005)
|
||||
stretched = torch.where(stretched >= self.eps, stretched, 2 * self.eps - stretched)
|
||||
stretched = stretched.clamp(0, 1)
|
||||
|
||||
uniform = (1 - self.eps) * torch.rand(batch_size, device=device) + self.eps
|
||||
prob = torch.rand(batch_size, device=device)
|
||||
return torch.where(prob > self.uniform_prob, stretched, uniform)
|
||||
|
||||
@staticmethod
|
||||
def _get_shift(seq_length, min_tok=1024, max_tok=4096, min_s=0.95, max_s=2.05):
|
||||
m = (max_s - min_s) / (max_tok - min_tok)
|
||||
return m * seq_length + (min_s - m * min_tok)
|
||||
|
||||
|
||||
# ─── Dataset ───
|
||||
|
||||
def build_speaker_map(index_paths, data_dirs):
|
||||
"""Map speaker → [(data_dir, sample_idx)] from index file(s).
|
||||
|
||||
The sample index comes from field 0 of the `~`-delimited row when it
|
||||
parses as int (allows subset indexes that keep original sample numbers),
|
||||
otherwise we fall back to the row's line number (legacy behaviour for
|
||||
string-keyed indexes like tts_training_data_podcast).
|
||||
"""
|
||||
speaker_to_samples = defaultdict(list)
|
||||
for index_path, data_dir in zip(index_paths, data_dirs):
|
||||
with open(index_path) as f:
|
||||
for line_num, line in enumerate(f):
|
||||
parts = line.strip().split("~")
|
||||
if len(parts) < 7:
|
||||
continue
|
||||
try:
|
||||
idx = int(parts[0])
|
||||
except ValueError:
|
||||
idx = line_num
|
||||
speaker_id = parts[1]
|
||||
speaker_to_samples[speaker_id].append((data_dir, idx))
|
||||
return {k: v for k, v in speaker_to_samples.items() if len(v) >= 2}
|
||||
|
||||
|
||||
class IDLoRADataset(Dataset):
|
||||
# Silence-latent reference loaded once, used to detect and strip any
|
||||
# leading silence frames baked into the preprocessed audio_latents. The
|
||||
# training loop ALREADY prepends 0-25 random silence frames, so we don't
|
||||
# want accidental silence in the source data compounding on top.
|
||||
_silence_ref = None
|
||||
|
||||
@classmethod
|
||||
def _load_silence_ref(cls):
|
||||
if cls._silence_ref is None:
|
||||
p = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
|
||||
"assets", "silence_latent_frame.pt")
|
||||
if os.path.exists(p):
|
||||
cls._silence_ref = torch.load(p, weights_only=True).float().squeeze() # [C, F]
|
||||
return cls._silence_ref
|
||||
|
||||
def __init__(self, speaker_map):
|
||||
self.samples = []
|
||||
self.speaker_map = {}
|
||||
for speaker, entries in speaker_map.items():
|
||||
valid = []
|
||||
for data_dir, idx in entries:
|
||||
audio_path = Path(data_dir) / "audio_latents" / f"sample_{idx:06d}.pt"
|
||||
cond_path = Path(data_dir) / "conditions" / f"sample_{idx:06d}.pt"
|
||||
if audio_path.exists() and cond_path.exists():
|
||||
valid.append((data_dir, idx))
|
||||
if len(valid) >= 2:
|
||||
self.speaker_map[speaker] = valid
|
||||
for speaker, entries in self.speaker_map.items():
|
||||
for entry in entries:
|
||||
self.samples.append((entry, speaker))
|
||||
IDLoRADataset._load_silence_ref()
|
||||
|
||||
def __len__(self):
|
||||
return len(self.samples)
|
||||
|
||||
def _load_sample(self, data_dir, idx):
|
||||
base = Path(data_dir)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
audio = torch.load(base / "audio_latents" / f"sample_{idx:06d}.pt", weights_only=True)
|
||||
# Prefer prefix-stripped text embeddings if they exist (re-encoded with
|
||||
# just the quoted dialogue, dropping the "A woman says, " / "A man
|
||||
# speaks with X accent, " scene-description prefix).
|
||||
stripped = base / "conditions_stripped" / f"sample_{idx:06d}.pt"
|
||||
cond_path = stripped if stripped.exists() else base / "conditions" / f"sample_{idx:06d}.pt"
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
cond = torch.load(cond_path, weights_only=True)
|
||||
if isinstance(audio, dict):
|
||||
audio = audio.get("audio_latent", audio.get("latent", list(audio.values())[0]))
|
||||
if audio.dim() == 2:
|
||||
audio = audio.unsqueeze(0)
|
||||
audio_feats = cond.get("audio_prompt_embeds", cond.get("prompt_embeds"))
|
||||
attn_mask = cond.get("prompt_attention_mask")
|
||||
# The audio_connector has num_learnable_registers=128 and asserts the
|
||||
# input sequence length is divisible by 128. Our new preprocessing
|
||||
# saved trimmed conditions (dropping left-padding to save disk), which
|
||||
# produces short/irregular sequence lengths. Left-pad back to the next
|
||||
# multiple of 128 with zeros (matching the tokenizer's left-padding
|
||||
# convention) so this assertion holds.
|
||||
REG = 128
|
||||
L = audio_feats.shape[0]
|
||||
target_L = ((L + REG - 1) // REG) * REG
|
||||
if target_L != L:
|
||||
pad_len = target_L - L
|
||||
pad_emb = torch.zeros(pad_len, audio_feats.shape[1],
|
||||
dtype=audio_feats.dtype)
|
||||
pad_mask = torch.zeros(pad_len, dtype=attn_mask.dtype)
|
||||
audio_feats = torch.cat([pad_emb, audio_feats], dim=0)
|
||||
attn_mask = torch.cat([pad_mask, attn_mask], dim=0)
|
||||
return audio, audio_feats, attn_mask
|
||||
|
||||
def __getitem__(self, idx):
|
||||
(data_dir, tgt_idx), speaker = self.samples[idx]
|
||||
tgt_latent, audio_feats, attn_mask = self._load_sample(data_dir, tgt_idx)
|
||||
|
||||
# Drop the reference entirely for non-voice-cloning categories:
|
||||
# - SFX samples (speaker starts with "sfx_"): descriptive sound events,
|
||||
# no speaker identity to clone.
|
||||
# - Song/music samples (suno dataset): prompts describe the music style,
|
||||
# reference audio doesn't transfer anything useful.
|
||||
# Return a zero-length ref so the model trains target-only for these.
|
||||
drop_ref = speaker.startswith("sfx_") or "preprocessed_ltx_suno" in str(data_dir)
|
||||
if drop_ref:
|
||||
C, F_dim = tgt_latent.shape[0], tgt_latent.shape[2]
|
||||
ref_latent = torch.zeros(C, 0, F_dim, dtype=tgt_latent.dtype)
|
||||
else:
|
||||
entries = self.speaker_map[speaker]
|
||||
ref_entry = random.choice([e for e in entries if e[1] != tgt_idx])
|
||||
ref_latent, _, _ = self._load_sample(*ref_entry)
|
||||
|
||||
return {
|
||||
"tgt_latent": tgt_latent,
|
||||
"ref_latent": ref_latent,
|
||||
"audio_features": audio_feats,
|
||||
"attention_mask": attn_mask,
|
||||
}
|
||||
|
||||
|
||||
# ─── Model building ───
|
||||
|
||||
def build_audio_only_model(checkpoint_path, device, dtype):
|
||||
from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder
|
||||
from ltx_core.loader.registry import DummyRegistry
|
||||
from ltx_core.loader.sd_ops import SDOps
|
||||
from ltx_core.model.transformer.model import LTXModel, LTXModelType
|
||||
from ltx_core.model.model_protocol import ModelConfigurator
|
||||
from ltx_core.model.transformer.attention import AttentionFunction
|
||||
from ltx_core.model.transformer.rope import LTXRopeType
|
||||
|
||||
sd_ops = SDOps("AO").with_matching(prefix="model.diffusion_model.").with_replacement("model.diffusion_model.", "")
|
||||
|
||||
class Cfg(ModelConfigurator[LTXModel]):
|
||||
@classmethod
|
||||
def from_config(cls, config):
|
||||
t = config.get("transformer", {})
|
||||
cp = None
|
||||
if not t.get("caption_proj_before_connector", False):
|
||||
from ltx_core.model.transformer.text_projection import create_caption_projection
|
||||
with torch.device("meta"):
|
||||
cp = create_caption_projection(t, audio=True)
|
||||
return LTXModel(
|
||||
model_type=LTXModelType.AudioOnly,
|
||||
audio_num_attention_heads=t.get("audio_num_attention_heads", 32),
|
||||
audio_attention_head_dim=t.get("audio_attention_head_dim", 64),
|
||||
audio_in_channels=t.get("audio_in_channels", 128),
|
||||
audio_out_channels=t.get("audio_out_channels", 128),
|
||||
num_layers=t.get("num_layers", 48),
|
||||
audio_cross_attention_dim=t.get("audio_cross_attention_dim", 2048),
|
||||
norm_eps=t.get("norm_eps", 1e-6),
|
||||
attention_type=AttentionFunction(t.get("attention_type", "default")),
|
||||
positional_embedding_theta=t.get("positional_embedding_theta", 10000.0),
|
||||
audio_positional_embedding_max_pos=t.get("audio_positional_embedding_max_pos", [20]),
|
||||
timestep_scale_multiplier=t.get("timestep_scale_multiplier", 1000),
|
||||
use_middle_indices_grid=t.get("use_middle_indices_grid", True),
|
||||
rope_type=LTXRopeType(t.get("rope_type", "interleaved")),
|
||||
double_precision_rope=t.get("frequencies_precision", False) == "float64",
|
||||
apply_gated_attention=t.get("apply_gated_attention", False),
|
||||
audio_caption_projection=cp,
|
||||
cross_attention_adaln=t.get("cross_attention_adaln", False),
|
||||
)
|
||||
|
||||
builder = Builder(model_path=checkpoint_path, model_class_configurator=Cfg,
|
||||
model_sd_ops=sd_ops, registry=DummyRegistry())
|
||||
return builder.build(device=device, dtype=dtype)
|
||||
|
||||
|
||||
def load_audio_connector(checkpoint_path, device, dtype):
|
||||
# ltx-trainer already on path via ltx2/
|
||||
from ltx_trainer.model_loader import load_embeddings_processor
|
||||
emb_proc = load_embeddings_processor(checkpoint_path, device=device, dtype=dtype)
|
||||
connector = emb_proc.audio_connector
|
||||
del emb_proc
|
||||
return connector
|
||||
|
||||
|
||||
def apply_lora(model, rank, alpha, dropout=0.0):
|
||||
from peft import LoraConfig, get_peft_model
|
||||
config = LoraConfig(
|
||||
r=rank, lora_alpha=alpha, lora_dropout=dropout, bias="none",
|
||||
target_modules=[
|
||||
# Self-attention over audio tokens (voice-transfer pathway via ref).
|
||||
"audio_attn1.to_k", "audio_attn1.to_q", "audio_attn1.to_v", "audio_attn1.to_out.0",
|
||||
# Cross-attention (audio ↔ text context) NOT adapted — keep base
|
||||
# model's prompt→audio behaviour intact and rely on dataset balance
|
||||
# to drive expressiveness. (v15c tried this with adaLN unfreeze,
|
||||
# that proved too destructive; v16 tries it adaLN-frozen.)
|
||||
# FFN — non-linear capacity for style/phonetic adaptation.
|
||||
"audio_ff.net.0.proj", "audio_ff.net.2",
|
||||
],
|
||||
)
|
||||
model = get_peft_model(model, config)
|
||||
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
|
||||
total = sum(p.numel() for p in model.parameters())
|
||||
logging.info(f"LoRA: {trainable:,} trainable / {total:,} total ({100*trainable/total:.1f}%)")
|
||||
return model
|
||||
|
||||
|
||||
@torch.no_grad()
|
||||
def prepare_audio_context(audio_connector, audio_features, attention_mask, device, dtype):
|
||||
from ltx_core.text_encoders.gemma.embeddings_processor import convert_to_additive_mask
|
||||
audio_features = audio_features.to(device=device, dtype=dtype)
|
||||
attention_mask = attention_mask.to(device=device)
|
||||
if audio_features.shape[0] > 1:
|
||||
results = []
|
||||
for i in range(audio_features.shape[0]):
|
||||
feat_i = audio_features[i:i+1]
|
||||
mask_i = attention_mask[i:i+1]
|
||||
additive = convert_to_additive_mask(mask_i, feat_i.dtype)
|
||||
enc_i, _ = audio_connector(feat_i, additive)
|
||||
results.append(enc_i)
|
||||
return torch.cat(results, dim=0)
|
||||
additive_mask = convert_to_additive_mask(attention_mask, audio_features.dtype)
|
||||
audio_encoded, _ = audio_connector(audio_features, additive_mask)
|
||||
return audio_encoded
|
||||
|
||||
|
||||
# ─── Validation ───
|
||||
|
||||
def _unwrap_model_safe(model):
|
||||
"""Strip DDP / peft wrappers without going through accelerate.unwrap_model,
|
||||
which imports deepspeed — broken in our env (torch API drift)."""
|
||||
while hasattr(model, "module"):
|
||||
model = model.module
|
||||
return model
|
||||
|
||||
|
||||
def run_validation(lora_path, val_config_path, output_dir, step, lora_rank=128):
|
||||
"""Call validate.py in a subprocess. It loads TTSServer (the same stack
|
||||
the warm server / Gradio app uses), attaches our LoRA, then iterates every
|
||||
entry in val_config with the same inference settings the user tests with.
|
||||
Single subprocess amortises the model-load cost across all val entries.
|
||||
|
||||
Forces validation onto VAL_GPU (default "0") because training already
|
||||
occupies the rest. Override via TRAIN_VAL_GPU env var.
|
||||
"""
|
||||
import subprocess
|
||||
val_dir = os.path.join(output_dir, "validation", f"step_{step:05d}")
|
||||
os.makedirs(val_dir, exist_ok=True)
|
||||
script = os.path.join(os.path.dirname(__file__), "validate.py")
|
||||
cmd = [
|
||||
sys.executable, script,
|
||||
"--val-config", val_config_path,
|
||||
"--output-dir", val_dir,
|
||||
"--lora", lora_path,
|
||||
"--lora-rank", str(lora_rank),
|
||||
# Use raw estimator output (no +10% buffer) so we can hear
|
||||
# whether the model needs more/less duration at current quality.
|
||||
"--duration-multiplier", "1.0",
|
||||
]
|
||||
log_path = os.path.join(val_dir, "validate.log")
|
||||
env = os.environ.copy()
|
||||
# Validation needs its OWN GPU (training fills the others).
|
||||
env["CUDA_VISIBLE_DEVICES"] = os.environ.get("TRAIN_VAL_GPU", "0")
|
||||
try:
|
||||
with open(log_path, "w") as logf:
|
||||
result = subprocess.run(
|
||||
cmd, stdout=logf, stderr=subprocess.STDOUT, timeout=1800, env=env,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
logging.info(f" Validation step {step}: OK → {val_dir}")
|
||||
else:
|
||||
logging.warning(f" Validation step {step} FAILED (see {log_path})")
|
||||
except subprocess.TimeoutExpired:
|
||||
logging.warning(f" Validation step {step} TIMEOUT (>30min)")
|
||||
|
||||
|
||||
# ─── Args ───
|
||||
|
||||
def collate_audio_batch(batch):
|
||||
"""Pad variable-length audio and track real lengths for loss masking."""
|
||||
# TTS Audio Suite patch: keep this callable at module scope so Windows
|
||||
# spawn-based DataLoader workers can pickle it.
|
||||
max_tgt_T = max(item["tgt_latent"].shape[1] for item in batch)
|
||||
max_ref_T = max(item["ref_latent"].shape[1] for item in batch)
|
||||
channels = batch[0]["tgt_latent"].shape[0]
|
||||
feature_dim = batch[0]["tgt_latent"].shape[2]
|
||||
|
||||
tgt_list, ref_list, feat_list, mask_list = [], [], [], []
|
||||
tgt_lengths, ref_lengths = [], []
|
||||
for item in batch:
|
||||
tgt = item["tgt_latent"]
|
||||
ref = item["ref_latent"]
|
||||
tgt_lengths.append(tgt.shape[1])
|
||||
ref_lengths.append(ref.shape[1])
|
||||
|
||||
if tgt.shape[1] < max_tgt_T:
|
||||
pad = torch.zeros(
|
||||
channels,
|
||||
max_tgt_T - tgt.shape[1],
|
||||
feature_dim,
|
||||
dtype=tgt.dtype,
|
||||
)
|
||||
tgt = torch.cat([tgt, pad], dim=1)
|
||||
tgt_list.append(tgt)
|
||||
|
||||
if ref.shape[1] < max_ref_T:
|
||||
pad = torch.zeros(
|
||||
channels,
|
||||
max_ref_T - ref.shape[1],
|
||||
feature_dim,
|
||||
dtype=ref.dtype,
|
||||
)
|
||||
ref = torch.cat([ref, pad], dim=1)
|
||||
ref_list.append(ref)
|
||||
feat_list.append(item["audio_features"])
|
||||
mask_list.append(item["attention_mask"])
|
||||
|
||||
return {
|
||||
"tgt_latent": torch.stack(tgt_list),
|
||||
"ref_latent": torch.stack(ref_list),
|
||||
"audio_features": torch.stack(feat_list),
|
||||
"attention_mask": torch.stack(mask_list),
|
||||
"tgt_lengths": torch.tensor(tgt_lengths),
|
||||
"ref_lengths": torch.tensor(ref_lengths),
|
||||
}
|
||||
|
||||
|
||||
def parse_args():
|
||||
# First pass: pull out --config so its values can become argparse defaults.
|
||||
cfg_parser = argparse.ArgumentParser(add_help=False)
|
||||
cfg_parser.add_argument("--config", default=None,
|
||||
help="YAML file with default values for any of the flags below. "
|
||||
"Explicit CLI flags still override the YAML.")
|
||||
cfg_args, remaining = cfg_parser.parse_known_args()
|
||||
yaml_defaults: dict = {}
|
||||
if cfg_args.config:
|
||||
import yaml as _yaml
|
||||
with open(cfg_args.config) as f:
|
||||
yaml_defaults = _yaml.safe_load(f) or {}
|
||||
# YAML keys are dashes-or-underscores → normalize to argparse dest (underscore).
|
||||
yaml_defaults = {k.replace("-", "_"): v for k, v in yaml_defaults.items()}
|
||||
|
||||
def _yaml(name, fallback):
|
||||
return yaml_defaults.get(name, fallback)
|
||||
|
||||
p = argparse.ArgumentParser(
|
||||
parents=[cfg_parser],
|
||||
description="Audio-Only IC-LoRA Training for Voice Cloning",
|
||||
)
|
||||
p.add_argument("--data-dir", required="data_dir" not in yaml_defaults,
|
||||
nargs="+", default=_yaml("data_dir", None))
|
||||
p.add_argument("--speaker-index", required="speaker_index" not in yaml_defaults,
|
||||
nargs="+", default=_yaml("speaker_index", None))
|
||||
p.add_argument("--output-dir", default=_yaml("output_dir", os.path.join(MODEL_DIR, "tts_iclora_v1")))
|
||||
p.add_argument("--checkpoint", default=_yaml("checkpoint", os.path.join(MODEL_DIR, "dramabox-dit-v1.safetensors")))
|
||||
p.add_argument("--full-checkpoint", default=_yaml("full_checkpoint", os.path.join(MODEL_DIR, "dramabox-audio-components.safetensors")))
|
||||
p.add_argument("--base-model", choices=["distilled", "dev"], default=_yaml("base_model", "dev"),
|
||||
help="Base model type: distilled uses DistilledTimestepSampler, dev uses ShiftedLogitNormal")
|
||||
p.add_argument("--lora-rank", type=int, default=_yaml("lora_rank", 128))
|
||||
p.add_argument("--lora-alpha", type=int, default=_yaml("lora_alpha", 128))
|
||||
p.add_argument("--lora-dropout", type=float, default=_yaml("lora_dropout", 0.0),
|
||||
help="Dropout applied to LoRA A/B matrices during training. "
|
||||
"Recommended ~0.1 for small datasets to regularize.")
|
||||
p.add_argument("--resume-lora", default=_yaml("resume_lora", None))
|
||||
p.add_argument("--resume-step-offset", type=int, default=_yaml("resume_step_offset", None),
|
||||
help="Step to add when naming saved checkpoints. If None, inferred "
|
||||
"from --resume-lora filename (e.g. lora_step_10000.safetensors → 10000). "
|
||||
"Set to 0 to start numbering at 0 regardless.")
|
||||
p.add_argument("--ref-ratio", type=float, default=_yaml("ref_ratio", 0.3),
|
||||
help="Fraction of target length to use as reference (default 0.3)")
|
||||
p.add_argument("--max-ref-tokens", type=int, default=_yaml("max_ref_tokens", 200),
|
||||
help="Maximum reference tokens after patchification (default 200)")
|
||||
p.add_argument("--text-dropout", type=float, default=_yaml("text_dropout", 0.0),
|
||||
help="Probability of dropping text conditioning (forces reliance on voice ref)")
|
||||
p.add_argument("--steps", type=int, default=_yaml("steps", 30000))
|
||||
p.add_argument("--lr", type=float, default=_yaml("lr", 3e-5))
|
||||
p.add_argument("--lr-scheduler", choices=["cosine", "linear", "constant"], default=_yaml("lr_scheduler", "cosine"))
|
||||
p.add_argument("--batch-size", type=int, default=_yaml("batch_size", 1))
|
||||
p.add_argument("--grad-accum", type=int, default=_yaml("grad_accum", 4))
|
||||
p.add_argument("--max-grad-norm", type=float, default=_yaml("max_grad_norm", 1.0))
|
||||
p.add_argument("--save-every", type=int, default=_yaml("save_every", 1000))
|
||||
p.add_argument("--log-every", type=int, default=_yaml("log_every", 50))
|
||||
p.add_argument("--seed", type=int, default=_yaml("seed", 42))
|
||||
p.add_argument("--warmup-steps", type=int, default=_yaml("warmup_steps", 100))
|
||||
p.add_argument("--val-config", default=_yaml("val_config", None))
|
||||
return p.parse_args(remaining)
|
||||
|
||||
|
||||
# ─── Main ───
|
||||
|
||||
def main():
|
||||
from accelerate import Accelerator
|
||||
from accelerate.utils import set_seed
|
||||
|
||||
args = parse_args()
|
||||
|
||||
accelerator = Accelerator(
|
||||
gradient_accumulation_steps=args.grad_accum,
|
||||
mixed_precision="bf16",
|
||||
)
|
||||
|
||||
is_main = accelerator.is_main_process
|
||||
if is_main:
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
else:
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
|
||||
set_seed(args.seed)
|
||||
device = accelerator.device
|
||||
dtype = torch.bfloat16
|
||||
|
||||
os.makedirs(args.output_dir, exist_ok=True)
|
||||
|
||||
# Save training args
|
||||
if is_main:
|
||||
import yaml
|
||||
args_dict = vars(args).copy()
|
||||
args_dict["_meta"] = {
|
||||
"world_size": accelerator.num_processes,
|
||||
"dtype": str(dtype),
|
||||
"timestamp": time.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"script": "train_audio_iclora.py",
|
||||
"pattern": "IC-LoRA (ref appended to end)",
|
||||
}
|
||||
with open(os.path.join(args.output_dir, "training_args.yaml"), "w") as f:
|
||||
yaml.dump(args_dict, f, default_flow_style=False, sort_keys=False)
|
||||
|
||||
from ltx_core.components.patchifiers import AudioPatchifier
|
||||
from ltx_core.model.transformer.modality import Modality
|
||||
from ltx_core.guidance.perturbations import BatchedPerturbationConfig
|
||||
from ltx_core.tools import AudioLatentTools
|
||||
from ltx_core.types import AudioLatentShape, LatentState
|
||||
from ltx_pipelines.utils.helpers import modality_from_latent_state, timesteps_from_mask
|
||||
|
||||
# Build speaker map
|
||||
if is_main:
|
||||
logging.info("Building speaker map...")
|
||||
speaker_map = build_speaker_map(args.speaker_index, args.data_dir)
|
||||
if is_main:
|
||||
logging.info(f"Speaker map: {len(speaker_map)} speakers, "
|
||||
f"{sum(len(v) for v in speaker_map.values())} samples")
|
||||
|
||||
# Load model
|
||||
if is_main:
|
||||
logging.info("Loading audio-only model...")
|
||||
model = build_audio_only_model(args.checkpoint, device, dtype)
|
||||
|
||||
if is_main:
|
||||
logging.info("Loading audio connector...")
|
||||
audio_connector = load_audio_connector(args.full_checkpoint, device, dtype)
|
||||
audio_connector.eval()
|
||||
for p in audio_connector.parameters():
|
||||
p.requires_grad = False
|
||||
|
||||
if is_main:
|
||||
logging.info(f"Applying LoRA (rank={args.lora_rank}, alpha={args.lora_alpha})...")
|
||||
model = apply_lora(model, args.lora_rank, args.lora_alpha, args.lora_dropout)
|
||||
|
||||
# Resume from checkpoint
|
||||
if args.resume_lora:
|
||||
from safetensors.torch import load_file as st_load
|
||||
if is_main:
|
||||
logging.info(f"Resuming from: {args.resume_lora}")
|
||||
lora_sd = st_load(args.resume_lora)
|
||||
mapped = {}
|
||||
for k, v in lora_sd.items():
|
||||
nk = k.replace(".lora_A.weight", ".lora_A.default.weight").replace(
|
||||
".lora_B.weight", ".lora_B.default.weight")
|
||||
mapped[nk] = v
|
||||
model.load_state_dict(mapped, strict=False)
|
||||
|
||||
# Determine step offset for save filenames. Without this, resuming a run
|
||||
# restarts step numbering at 0 and would overwrite earlier phase-1
|
||||
# checkpoints with the same save_every cadence.
|
||||
if args.resume_step_offset is None:
|
||||
resume_offset = 0
|
||||
if args.resume_lora:
|
||||
import re as _re
|
||||
m = _re.search(r"lora_step_(\d+)", os.path.basename(args.resume_lora))
|
||||
if m:
|
||||
resume_offset = int(m.group(1))
|
||||
args.resume_step_offset = resume_offset
|
||||
if is_main and args.resume_step_offset:
|
||||
logging.info(f"Save-step offset: +{args.resume_step_offset}")
|
||||
|
||||
model.train()
|
||||
model.base_model.model.set_gradient_checkpointing(True)
|
||||
|
||||
# Dataset & DataLoader
|
||||
dataset = IDLoRADataset(speaker_map)
|
||||
if is_main:
|
||||
logging.info(f"Dataset: {len(dataset)} samples, {len(dataset.speaker_map)} speakers")
|
||||
|
||||
dataloader = DataLoader(dataset, batch_size=args.batch_size, shuffle=True, num_workers=2,
|
||||
pin_memory=True, drop_last=True, collate_fn=collate_audio_batch)
|
||||
|
||||
# Optimizer & Scheduler
|
||||
optimizer = torch.optim.AdamW(
|
||||
[p for p in model.parameters() if p.requires_grad],
|
||||
lr=args.lr, betas=(0.9, 0.999), weight_decay=0.01,
|
||||
)
|
||||
|
||||
from torch.optim.lr_scheduler import CosineAnnealingLR, LinearLR, SequentialLR, ConstantLR
|
||||
warmup = LinearLR(optimizer, start_factor=0.01, end_factor=1.0, total_iters=args.warmup_steps)
|
||||
remaining = args.steps - args.warmup_steps
|
||||
if args.lr_scheduler == "cosine":
|
||||
# Warmup -> constant hold (20% of remaining) -> cosine decay
|
||||
hold_steps = max(remaining // 5, 0)
|
||||
decay_steps = max(remaining - hold_steps, 1)
|
||||
hold_sched = ConstantLR(optimizer, factor=1.0, total_iters=hold_steps)
|
||||
decay_sched = CosineAnnealingLR(optimizer, T_max=decay_steps, eta_min=1e-6)
|
||||
scheduler = SequentialLR(
|
||||
optimizer,
|
||||
[warmup, hold_sched, decay_sched],
|
||||
milestones=[args.warmup_steps, args.warmup_steps + hold_steps],
|
||||
)
|
||||
elif args.lr_scheduler == "linear":
|
||||
main_sched = LinearLR(optimizer, start_factor=1.0, end_factor=0.01, total_iters=max(remaining, 1))
|
||||
scheduler = SequentialLR(optimizer, [warmup, main_sched], milestones=[args.warmup_steps])
|
||||
else:
|
||||
main_sched = ConstantLR(optimizer, factor=1.0, total_iters=max(remaining, 1))
|
||||
scheduler = SequentialLR(optimizer, [warmup, main_sched], milestones=[args.warmup_steps])
|
||||
|
||||
# Prepare with Accelerate — but NOT the scheduler. AcceleratedScheduler
|
||||
# calls the underlying scheduler.step() `num_processes` times per sync,
|
||||
# which silently scales down our warmup/cosine spans by that factor.
|
||||
# We call scheduler.step() ourselves, gated on sync_gradients → exactly
|
||||
# one advance per optimizer step, as the yaml spec intends.
|
||||
model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
|
||||
|
||||
patchifier = AudioPatchifier(patch_size=1)
|
||||
|
||||
# Select timestep sampler based on base model type
|
||||
if args.base_model == "distilled":
|
||||
timestep_sampler = DistilledTimestepSampler()
|
||||
if is_main:
|
||||
logging.info("Using DistilledTimestepSampler (matching distilled model sigmas)")
|
||||
else:
|
||||
timestep_sampler = ShiftedLogitNormalTimestepSampler()
|
||||
if is_main:
|
||||
logging.info("Using ShiftedLogitNormalTimestepSampler (dev model)")
|
||||
|
||||
# Training loop
|
||||
if is_main:
|
||||
logging.info(f"Training: {args.steps} steps, lr={args.lr}, scheduler={args.lr_scheduler}, "
|
||||
f"batch={args.batch_size}, grad_accum={args.grad_accum}, "
|
||||
f"world_size={accelerator.num_processes}, "
|
||||
f"ref_ratio={args.ref_ratio}, max_ref_tokens={args.max_ref_tokens}")
|
||||
logging.info("IC-LoRA pattern: ref tokens APPENDED to target, loss on target only")
|
||||
|
||||
data_iter = iter(dataloader)
|
||||
step = 0
|
||||
accum_loss = 0.0
|
||||
best_loss = float("inf")
|
||||
best_step = 0
|
||||
t0 = time.time()
|
||||
|
||||
total_micro_steps = args.steps * args.grad_accum
|
||||
|
||||
for micro_step in range(total_micro_steps):
|
||||
try:
|
||||
batch = next(data_iter)
|
||||
except StopIteration:
|
||||
data_iter = iter(dataloader)
|
||||
batch = next(data_iter)
|
||||
|
||||
is_opt_step = (micro_step + 1) % args.grad_accum == 0
|
||||
if is_opt_step:
|
||||
step += 1
|
||||
if is_main:
|
||||
# TTS Audio Suite patch: provide lightweight per-step telemetry
|
||||
# to the parent process even when human logs use log_every=50.
|
||||
print(
|
||||
f"TTS_SUITE_PROGRESS step={step} total={args.steps}",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
with accelerator.accumulate(model):
|
||||
tgt_latent = batch["tgt_latent"].to(dtype=dtype) # [B, C, max_tgt_T, F]
|
||||
ref_latent = batch["ref_latent"].to(dtype=dtype) # [B, C, max_ref_T, F]
|
||||
tgt_lengths = batch["tgt_lengths"].to(device=device) # [B]
|
||||
B = tgt_latent.shape[0]
|
||||
|
||||
# ── Random silence padding (0-1s) ── ltx_audio_tts baseline.
|
||||
# User observed reference-audio leak at end of generations when this
|
||||
# was reduced to 5 (v14) or 10 frames (v16/v17) — the model seemed
|
||||
# to use the extra target budget to regurgitate ref content. Full
|
||||
# 25 frames (0-1s avg 500ms) was apparently load-bearing for
|
||||
# regularising the boundary and reducing hallucinations.
|
||||
# Uses the real silence latent (not zeros) so the VAE decodes it as
|
||||
# true silence instead of static noise.
|
||||
max_pad_frames = 25 # ~1s at 25 latent frames/sec
|
||||
pad_frames = random.randint(0, max_pad_frames)
|
||||
if pad_frames > 0:
|
||||
C, F_dim = tgt_latent.shape[1], tgt_latent.shape[3]
|
||||
if not hasattr(args, '_silence_frame') or args._silence_frame is None:
|
||||
_sf_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "assets", "silence_latent_frame.pt")
|
||||
if os.path.exists(_sf_path):
|
||||
args._silence_frame = torch.load(_sf_path, weights_only=True) # [C, 1, F]
|
||||
if is_main:
|
||||
logging.info(f"Loaded silence latent from {_sf_path}")
|
||||
else:
|
||||
args._silence_frame = False # fallback to zeros
|
||||
if is_main:
|
||||
logging.warning(f"silence_latent_frame.pt not found, using zeros")
|
||||
if args._silence_frame is not False:
|
||||
sf = args._silence_frame.to(dtype=dtype, device=device) # [C, 1, F]
|
||||
silence_pad = sf.unsqueeze(0).expand(B, -1, pad_frames, -1) # [B, C, pad, F]
|
||||
else:
|
||||
silence_pad = torch.zeros(B, C, pad_frames, F_dim, dtype=dtype, device=device)
|
||||
tgt_latent = torch.cat([silence_pad, tgt_latent], dim=2)
|
||||
|
||||
# Cap reference to max_ref_tokens (in latent frames, before patchification)
|
||||
# After patchification, ref_T tokens = ref frames (patch_size=1)
|
||||
ref_T_frames = min(ref_latent.shape[2], args.max_ref_tokens)
|
||||
ref_latent = ref_latent[:, :, :ref_T_frames, :]
|
||||
|
||||
tgt_T_frames = tgt_latent.shape[2] # max (padded) target frames
|
||||
|
||||
# ── Step 1: Create target AudioLatentShape and AudioLatentTools ──
|
||||
tgt_shape = AudioLatentShape(
|
||||
batch=B,
|
||||
channels=tgt_latent.shape[1], # 8
|
||||
frames=tgt_T_frames,
|
||||
mel_bins=tgt_latent.shape[3], # 16
|
||||
)
|
||||
|
||||
audio_tools = AudioLatentTools(
|
||||
patchifier=patchifier,
|
||||
target_shape=tgt_shape,
|
||||
)
|
||||
|
||||
# ── Step 2: Create initial state from target latent ──
|
||||
# create_initial_state patchifies: [B, C, T, F] -> [B, T, C*F]
|
||||
# Also creates denoise_mask=1 (all target tokens will be denoised)
|
||||
# and computes temporal positions
|
||||
state = audio_tools.create_initial_state(
|
||||
device=device,
|
||||
dtype=dtype,
|
||||
initial_latent=tgt_latent,
|
||||
)
|
||||
# state.latent: [B, tgt_T, 128], state.denoise_mask: [B, tgt_T, 1]
|
||||
# state.positions: [B, 1, tgt_T, 2]
|
||||
|
||||
tgt_T = audio_tools.target_shape.token_count() # = tgt_T_frames
|
||||
|
||||
# ── Step 3: Apply flow-matching noise to target BEFORE appending ref ──
|
||||
# Sample sigma
|
||||
total_tokens = tgt_T + ref_T_frames
|
||||
sigma = timestep_sampler.sample(B, total_tokens, device=device)
|
||||
sigma_exp = sigma.view(-1, 1, 1) # [B, 1, 1]
|
||||
|
||||
noise = torch.randn_like(state.latent) # [B, tgt_T, 128]
|
||||
noisy_tgt = (1 - sigma_exp) * state.latent + sigma_exp * noise
|
||||
|
||||
# Replace the latent in state with the noisy version
|
||||
# (clean_latent stays clean for post_process_latent pattern)
|
||||
state = LatentState(
|
||||
latent=noisy_tgt,
|
||||
denoise_mask=state.denoise_mask,
|
||||
positions=state.positions,
|
||||
clean_latent=state.clean_latent,
|
||||
attention_mask=state.attention_mask,
|
||||
)
|
||||
|
||||
# ── Step 4: Append reference tokens using AudioConditionByReferenceLatent ──
|
||||
# This appends ref tokens to the END with denoise_mask=0 (frozen/clean)
|
||||
# Skip entirely when ref_T=0 (SFX / song samples): the model trains
|
||||
# target-only for those categories since there's no voice to clone.
|
||||
if ref_T_frames > 0:
|
||||
ref_conditioning = AudioConditionByReferenceLatent(
|
||||
latent=ref_latent,
|
||||
strength=1.0, # 1.0 = ref fully clean (denoise_mask=0)
|
||||
)
|
||||
state = ref_conditioning.apply_to(
|
||||
latent_state=state,
|
||||
latent_tools=audio_tools,
|
||||
)
|
||||
# state.latent: [B, tgt_T + ref_T, 128]
|
||||
# state.denoise_mask: [B, tgt_T + ref_T, 1]
|
||||
# target tokens: 1.0 (denoise), ref tokens: 0.0 (frozen)
|
||||
# state.positions: [B, 1, tgt_T + ref_T, 2]
|
||||
|
||||
# ── Step 5: Build loss mask for target tokens (excluding padding) ──
|
||||
# loss_mask: 1 for real target tokens, 0 for padding and ref tokens
|
||||
loss_mask = torch.zeros(B, tgt_T, device=device)
|
||||
for b_idx in range(B):
|
||||
real_len = min(tgt_lengths[b_idx].item(), tgt_T)
|
||||
loss_mask[b_idx, :real_len] = 1.0
|
||||
|
||||
# ── Step 6: Prepare text context ──
|
||||
# Text conditioning dropout: randomly zero out text context to force
|
||||
# the model to rely on the voice reference for identity/style.
|
||||
with torch.no_grad():
|
||||
audio_context = prepare_audio_context(
|
||||
audio_connector, batch["audio_features"],
|
||||
batch["attention_mask"], device, dtype)
|
||||
if args.text_dropout > 0 and random.random() < args.text_dropout:
|
||||
audio_context = torch.zeros_like(audio_context)
|
||||
|
||||
# ── Step 7: Build Modality using modality_from_latent_state ──
|
||||
# timesteps = sigma * denoise_mask (ref gets 0, target gets sigma)
|
||||
audio_mod = modality_from_latent_state(
|
||||
state=state,
|
||||
context=audio_context,
|
||||
sigma=sigma,
|
||||
enabled=True,
|
||||
)
|
||||
|
||||
# ── Step 8: Forward pass ──
|
||||
perturbations = BatchedPerturbationConfig.empty(B)
|
||||
with torch.autocast(device_type="cuda", dtype=dtype):
|
||||
_, velocity_pred = model(video=None, audio=audio_mod, perturbations=perturbations)
|
||||
|
||||
# ── Step 9: Compute loss (IC-LoRA pattern) ──
|
||||
# Target is at the FRONT (indices 0..tgt_T), ref at the END
|
||||
# velocity target = noise - clean
|
||||
tgt_patchified = audio_tools.patchifier.patchify(tgt_latent) # [B, tgt_T, 128]
|
||||
target_velocity = noise - tgt_patchified
|
||||
|
||||
# Extract target portion of prediction
|
||||
pred_tgt = velocity_pred[:, :tgt_T] # [B, tgt_T, 128]
|
||||
|
||||
# MSE loss with mask: only on real target tokens (not padding or ref)
|
||||
per_token_mse = (pred_tgt - target_velocity).pow(2).mean(dim=-1) # [B, tgt_T]
|
||||
loss = per_token_mse.mul(loss_mask).div(loss_mask.mean().clamp(min=1e-6)).mean()
|
||||
|
||||
accelerator.backward(loss)
|
||||
|
||||
if accelerator.sync_gradients and args.max_grad_norm > 0:
|
||||
accelerator.clip_grad_norm_(model.parameters(), args.max_grad_norm)
|
||||
|
||||
optimizer.step()
|
||||
optimizer.zero_grad()
|
||||
# Only advance the LR scheduler once per OPTIMIZER step (not per
|
||||
# micro-step). Mirrors AcceleratedOptimizer.step() which is
|
||||
# internally gated on sync_gradients.
|
||||
if accelerator.sync_gradients:
|
||||
scheduler.step()
|
||||
|
||||
accum_loss += loss.item()
|
||||
|
||||
# Logging & saving on optimization steps only
|
||||
if is_opt_step and step % args.log_every == 0 and is_main:
|
||||
avg_loss = accum_loss / (args.log_every * args.grad_accum)
|
||||
lr = optimizer.param_groups[0]["lr"]
|
||||
elapsed = time.time() - t0
|
||||
sps = step / elapsed if elapsed > 0 else 0
|
||||
eta = (args.steps - step) / sps if sps > 0 else 0
|
||||
logging.info(
|
||||
f"Step {step}/{args.steps} | loss={avg_loss:.4f} | lr={lr:.2e} | "
|
||||
f"tgt_T={tgt_T} ref_T={ref_T_frames} total={tgt_T + ref_T_frames} | "
|
||||
f"{sps:.1f} steps/s | ETA {eta/60:.0f}min"
|
||||
)
|
||||
|
||||
# Save best whenever loss improves — no warmup gate, so we can
|
||||
# observe best checkpoints during warmup too.
|
||||
if avg_loss < best_loss:
|
||||
best_loss = avg_loss
|
||||
old_best = os.path.join(args.output_dir, f"best_step_{best_step:05d}.safetensors")
|
||||
best_step = step + args.resume_step_offset
|
||||
new_best = os.path.join(args.output_dir, f"best_step_{best_step:05d}.safetensors")
|
||||
unwrapped = _unwrap_model_safe(model)
|
||||
unwrapped.save_pretrained(args.output_dir)
|
||||
adapter = os.path.join(args.output_dir, "adapter_model.safetensors")
|
||||
if os.path.exists(adapter):
|
||||
shutil.copy(adapter, new_best)
|
||||
if old_best != new_best and os.path.exists(old_best):
|
||||
os.remove(old_best)
|
||||
logging.info(f"New best: loss={best_loss:.4f} at step {best_step}")
|
||||
|
||||
accum_loss = 0.0
|
||||
|
||||
if is_opt_step and step % args.save_every == 0 and is_main:
|
||||
global_step = step + args.resume_step_offset
|
||||
save_path = os.path.join(args.output_dir, f"lora_step_{global_step:05d}.safetensors")
|
||||
logging.info(f"Saving: {save_path}")
|
||||
unwrapped = _unwrap_model_safe(model)
|
||||
unwrapped.save_pretrained(args.output_dir)
|
||||
adapter = os.path.join(args.output_dir, "adapter_model.safetensors")
|
||||
if os.path.exists(adapter):
|
||||
shutil.copy(adapter, save_path)
|
||||
|
||||
if args.val_config:
|
||||
logging.info(f"Running validation at step {global_step}...")
|
||||
model.eval()
|
||||
run_validation(save_path, args.val_config, args.output_dir, global_step,
|
||||
lora_rank=args.lora_rank)
|
||||
model.train()
|
||||
|
||||
# Final save
|
||||
if is_main:
|
||||
unwrapped = _unwrap_model_safe(model)
|
||||
unwrapped.save_pretrained(args.output_dir)
|
||||
adapter = os.path.join(args.output_dir, "adapter_model.safetensors")
|
||||
global_step = step + args.resume_step_offset
|
||||
save_path = os.path.join(args.output_dir, f"lora_step_{global_step:05d}.safetensors")
|
||||
if os.path.exists(adapter):
|
||||
shutil.copy(adapter, save_path)
|
||||
logging.info(f"Training complete! {step} steps in {time.time()-t0:.0f}s")
|
||||
logging.info(f"Best loss: {best_loss:.4f} at step {best_step}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
-370
@@ -1,370 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Warm validation runner — loads base dev + LoRA + all aux models ONCE,
|
||||
then iterates every speaker in val_config generating each output.
|
||||
|
||||
Matches the same generation path as inference.py but keeps Gemma / audio VAE
|
||||
/ velocity model / audio decoder resident across entries. Inference
|
||||
settings default to the Gradio warm-server values (cfg=2.5, stg=1.5,
|
||||
modality=1.0, rescale=0, 30 steps, fps=25) — use --inference-params to
|
||||
override.
|
||||
"""
|
||||
import argparse
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import traceback
|
||||
|
||||
import torch
|
||||
import torchaudio
|
||||
|
||||
REPO_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
MODEL_DIR = REPO_DIR
|
||||
sys.path.insert(0, os.path.join(REPO_DIR, "ltx2"))
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
DEV_FULL_CKPT = os.environ.get(
|
||||
"LTX_FULL_CHECKPOINT",
|
||||
os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "ltx-2.3-22b-dev.safetensors"),
|
||||
)
|
||||
# TTS Audio Suite patch: organized DramaBox installs keep the transformer and
|
||||
# audio components in separate checkpoints, unlike the upstream full checkpoint.
|
||||
DRAMABOX_TRANSFORMER_CKPT = os.environ.get("LTX_CHECKPOINT", DEV_FULL_CKPT)
|
||||
GEMMA_ROOT = os.environ.get(
|
||||
"GEMMA_ROOT",
|
||||
os.path.expanduser("~/.cache/dramabox/gemma-3-12b-it-bnb-4bit"),
|
||||
)
|
||||
|
||||
|
||||
def parse_args():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--val-config", required=True)
|
||||
p.add_argument("--output-dir", required=True)
|
||||
p.add_argument("--lora", default=None)
|
||||
p.add_argument("--lora-rank", type=int, default=128)
|
||||
p.add_argument("--checkpoint", default=DRAMABOX_TRANSFORMER_CKPT)
|
||||
p.add_argument("--full-checkpoint", default=DEV_FULL_CKPT)
|
||||
p.add_argument("--gemma-root", default=GEMMA_ROOT)
|
||||
p.add_argument("--cfg-scale", type=float, default=2.5)
|
||||
p.add_argument("--stg-scale", type=float, default=1.5)
|
||||
p.add_argument("--rescale-scale", type=float, default=0.0)
|
||||
p.add_argument("--modality-scale", type=float, default=1.0)
|
||||
p.add_argument("--steps", type=int, default=30)
|
||||
p.add_argument("--fps", type=float, default=25.0)
|
||||
p.add_argument("--stg-block", type=int, default=29)
|
||||
p.add_argument("--cfg-clamp", type=float, default=0.0)
|
||||
p.add_argument("--seed", type=int, default=42)
|
||||
p.add_argument("--duration-multiplier", type=float, default=1.1)
|
||||
# Match Gradio / inference_server.py DEFAULT_NEG exactly
|
||||
p.add_argument("--negative-prompt", default=(
|
||||
"worst quality, inconsistent, robotic, distorted, noise, static, "
|
||||
"muffled, unclear, unnatural, monotone"
|
||||
))
|
||||
return p.parse_args()
|
||||
|
||||
|
||||
def estimate_speech_duration(prompt: str, speed: float = 1.0) -> float:
|
||||
import re
|
||||
quoted = re.findall(r'"([^"]*)"', prompt) or re.findall(r"'([^']*)'", prompt)
|
||||
text = " ".join(quoted) if quoted else prompt
|
||||
duration = len(text) * 0.065 / max(speed, 0.1) + 1.5
|
||||
return max(3.0, round(duration, 1))
|
||||
|
||||
|
||||
class WarmValidator:
|
||||
def __init__(self, checkpoint, full_checkpoint, gemma_root, lora_path=None, lora_rank=128,
|
||||
device="cuda", dtype=torch.bfloat16):
|
||||
from audio_conditioning import AudioConditionByReferenceLatent # noqa: F401 (imported by inference.py)
|
||||
from ltx_core.components.patchifiers import AudioPatchifier
|
||||
from ltx_pipelines.utils.blocks import PromptEncoder, AudioConditioner, AudioDecoder
|
||||
|
||||
self.device = torch.device(device)
|
||||
self.dtype = dtype
|
||||
self.full_checkpoint = full_checkpoint
|
||||
self.gemma_root = gemma_root
|
||||
self.patchifier = AudioPatchifier(patch_size=1)
|
||||
|
||||
logging.info("Loading PromptEncoder (Gemma + embeddings_processor)...")
|
||||
t0 = time.time()
|
||||
self.prompt_encoder = PromptEncoder(
|
||||
checkpoint_path=full_checkpoint, gemma_root=gemma_root,
|
||||
dtype=dtype, device=self.device, warm=True, audio_only=True,
|
||||
)
|
||||
logging.info(f" PromptEncoder ready in {time.time()-t0:.1f}s")
|
||||
|
||||
logging.info("Loading AudioConditioner (audio VAE encoder)...")
|
||||
t0 = time.time()
|
||||
self.audio_conditioner = AudioConditioner(
|
||||
checkpoint_path=full_checkpoint, dtype=dtype, device=self.device, warm=True,
|
||||
)
|
||||
logging.info(f" AudioConditioner ready in {time.time()-t0:.1f}s")
|
||||
|
||||
logging.info("Loading AudioDecoder...")
|
||||
t0 = time.time()
|
||||
self.audio_decoder = AudioDecoder(
|
||||
checkpoint_path=full_checkpoint, dtype=dtype, device=self.device, warm=True,
|
||||
)
|
||||
logging.info(f" AudioDecoder ready in {time.time()-t0:.1f}s")
|
||||
|
||||
logging.info("Building velocity model (audio-only from base dev)...")
|
||||
t0 = time.time()
|
||||
# TTS Audio Suite patch: build the DiT from the dedicated transformer
|
||||
# checkpoint while keeping the audio connector/VAE/decoder checkpoint separate.
|
||||
self.velocity_model = self._build_velocity_model(checkpoint, lora_path, lora_rank)
|
||||
logging.info(f" Velocity model ready in {time.time()-t0:.1f}s "
|
||||
f"({sum(p.numel() for p in self.velocity_model.parameters()) / 1e9:.1f}B params)")
|
||||
|
||||
def _build_velocity_model(self, checkpoint_path, lora_path, lora_rank):
|
||||
from ltx_core.loader.registry import DummyRegistry
|
||||
from ltx_core.loader.sd_ops import SDOps
|
||||
from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder
|
||||
from ltx_core.model.model_protocol import ModelConfigurator
|
||||
from ltx_core.model.transformer.attention import AttentionFunction
|
||||
from ltx_core.model.transformer.model import LTXModel, LTXModelType
|
||||
from ltx_core.model.transformer.rope import LTXRopeType
|
||||
|
||||
sd_ops = (
|
||||
SDOps("AO")
|
||||
.with_matching(prefix="model.diffusion_model.")
|
||||
.with_replacement("model.diffusion_model.", "")
|
||||
)
|
||||
|
||||
class Cfg(ModelConfigurator[LTXModel]):
|
||||
@classmethod
|
||||
def from_config(cls, config):
|
||||
t = config.get("transformer", {})
|
||||
cp = None
|
||||
if not t.get("caption_proj_before_connector", False):
|
||||
from ltx_core.model.transformer.text_projection import create_caption_projection
|
||||
with torch.device("meta"):
|
||||
cp = create_caption_projection(t, audio=True)
|
||||
return LTXModel(
|
||||
model_type=LTXModelType.AudioOnly,
|
||||
audio_num_attention_heads=t.get("audio_num_attention_heads", 32),
|
||||
audio_attention_head_dim=t.get("audio_attention_head_dim", 64),
|
||||
audio_in_channels=t.get("audio_in_channels", 128),
|
||||
audio_out_channels=t.get("audio_out_channels", 128),
|
||||
num_layers=t.get("num_layers", 48),
|
||||
audio_cross_attention_dim=t.get("audio_cross_attention_dim", 2048),
|
||||
norm_eps=t.get("norm_eps", 1e-6),
|
||||
attention_type=AttentionFunction(t.get("attention_type", "default")),
|
||||
positional_embedding_theta=10000.0,
|
||||
audio_positional_embedding_max_pos=[20.0],
|
||||
timestep_scale_multiplier=t.get("timestep_scale_multiplier", 1000),
|
||||
use_middle_indices_grid=t.get("use_middle_indices_grid", True),
|
||||
rope_type=LTXRopeType(t.get("rope_type", "interleaved")),
|
||||
double_precision_rope=t.get("frequencies_precision", False) == "float64",
|
||||
apply_gated_attention=t.get("apply_gated_attention", False),
|
||||
audio_caption_projection=cp,
|
||||
cross_attention_adaln=t.get("cross_attention_adaln", False),
|
||||
)
|
||||
|
||||
builder = Builder(
|
||||
model_path=checkpoint_path, model_class_configurator=Cfg,
|
||||
model_sd_ops=sd_ops, registry=DummyRegistry(),
|
||||
)
|
||||
velocity = builder.build(device=self.device, dtype=self.dtype).to(self.device).eval()
|
||||
|
||||
if lora_path and os.path.exists(lora_path):
|
||||
from peft import LoraConfig, get_peft_model
|
||||
from safetensors.torch import load_file as st_load
|
||||
logging.info(f"Attaching LoRA: {lora_path}")
|
||||
lora_sd = st_load(lora_path)
|
||||
is_peft = any("base_model.model." in k for k in lora_sd.keys())
|
||||
is_iclora = any("diffusion_model." in k for k in lora_sd.keys())
|
||||
cfg = LoraConfig(
|
||||
r=lora_rank, lora_alpha=lora_rank, lora_dropout=0.0, bias="none",
|
||||
target_modules=[
|
||||
"audio_attn1.to_k", "audio_attn1.to_q",
|
||||
"audio_attn1.to_v", "audio_attn1.to_out.0",
|
||||
"audio_attn2.to_k", "audio_attn2.to_q",
|
||||
"audio_attn2.to_v", "audio_attn2.to_out.0",
|
||||
"audio_ff.net.0.proj", "audio_ff.net.2",
|
||||
],
|
||||
)
|
||||
velocity = get_peft_model(velocity, cfg)
|
||||
|
||||
if is_peft:
|
||||
mapped = {}
|
||||
for k, v in lora_sd.items():
|
||||
nk = k
|
||||
if ".lora_A.weight" in k and ".lora_A.default.weight" not in k:
|
||||
nk = k.replace(".lora_A.weight", ".lora_A.default.weight")
|
||||
if ".lora_B.weight" in k and ".lora_B.default.weight" not in k:
|
||||
nk = k.replace(".lora_B.weight", ".lora_B.default.weight")
|
||||
mapped[nk] = v
|
||||
_, unexpected = velocity.load_state_dict(mapped, strict=False)
|
||||
logging.info(f" Loaded {len(mapped) - len(unexpected)} LoRA weights (peft)")
|
||||
elif is_iclora:
|
||||
audio_keys = {k: v for k, v in lora_sd.items()
|
||||
if "audio_attn1" in k or "audio_attn2" in k or "audio_ff" in k}
|
||||
mapped = {}
|
||||
for k, v in audio_keys.items():
|
||||
nk = k.replace("diffusion_model.", "base_model.model.")
|
||||
nk = nk.replace(".lora_A.weight", ".lora_A.default.weight")
|
||||
nk = nk.replace(".lora_B.weight", ".lora_B.default.weight")
|
||||
mapped[nk] = v
|
||||
_, unexpected = velocity.load_state_dict(mapped, strict=False)
|
||||
logging.info(f" Loaded {len(mapped) - len(unexpected)} LoRA weights (iclora)")
|
||||
|
||||
velocity = velocity.merge_and_unload()
|
||||
logging.info(" Merged LoRA into base weights")
|
||||
|
||||
return velocity
|
||||
|
||||
@torch.inference_mode()
|
||||
def generate(self, prompt, output_path, voice_ref=None, args=None):
|
||||
from audio_conditioning import AudioConditionByReferenceLatent
|
||||
from ltx_core.batch_split import BatchSplitAdapter
|
||||
from ltx_core.components.diffusion_steps import EulerDiffusionStep
|
||||
from ltx_core.components.guiders import MultiModalGuider, MultiModalGuiderParams
|
||||
from ltx_core.components.noisers import GaussianNoiser
|
||||
from ltx_core.components.schedulers import LTX2Scheduler
|
||||
from ltx_core.model.audio_vae import encode_audio as vae_encode_audio
|
||||
from ltx_core.model.transformer.model import X0Model
|
||||
from ltx_core.tools import AudioLatentTools
|
||||
from ltx_core.types import Audio, AudioLatentShape, VideoPixelShape
|
||||
from ltx_pipelines.utils.denoisers import GuidedDenoiser, SimpleDenoiser
|
||||
from ltx_pipelines.utils.gpu_model import gpu_model
|
||||
from ltx_pipelines.utils.media_io import decode_audio_from_file
|
||||
from ltx_pipelines.utils.samplers import euler_denoising_loop
|
||||
|
||||
t_total = time.time()
|
||||
|
||||
# ---- Duration + shape ----
|
||||
gen_dur = estimate_speech_duration(prompt) * args.duration_multiplier
|
||||
raw_frames = int(round(gen_dur * args.fps)) + 1
|
||||
num_frames = ((raw_frames - 1 + 4) // 8) * 8 + 1
|
||||
pixel_shape = VideoPixelShape(batch=1, frames=num_frames, height=64, width=64, fps=args.fps)
|
||||
tgt_shape = AudioLatentShape.from_video_pixel_shape(pixel_shape)
|
||||
audio_tools = AudioLatentTools(patchifier=self.patchifier, target_shape=tgt_shape)
|
||||
|
||||
state = audio_tools.create_initial_state(self.device, self.dtype)
|
||||
|
||||
# ---- Voice reference ----
|
||||
if voice_ref and os.path.exists(voice_ref):
|
||||
voice = decode_audio_from_file(voice_ref, self.device, 0.0, 10.0)
|
||||
if voice is not None:
|
||||
w = voice.waveform
|
||||
if w.dim() == 2:
|
||||
if w.shape[0] == 1:
|
||||
w = w.repeat(2, 1)
|
||||
w = w.unsqueeze(0)
|
||||
elif w.dim() == 3 and w.shape[1] == 1:
|
||||
w = w.repeat(1, 2, 1)
|
||||
target_samples = int(10.0 * voice.sampling_rate)
|
||||
if w.shape[-1] < target_samples:
|
||||
w = w.repeat(1, 1, (target_samples // w.shape[-1]) + 1)
|
||||
w = w[..., :target_samples]
|
||||
peak = w.abs().max()
|
||||
if peak > 0:
|
||||
w = w * (10 ** (-4.0 / 20) / peak)
|
||||
voice = Audio(waveform=w, sampling_rate=voice.sampling_rate)
|
||||
ref_latent = self.audio_conditioner(lambda enc: vae_encode_audio(voice, enc, None))
|
||||
cond = AudioConditionByReferenceLatent(
|
||||
latent=ref_latent.to(self.device, self.dtype), strength=1.0,
|
||||
)
|
||||
state = cond.apply_to(latent_state=state, latent_tools=audio_tools)
|
||||
|
||||
# ---- Noise ----
|
||||
gen = torch.Generator(device=self.device).manual_seed(args.seed)
|
||||
noiser = GaussianNoiser(generator=gen)
|
||||
state = noiser(state, noise_scale=1.0)
|
||||
|
||||
# ---- Prompt encode ----
|
||||
use_cfg = args.cfg_scale > 1.0
|
||||
prompts = [prompt, args.negative_prompt] if use_cfg else [prompt]
|
||||
ctx = self.prompt_encoder(prompts, streaming_prefetch_count=None)
|
||||
a_ctx = ctx[0].audio_encoding
|
||||
a_ctx_neg = ctx[1].audio_encoding if use_cfg else None
|
||||
|
||||
# ---- Denoiser ----
|
||||
needs_guidance = args.cfg_scale > 1.0 or args.stg_scale > 0.0 or args.modality_scale > 1.0
|
||||
if needs_guidance:
|
||||
guider = MultiModalGuider(
|
||||
params=MultiModalGuiderParams(
|
||||
cfg_scale=args.cfg_scale, stg_scale=args.stg_scale,
|
||||
stg_blocks=[args.stg_block] if args.stg_scale > 0 else [],
|
||||
rescale_scale=args.rescale_scale,
|
||||
modality_scale=args.modality_scale,
|
||||
cfg_clamp_scale=args.cfg_clamp,
|
||||
),
|
||||
negative_context=a_ctx_neg,
|
||||
)
|
||||
denoiser = GuidedDenoiser(
|
||||
v_context=None, a_context=a_ctx,
|
||||
video_guider=None, audio_guider=guider,
|
||||
)
|
||||
else:
|
||||
denoiser = SimpleDenoiser(v_context=None, a_context=a_ctx)
|
||||
|
||||
sigmas = LTX2Scheduler().execute(steps=args.steps, latent=state.latent).to(self.device)
|
||||
|
||||
# ---- Denoise ----
|
||||
# NOTE: don't wrap in gpu_model() — that context manager moves the
|
||||
# model back off GPU on exit, which breaks subsequent iterations of
|
||||
# our warm validator. We keep the velocity model resident.
|
||||
x0 = X0Model(self.velocity_model)
|
||||
batched = BatchSplitAdapter(x0, max_batch_size=1)
|
||||
_, audio_state = euler_denoising_loop(
|
||||
sigmas=sigmas, video_state=None, audio_state=state,
|
||||
stepper=EulerDiffusionStep(), transformer=batched, denoiser=denoiser,
|
||||
)
|
||||
|
||||
audio_state = audio_tools.clear_conditioning(audio_state)
|
||||
audio_state = audio_tools.unpatchify(audio_state)
|
||||
decoded = self.audio_decoder(audio_state.latent)
|
||||
|
||||
wav = decoded.waveform
|
||||
if wav.dim() == 1:
|
||||
wav = wav.unsqueeze(0)
|
||||
os.makedirs(os.path.dirname(output_path) or ".", exist_ok=True)
|
||||
torchaudio.save(output_path, wav.float().cpu(), decoded.sampling_rate)
|
||||
logging.info(f" -> {output_path} ({wav.shape[-1]/decoded.sampling_rate:.1f}s, "
|
||||
f"{time.time()-t_total:.1f}s)")
|
||||
|
||||
|
||||
def main():
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
args = parse_args()
|
||||
import yaml
|
||||
with open(args.val_config) as f:
|
||||
val_cfg = yaml.safe_load(f)
|
||||
os.makedirs(args.output_dir, exist_ok=True)
|
||||
|
||||
# Build validator once (models warm for all entries).
|
||||
validator = WarmValidator(
|
||||
checkpoint=args.checkpoint,
|
||||
full_checkpoint=args.full_checkpoint,
|
||||
gemma_root=args.gemma_root,
|
||||
lora_path=args.lora,
|
||||
lora_rank=args.lora_rank,
|
||||
device="cuda" if torch.cuda.is_available() else "cpu",
|
||||
dtype=torch.bfloat16,
|
||||
)
|
||||
|
||||
n_ok = n_fail = 0
|
||||
t0 = time.time()
|
||||
for entry in val_cfg.get("speakers", []):
|
||||
name = entry["name"]
|
||||
out_path = os.path.join(args.output_dir, f"{name}.wav")
|
||||
try:
|
||||
validator.generate(
|
||||
prompt=entry["prompt"],
|
||||
output_path=out_path,
|
||||
voice_ref=entry.get("reference"),
|
||||
args=args,
|
||||
)
|
||||
n_ok += 1
|
||||
logging.info(f" [{name}] OK")
|
||||
except Exception as e:
|
||||
n_fail += 1
|
||||
logging.warning(f" [{name}] FAILED: {e}")
|
||||
traceback.print_exc()
|
||||
|
||||
logging.info(f"Validation done: ok={n_ok} fail={n_fail} in {(time.time()-t0)/60:.1f}min "
|
||||
f"at {args.output_dir}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -184,11 +184,9 @@ class ECAPA_TDNN(nn.Module):
|
||||
torch.hub._validate_not_a_forked_repo = lambda a, b, c: True
|
||||
try:
|
||||
local_s3prl_path = os.path.expanduser("~/.cache/torch/hub/s3prl_s3prl_main")
|
||||
# TTS Audio Suite patch: restrict checkpoint deserialization.
|
||||
self.feature_extract = torch.hub.load(local_s3prl_path, feat_type, source="local", config_path=config_path, weights_only=True)
|
||||
self.feature_extract = torch.hub.load(local_s3prl_path, feat_type, source="local", config_path=config_path)
|
||||
except: # noqa: E722
|
||||
# TTS Audio Suite patch: restrict checkpoint deserialization.
|
||||
self.feature_extract = torch.hub.load("s3prl/s3prl", feat_type, weights_only=True)
|
||||
self.feature_extract = torch.hub.load("s3prl/s3prl", feat_type)
|
||||
|
||||
if len(self.feature_extract.model.encoder.layers) == 24 and hasattr(
|
||||
self.feature_extract.model.encoder.layers[23].self_attn, "fp32_attention"
|
||||
|
||||
@@ -15,8 +15,7 @@ def main():
|
||||
|
||||
device = "cuda" if torch.cuda.is_available() else "xpu" if torch.xpu.is_available() else "cpu"
|
||||
|
||||
# TTS Audio Suite patch: restrict checkpoint deserialization.
|
||||
predictor = torch.hub.load("tarepan/SpeechMOS:v1.2.0", "utmos22_strong", trust_repo=True, weights_only=True)
|
||||
predictor = torch.hub.load("tarepan/SpeechMOS:v1.2.0", "utmos22_strong", trust_repo=True)
|
||||
predictor = predictor.to(device)
|
||||
|
||||
audio_paths = list(Path(args.audio_dir).rglob(f"*.{args.ext}"))
|
||||
|
||||
@@ -203,8 +203,7 @@ def convert_timm_dit(args, mapping, dtype="float32"):
|
||||
torch_dtype = str_dtype_to_torch(dtype)
|
||||
tensor_parallel = mapping.tp_size
|
||||
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
model_params = dict(torch.load(args.timm_ckpt, weights_only=True))
|
||||
model_params = dict(torch.load(args.timm_ckpt))
|
||||
model_params = {
|
||||
k: v for k, v in model_params["ema_model_state_dict"].items() if k.startswith("ema_model.transformer")
|
||||
}
|
||||
|
||||
@@ -968,8 +968,7 @@ def expand_model_embeddings(ckpt_path, new_ckpt_path, num_new_tokens=42):
|
||||
ckpt = load_file(ckpt_path, device="cpu")
|
||||
ckpt = {"ema_model_state_dict": ckpt}
|
||||
elif ckpt_path.endswith(".pt"):
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=True)
|
||||
ckpt = torch.load(ckpt_path, map_location="cpu")
|
||||
|
||||
ema_sd = ckpt.get("ema_model_state_dict", {})
|
||||
embed_key_ema = "ema_model.transformer.text_embed.text_embed.weight"
|
||||
|
||||
@@ -8,8 +8,6 @@ import folder_paths
|
||||
from utils.downloads.unified_downloader import unified_downloader
|
||||
|
||||
|
||||
from utils.security.path_access import validate_model_paths, allowed_path
|
||||
|
||||
class FishAudioS2Downloader:
|
||||
VARIANTS = {
|
||||
"s2-pro": {
|
||||
@@ -93,7 +91,6 @@ class FishAudioS2Downloader:
|
||||
return "s2-pro"
|
||||
|
||||
@classmethod
|
||||
@validate_model_paths
|
||||
def resolve_model_path(cls, selection: str) -> str:
|
||||
"""Resolve local selections or download the selected official variant."""
|
||||
if str(selection).startswith("local:"):
|
||||
|
||||
@@ -20,8 +20,6 @@ from utils.models.extra_paths import get_preferred_download_path
|
||||
import folder_paths
|
||||
|
||||
|
||||
from utils.security.path_access import validate_model_paths, allowed_path
|
||||
|
||||
class GraniteASRDownloader:
|
||||
"""Downloader for Granite ASR models using unified download system."""
|
||||
|
||||
@@ -148,7 +146,6 @@ class GraniteASRDownloader:
|
||||
if verbose:
|
||||
print("✅ Granite model verification passed")
|
||||
|
||||
@validate_model_paths
|
||||
def resolve_model_path(self, model_identifier: str) -> str:
|
||||
if not model_identifier:
|
||||
model_identifier = "granite-speech-4.1-2b"
|
||||
|
||||
+1
-2
@@ -1,4 +1,3 @@
|
||||
from utils.security.data_loading import load_numpy_data # TTS Audio Suite patch: NumPy dictionaries must not execute pickle globals.
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
@@ -46,7 +45,7 @@ class DACFile:
|
||||
|
||||
@classmethod
|
||||
def load(cls, path):
|
||||
artifacts = load_numpy_data(path)[()]
|
||||
artifacts = np.load(path, allow_pickle=True)[()]
|
||||
codes = torch.from_numpy(artifacts["codes"].astype(int))
|
||||
if artifacts["metadata"].get("dac_version", None) not in SUPPORTED_VERSIONS:
|
||||
raise RuntimeError(f"Given file {path} can't be loaded with this version of descript-audio-codec.")
|
||||
|
||||
@@ -357,8 +357,7 @@ def load_higgs_audio_tokenizer(tokenizer_name_or_path, device="cuda"):
|
||||
**config,
|
||||
device=device,
|
||||
)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
parameter_dict = torch.load(model_path, map_location=device, weights_only=True)
|
||||
parameter_dict = torch.load(model_path, map_location=device)
|
||||
model.load_state_dict(parameter_dict, strict=False)
|
||||
model.to(device)
|
||||
model.eval()
|
||||
|
||||
@@ -12,8 +12,6 @@ from utils.hf_download_logging import quiet_hf_download_logs
|
||||
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
|
||||
|
||||
|
||||
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
|
||||
|
||||
class HiggsAudioV3Downloader:
|
||||
"""Resolve and download official Higgs Audio v3 model folders."""
|
||||
|
||||
@@ -47,7 +45,6 @@ class HiggsAudioV3Downloader:
|
||||
self.base_path = base_path
|
||||
os.makedirs(self.base_path, exist_ok=True)
|
||||
|
||||
@validate_model_paths
|
||||
def resolve_model_path(self, model_identifier: str) -> str:
|
||||
"""Resolve local paths, local: names, or known Higgs v3 model identifiers."""
|
||||
model_identifier = model_identifier or self.MODEL_NAME
|
||||
@@ -104,7 +101,7 @@ class HiggsAudioV3Downloader:
|
||||
with quiet_hf_download_logs():
|
||||
snapshot_download(
|
||||
repo_id=repo_id,
|
||||
local_dir=allowed_path(model_dir),
|
||||
local_dir=model_dir,
|
||||
local_dir_use_symlinks=False,
|
||||
resume_download=True,
|
||||
force_download=force,
|
||||
|
||||
@@ -13,7 +13,7 @@ from utils.models.factory_config import ModelLoadConfig
|
||||
from utils.models.extra_paths import find_model_in_paths, get_preferred_download_path, get_all_tts_model_paths
|
||||
|
||||
|
||||
class IndexTTSEngine:
|
||||
class IndexTTSEngine:
|
||||
"""
|
||||
IndexTTS-2 Engine wrapper for TTS Audio Suite integration.
|
||||
|
||||
@@ -25,14 +25,7 @@ class IndexTTSEngine:
|
||||
- High-quality emotional expression
|
||||
"""
|
||||
|
||||
EMOTION_LABELS = ["happy", "angry", "sad", "afraid", "disgusted", "melancholic", "surprised", "calm"]
|
||||
LANGUAGE_CODES = {
|
||||
"zh": "ZH", "zh-cn": "ZH", "chinese": "ZH", "mandarin": "ZH",
|
||||
"en": "EN", "en-us": "EN", "en-gb": "EN", "english": "EN",
|
||||
"ja": "JA", "jp": "JA", "japanese": "JA",
|
||||
"es": "ES", "spanish": "ES",
|
||||
"ar": "AR", "arabic": "AR",
|
||||
}
|
||||
EMOTION_LABELS = ["happy", "angry", "sad", "afraid", "disgusted", "melancholic", "surprised", "calm"]
|
||||
|
||||
def __init__(self, model_dir: str = "IndexTTS-2", device: str = "auto",
|
||||
use_fp16: bool = True, use_cuda_kernel: Optional[bool] = None,
|
||||
@@ -51,10 +44,8 @@ class IndexTTSEngine:
|
||||
use_accel: Enable GPT2 acceleration with FlashAttention
|
||||
low_vram: Enable Low VRAM mode (sequential offloading)
|
||||
"""
|
||||
# Resolve model directory using extra_model_paths
|
||||
self.model_dir = self._find_model_directory(model_dir)
|
||||
self.model_name = os.path.basename(self.model_dir.rstrip("/\\")) or str(model_dir)
|
||||
self.model_version = "2.5" if os.path.isfile(os.path.join(self.model_dir, "codec.pth")) or "2.5" in self.model_name else "2"
|
||||
# Resolve model directory using extra_model_paths
|
||||
self.model_dir = self._find_model_directory(model_dir)
|
||||
|
||||
self.device = self._resolve_device(device)
|
||||
self.use_fp16 = use_fp16 and self.device != "cpu"
|
||||
@@ -143,10 +134,10 @@ class IndexTTSEngine:
|
||||
return
|
||||
|
||||
# Create model configuration
|
||||
self._model_config = ModelLoadConfig(
|
||||
self._model_config = ModelLoadConfig(
|
||||
engine_name="index_tts",
|
||||
model_type="tts",
|
||||
model_name=self.model_name,
|
||||
model_name="IndexTTS-2",
|
||||
device=self.device,
|
||||
model_path=self.model_dir,
|
||||
additional_params={
|
||||
@@ -155,8 +146,7 @@ class IndexTTSEngine:
|
||||
"use_deepspeed": self.use_deepspeed,
|
||||
"use_torch_compile": self.use_torch_compile,
|
||||
"use_accel": self.use_accel,
|
||||
"low_vram": self.low_vram,
|
||||
"model_version": self.model_version,
|
||||
"low_vram": self.low_vram
|
||||
}
|
||||
)
|
||||
|
||||
@@ -187,11 +177,8 @@ class IndexTTSEngine:
|
||||
length_penalty: float = 0.0,
|
||||
num_beams: int = 3,
|
||||
repetition_penalty: float = 10.0,
|
||||
max_mel_tokens: int = 1500,
|
||||
language: str = "EN",
|
||||
duration_factor: float = 1.0,
|
||||
text_normalization: bool = True,
|
||||
**kwargs
|
||||
max_mel_tokens: int = 1500,
|
||||
**kwargs
|
||||
) -> torch.Tensor:
|
||||
"""
|
||||
Generate speech using IndexTTS-2.
|
||||
@@ -214,10 +201,7 @@ class IndexTTSEngine:
|
||||
length_penalty: Length penalty for beam search
|
||||
num_beams: Number of beams for beam search
|
||||
repetition_penalty: Repetition penalty
|
||||
max_mel_tokens: Maximum mel tokens to generate
|
||||
language: IndexTTS-2.5 language code/name
|
||||
duration_factor: Official 2.5 internal feature-duration multiplier (0.5-2.0)
|
||||
text_normalization: Enable upstream multilingual text normalization
|
||||
max_mel_tokens: Maximum mel tokens to generate
|
||||
|
||||
Returns:
|
||||
Generated audio as torch.Tensor with shape [1, samples]
|
||||
@@ -351,8 +335,9 @@ class IndexTTSEngine:
|
||||
if unsupported_keys:
|
||||
print(f"⚠️ Filtering unsupported kwargs: {unsupported_keys}")
|
||||
|
||||
infer_kwargs = dict(
|
||||
spk_audio_prompt=speaker_audio,
|
||||
# Call IndexTTS-2 inference
|
||||
result = self._tts_engine.infer(
|
||||
spk_audio_prompt=speaker_audio,
|
||||
text=text,
|
||||
output_path=None,
|
||||
emo_audio_prompt=emotion_audio,
|
||||
@@ -371,49 +356,11 @@ class IndexTTSEngine:
|
||||
length_penalty=length_penalty,
|
||||
num_beams=num_beams,
|
||||
repetition_penalty=repetition_penalty,
|
||||
max_mel_tokens=max_mel_tokens,
|
||||
**supported_kwargs
|
||||
)
|
||||
if self.model_version == "2.5":
|
||||
language_key = str(language or "EN").strip().lower()
|
||||
language_code = self.LANGUAGE_CODES.get(language_key, str(language or "EN").upper())
|
||||
if language_code not in {"ZH", "EN", "JA", "ES", "AR"}:
|
||||
raise ValueError(
|
||||
f"Unsupported IndexTTS-2.5 language '{language}'. "
|
||||
"Choose Chinese, English, Japanese, Spanish, or Arabic."
|
||||
)
|
||||
duration_factor = float(duration_factor)
|
||||
if not 0.5 <= duration_factor <= 2.0:
|
||||
raise ValueError("IndexTTS-2.5 duration_factor must be between 0.5 and 2.0")
|
||||
infer_kwargs.update(
|
||||
lang=language_code,
|
||||
duration_factor=duration_factor,
|
||||
text_normalization=bool(text_normalization),
|
||||
)
|
||||
|
||||
# Call the selected IndexTTS backend.
|
||||
result = self._tts_engine.infer(**infer_kwargs)
|
||||
|
||||
if supported_kwargs.get("stream_return", False):
|
||||
# TTS Audio Suite patch: Normalize native streamed int16 chunks
|
||||
# instead of trying to unpack the generator as a final WAV tuple.
|
||||
def normalized_stream():
|
||||
for chunk in result:
|
||||
if not isinstance(chunk, torch.Tensor):
|
||||
continue
|
||||
chunk = chunk.detach().cpu()
|
||||
if chunk.dtype == torch.int16:
|
||||
chunk = chunk.float() / 32767.0
|
||||
else:
|
||||
chunk = chunk.float()
|
||||
if chunk.dim() == 1:
|
||||
chunk = chunk.unsqueeze(0)
|
||||
elif chunk.dim() > 2:
|
||||
chunk = chunk.reshape(-1, chunk.shape[-1]).mean(dim=0, keepdim=True)
|
||||
yield chunk
|
||||
return normalized_stream()
|
||||
|
||||
# Get audio tensor directly from infer result
|
||||
max_mel_tokens=max_mel_tokens,
|
||||
**supported_kwargs
|
||||
)
|
||||
|
||||
# Get audio tensor directly from infer result
|
||||
# infer() with output_path=None returns a tuple (sampling_rate, wav_data)
|
||||
# where wav_data is a numpy array of shape (samples, channels) in int16 format
|
||||
sampling_rate, wav_data = result
|
||||
|
||||
@@ -57,7 +57,7 @@ class IndexTTSDownloader:
|
||||
],
|
||||
"description": "CampPlus speaker embedding model for IndexTTS-2"
|
||||
},
|
||||
"IndexTTS-2": {
|
||||
"IndexTTS-2": {
|
||||
"repo_id": "IndexTeam/IndexTTS-2",
|
||||
"files": [
|
||||
"config.yaml",
|
||||
@@ -80,36 +80,8 @@ class IndexTTSDownloader:
|
||||
"qwen0.6bemo4-merge/tokenizer_config.json",
|
||||
"qwen0.6bemo4-merge/vocab.json"
|
||||
],
|
||||
"description": "IndexTTS-2 main model with emotion control"
|
||||
},
|
||||
"IndexTTS-2.5": {
|
||||
"repo_id": "IndexTeam/IndexTTS-2.5",
|
||||
# TTS Audio Suite patch: Pin the audited release snapshot because
|
||||
# the upstream repository is changing rapidly immediately post-release.
|
||||
"revision": "ba2480d9f7f629eb18f6acaebb357679d9ba88a4",
|
||||
"files": [
|
||||
"config.yaml",
|
||||
"codec.pth",
|
||||
"feat1.pt",
|
||||
"feat2.pt",
|
||||
"gpt.pth",
|
||||
"s2mel.pth",
|
||||
"multilingual_zh_ja_yue_char_del.tiktoken",
|
||||
"wav2vec2bert_stats.pt",
|
||||
"qwen0.6bemo4-merge/Modelfile",
|
||||
"qwen0.6bemo4-merge/added_tokens.json",
|
||||
"qwen0.6bemo4-merge/chat_template.jinja",
|
||||
"qwen0.6bemo4-merge/config.json",
|
||||
"qwen0.6bemo4-merge/generation_config.json",
|
||||
"qwen0.6bemo4-merge/merges.txt",
|
||||
"qwen0.6bemo4-merge/model.safetensors",
|
||||
"qwen0.6bemo4-merge/special_tokens_map.json",
|
||||
"qwen0.6bemo4-merge/tokenizer.json",
|
||||
"qwen0.6bemo4-merge/tokenizer_config.json",
|
||||
"qwen0.6bemo4-merge/vocab.json",
|
||||
],
|
||||
"description": "IndexTTS-2.5 multilingual model with official duration-factor scaling and emotion control",
|
||||
}
|
||||
"description": "IndexTTS-2 main model with emotion control"
|
||||
}
|
||||
}
|
||||
|
||||
def __init__(self, base_path: Optional[str] = None):
|
||||
@@ -180,14 +152,13 @@ class IndexTTSDownloader:
|
||||
})
|
||||
|
||||
# Download model files using unified downloader
|
||||
result_path = self.downloader.download_huggingface_model(
|
||||
repo_id=model_info["repo_id"],
|
||||
model_name=model_name,
|
||||
files=file_list,
|
||||
engine_type="IndexTTS",
|
||||
revision=model_info.get("revision"),
|
||||
**kwargs
|
||||
)
|
||||
result_path = self.downloader.download_huggingface_model(
|
||||
repo_id=model_info["repo_id"],
|
||||
model_name=model_name,
|
||||
files=file_list,
|
||||
engine_type="IndexTTS",
|
||||
**kwargs
|
||||
)
|
||||
|
||||
if not result_path:
|
||||
raise RuntimeError("HuggingFace download failed")
|
||||
@@ -320,4 +291,4 @@ def download_index_tts_model(model_name: str = "IndexTTS-2",
|
||||
|
||||
def is_index_tts_available(model_name: str = "IndexTTS-2") -> bool:
|
||||
"""Check if IndexTTS-2 model is available locally."""
|
||||
return index_tts_downloader.is_model_available(model_name)
|
||||
return index_tts_downloader.is_model_available(model_name)
|
||||
@@ -520,8 +520,7 @@ class BigVGAN(
|
||||
local_files_only=local_files_only,
|
||||
)
|
||||
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint_dict = torch.load(model_file, map_location=map_location, weights_only=True)
|
||||
checkpoint_dict = torch.load(model_file, map_location=map_location)
|
||||
|
||||
try:
|
||||
model.load_state_dict(checkpoint_dict["generator"])
|
||||
|
||||
@@ -63,8 +63,7 @@ def get_padding(kernel_size, dilation=1):
|
||||
def load_checkpoint(filepath, device):
|
||||
assert os.path.isfile(filepath)
|
||||
print(f"Loading '{filepath}'")
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint_dict = torch.load(filepath, map_location=device, weights_only=True)
|
||||
checkpoint_dict = torch.load(filepath, map_location=device)
|
||||
print("Complete.")
|
||||
return checkpoint_dict
|
||||
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
# TTS Audio Suite patch: Package marker for the bundled official IndexTTS 2.5 semantic codec.
|
||||
@@ -1 +0,0 @@
|
||||
# TTS Audio Suite patch: Package marker for the bundled IndexTTS 2.5 codec quantizers.
|
||||
@@ -1,14 +0,0 @@
|
||||
# TTS Audio Suite patch: Bundled from the official IndexTTS 2.5 runtime to avoid upstream dependency pins conflicting with ComfyUI.
|
||||
# Copyright (c) 2024 Amphion.
|
||||
#
|
||||
# This source code is licensed under the MIT license found in the
|
||||
# LICENSE file in the root directory of this source tree.
|
||||
|
||||
from indextts.codec.amphion_codec.quantize.factorized_vector_quantize import (
|
||||
FactorizedVectorQuantize,
|
||||
)
|
||||
from indextts.codec.amphion_codec.quantize.vector_quantize import VectorQuantize
|
||||
from indextts.codec.amphion_codec.quantize.lookup_free_quantize import LookupFreeQuantize
|
||||
from indextts.codec.amphion_codec.quantize.residual_vq import ResidualVQ
|
||||
|
||||
|
||||
-153
@@ -1,153 +0,0 @@
|
||||
# TTS Audio Suite patch: Bundled from the official IndexTTS 2.5 runtime to avoid upstream dependency pins conflicting with ComfyUI.
|
||||
# Copyright (c) 2024 Amphion.
|
||||
#
|
||||
# This source code is licensed under the MIT license found in the
|
||||
# LICENSE file in the root directory of this source tree.
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
from einops import rearrange
|
||||
from torch.nn.utils import weight_norm
|
||||
|
||||
|
||||
def WNConv1d(*args, **kwargs):
|
||||
return weight_norm(nn.Conv1d(*args, **kwargs))
|
||||
|
||||
|
||||
def WNConvTranspose1d(*args, **kwargs):
|
||||
return weight_norm(nn.ConvTranspose1d(*args, **kwargs))
|
||||
|
||||
|
||||
class FactorizedVectorQuantize(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
input_dim,
|
||||
codebook_size,
|
||||
codebook_dim,
|
||||
commitment=0.005,
|
||||
codebook_loss_weight=1.0,
|
||||
use_l2_normlize=True,
|
||||
):
|
||||
super().__init__()
|
||||
self.input_dim = input_dim
|
||||
self.codebook_size = codebook_size
|
||||
self.codebook_dim = codebook_dim
|
||||
self.commitment = commitment
|
||||
self.codebook_loss_weight = codebook_loss_weight
|
||||
self.use_l2_normlize = use_l2_normlize
|
||||
|
||||
if self.input_dim != self.codebook_dim:
|
||||
self.in_project = WNConv1d(self.input_dim, self.codebook_dim, kernel_size=1)
|
||||
self.out_project = WNConv1d(
|
||||
self.codebook_dim, self.input_dim, kernel_size=1
|
||||
)
|
||||
|
||||
else:
|
||||
self.in_project = nn.Identity()
|
||||
self.out_project = nn.Identity()
|
||||
|
||||
self.codebook = nn.Embedding(self.codebook_size, self.codebook_dim)
|
||||
|
||||
def forward(self, z):
|
||||
"""
|
||||
Parameters
|
||||
----------
|
||||
z: torch.Tensor[B x D x T]
|
||||
|
||||
Returns
|
||||
-------
|
||||
z_q: torch.Tensor[B x D x T]
|
||||
Quantized continuous representation of input
|
||||
commit_loss: Tensor[B]
|
||||
Commitment loss to train encoder to predict vectors closer to codebook entries
|
||||
codebook_loss: Tensor[B]
|
||||
Codebook loss to update the codebook
|
||||
indices: torch.Tensor[B x T]
|
||||
Codebook indices (quantized discrete representation of input)
|
||||
z_e: torch.Tensor[B x D x T]
|
||||
Projected latents (continuous representation of input before quantization)
|
||||
"""
|
||||
|
||||
# Factorized codes project input into low-dimensional space if self.input_dim != self.codebook_dim
|
||||
z_e = self.in_project(z)
|
||||
z_q, indices = self.decode_latents(z_e)
|
||||
|
||||
# Compute commitment loss and codebook loss
|
||||
if self.training:
|
||||
commit_loss = (
|
||||
F.mse_loss(z_e, z_q.detach(), reduction="none").mean([1, 2])
|
||||
* self.commitment
|
||||
)
|
||||
codebook_loss = (
|
||||
F.mse_loss(z_q, z_e.detach(), reduction="none").mean([1, 2])
|
||||
* self.codebook_loss_weight
|
||||
)
|
||||
else:
|
||||
commit_loss = torch.zeros(z.shape[0], device=z.device)
|
||||
codebook_loss = torch.zeros(z.shape[0], device=z.device)
|
||||
|
||||
z_q = z_e + (z_q - z_e).detach()
|
||||
|
||||
z_q = self.out_project(z_q)
|
||||
|
||||
return z_q, commit_loss, codebook_loss, indices, z_e
|
||||
|
||||
def embed_code(self, embed_id):
|
||||
return F.embedding(embed_id, self.codebook.weight)
|
||||
|
||||
def decode_code(self, embed_id):
|
||||
return self.embed_code(embed_id).transpose(1, 2)
|
||||
|
||||
def decode_latents(self, latents):
|
||||
encodings = rearrange(latents, "b d t -> (b t) d")
|
||||
codebook = self.codebook.weight
|
||||
|
||||
# L2 normalize encodings and codebook
|
||||
if self.use_l2_normlize:
|
||||
encodings = F.normalize(encodings)
|
||||
codebook = F.normalize(codebook)
|
||||
|
||||
# Compute euclidean distance between encodings and codebook,
|
||||
# if use_l2_normlize is True, the distance is equal to cosine distance
|
||||
dist = (
|
||||
encodings.pow(2).sum(1, keepdim=True)
|
||||
- 2 * encodings @ codebook.t()
|
||||
+ codebook.pow(2).sum(1, keepdim=True).t()
|
||||
)
|
||||
indices = rearrange((-dist).max(1)[1], "(b t) -> b t", b=latents.size(0))
|
||||
z_q = self.decode_code(indices)
|
||||
|
||||
return z_q, indices
|
||||
|
||||
def vq2emb(self, vq, out_proj=True):
|
||||
emb = self.decode_code(vq)
|
||||
if out_proj:
|
||||
emb = self.out_project(emb)
|
||||
return emb
|
||||
|
||||
def latent2dist(self, latents):
|
||||
encodings = rearrange(latents, "b d t -> (b t) d")
|
||||
codebook = self.codebook.weight
|
||||
|
||||
# L2 normalize encodings and codebook
|
||||
if self.use_l2_normlize:
|
||||
encodings = F.normalize(encodings)
|
||||
codebook = F.normalize(codebook)
|
||||
|
||||
# Compute euclidean distance between encodings and codebook,
|
||||
# if use_l2_normlize is True, the distance is equal to cosine distance
|
||||
dist = (
|
||||
encodings.pow(2).sum(1, keepdim=True)
|
||||
- 2 * encodings @ codebook.t()
|
||||
+ codebook.pow(2).sum(1, keepdim=True).t()
|
||||
) # (b*t, k)
|
||||
|
||||
indices = rearrange((-dist).max(1)[1], "(b t) -> b t", b=latents.size(0))
|
||||
dist = rearrange(dist, "(b t) k -> b t k", b=latents.size(0))
|
||||
z_q = self.decode_code(indices)
|
||||
|
||||
return -dist, indices, z_q
|
||||
|
||||
|
||||
@@ -1,80 +0,0 @@
|
||||
# TTS Audio Suite patch: Bundled from the official IndexTTS 2.5 runtime to avoid upstream dependency pins conflicting with ComfyUI.
|
||||
# Copyright (c) 2024 Amphion.
|
||||
#
|
||||
# This source code is licensed under the MIT license found in the
|
||||
# LICENSE file in the root directory of this source tree.
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
from einops import rearrange
|
||||
from torch.nn.utils import weight_norm
|
||||
|
||||
|
||||
def WNConv1d(*args, **kwargs):
|
||||
return weight_norm(nn.Conv1d(*args, **kwargs))
|
||||
|
||||
|
||||
def WNConvTranspose1d(*args, **kwargs):
|
||||
return weight_norm(nn.ConvTranspose1d(*args, **kwargs))
|
||||
|
||||
|
||||
class LookupFreeQuantize(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
input_dim,
|
||||
codebook_size,
|
||||
codebook_dim,
|
||||
):
|
||||
super().__init__()
|
||||
self.input_dim = input_dim
|
||||
self.codebook_size = codebook_size
|
||||
self.codebook_dim = codebook_dim
|
||||
|
||||
assert 2**codebook_dim == codebook_size
|
||||
|
||||
if self.input_dim != self.codebook_dim:
|
||||
self.in_project = WNConv1d(self.input_dim, self.codebook_dim, kernel_size=1)
|
||||
self.out_project = WNConv1d(
|
||||
self.codebook_dim, self.input_dim, kernel_size=1
|
||||
)
|
||||
|
||||
else:
|
||||
self.in_project = nn.Identity()
|
||||
self.out_project = nn.Identity()
|
||||
|
||||
def forward(self, z):
|
||||
z_e = self.in_project(z)
|
||||
z_e = F.sigmoid(z_e)
|
||||
|
||||
z_q = z_e + (torch.round(z_e) - z_e).detach()
|
||||
|
||||
z_q = self.out_project(z_q)
|
||||
|
||||
commit_loss = torch.zeros(z.shape[0], device=z.device)
|
||||
codebook_loss = torch.zeros(z.shape[0], device=z.device)
|
||||
|
||||
bits = (
|
||||
2
|
||||
** torch.arange(self.codebook_dim, device=z.device)
|
||||
.unsqueeze(0)
|
||||
.unsqueeze(-1)
|
||||
.long()
|
||||
) # (1, d, 1)
|
||||
indices = (torch.round(z_e.clone().detach()).long() * bits).sum(1).long()
|
||||
|
||||
return z_q, commit_loss, codebook_loss, indices, z_e
|
||||
|
||||
def vq2emb(self, vq, out_proj=True):
|
||||
emb = torch.zeros(
|
||||
vq.shape[0], self.codebook_dim, vq.shape[-1], device=vq.device
|
||||
) # (B, d, T)
|
||||
for i in range(self.codebook_dim):
|
||||
emb[:, i, :] = (vq % 2).float()
|
||||
vq = vq // 2
|
||||
if out_proj:
|
||||
emb = self.out_project(emb)
|
||||
return emb
|
||||
|
||||
|
||||
@@ -1,180 +0,0 @@
|
||||
# TTS Audio Suite patch: Bundled from the official IndexTTS 2.5 runtime to avoid upstream dependency pins conflicting with ComfyUI.
|
||||
# Copyright (c) 2024 Amphion.
|
||||
#
|
||||
# This source code is licensed under the MIT license found in the
|
||||
# LICENSE file in the root directory of this source tree.
|
||||
|
||||
from typing import Union
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
from einops import rearrange
|
||||
from torch.nn.utils import weight_norm
|
||||
|
||||
from indextts.codec.amphion_codec.quantize.factorized_vector_quantize import (
|
||||
FactorizedVectorQuantize,
|
||||
)
|
||||
from indextts.codec.amphion_codec.quantize.vector_quantize import VectorQuantize
|
||||
from indextts.codec.amphion_codec.quantize.lookup_free_quantize import LookupFreeQuantize
|
||||
|
||||
|
||||
class ResidualVQ(nn.Module):
|
||||
"""
|
||||
Introduced in SoundStream: An end2end neural audio codec
|
||||
https://arxiv.org/abs/2107.03312
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
input_dim: int = 256,
|
||||
num_quantizers: int = 8,
|
||||
codebook_size: int = 1024,
|
||||
codebook_dim: int = 256,
|
||||
quantizer_type: str = "vq", # "vq" or "fvq" or "lfq"
|
||||
quantizer_dropout: float = 0.5,
|
||||
**kwargs,
|
||||
):
|
||||
super().__init__()
|
||||
|
||||
self.input_dim = input_dim
|
||||
self.num_quantizers = num_quantizers
|
||||
self.codebook_size = codebook_size
|
||||
self.codebook_dim = codebook_dim
|
||||
self.quantizer_type = quantizer_type
|
||||
self.quantizer_dropout = quantizer_dropout
|
||||
|
||||
if quantizer_type == "vq":
|
||||
VQ = VectorQuantize
|
||||
elif quantizer_type == "fvq":
|
||||
VQ = FactorizedVectorQuantize
|
||||
elif quantizer_type == "lfq":
|
||||
VQ = LookupFreeQuantize
|
||||
else:
|
||||
raise ValueError(f"Unknown quantizer type {quantizer_type}")
|
||||
|
||||
self.quantizers = nn.ModuleList(
|
||||
[
|
||||
VQ(
|
||||
input_dim=input_dim,
|
||||
codebook_size=codebook_size,
|
||||
codebook_dim=codebook_dim,
|
||||
**kwargs,
|
||||
)
|
||||
for _ in range(num_quantizers)
|
||||
]
|
||||
)
|
||||
|
||||
def forward(self, z, n_quantizers: int = None):
|
||||
"""
|
||||
Parameters
|
||||
----------
|
||||
z : Tensor[B x D x T]
|
||||
n_quantizers : int, optional
|
||||
No. of quantizers to use
|
||||
(n_quantizers < self.n_codebooks ex: for quantizer dropout)
|
||||
Note: if `self.quantizer_dropout` is True, this argument is ignored
|
||||
when in training mode, and a random number of quantizers is used.
|
||||
Returns
|
||||
-------
|
||||
"quantized_out" : Tensor[B x D x T]
|
||||
Quantized continuous representation of input
|
||||
"all_indices" : Tensor[N x B x T]
|
||||
Codebook indices for each codebook
|
||||
(quantized discrete representation of input)
|
||||
"all_commit_losses" : Tensor[N]
|
||||
"all_codebook_losses" : Tensor[N]
|
||||
"all_quantized" : Tensor[N x B x D x T]
|
||||
"""
|
||||
|
||||
quantized_out = 0.0
|
||||
residual = z
|
||||
|
||||
all_commit_losses = []
|
||||
all_codebook_losses = []
|
||||
all_indices = []
|
||||
all_quantized = []
|
||||
|
||||
if n_quantizers is None:
|
||||
n_quantizers = self.num_quantizers
|
||||
|
||||
if self.training:
|
||||
n_quantizers = torch.ones((z.shape[0],)) * self.num_quantizers + 1
|
||||
dropout = torch.randint(1, self.num_quantizers + 1, (z.shape[0],))
|
||||
n_dropout = int(z.shape[0] * self.quantizer_dropout)
|
||||
n_quantizers[:n_dropout] = dropout[:n_dropout]
|
||||
n_quantizers = n_quantizers.to(z.device)
|
||||
|
||||
for i, quantizer in enumerate(self.quantizers):
|
||||
if self.training is False and i >= n_quantizers:
|
||||
break
|
||||
|
||||
z_q_i, commit_loss_i, codebook_loss_i, indices_i, z_e_i = quantizer(
|
||||
residual
|
||||
)
|
||||
|
||||
# Create mask to apply quantizer dropout
|
||||
mask = (
|
||||
torch.full((z.shape[0],), fill_value=i, device=z.device) < n_quantizers
|
||||
)
|
||||
quantized_out = quantized_out + z_q_i * mask[:, None, None]
|
||||
residual = residual - z_q_i
|
||||
|
||||
commit_loss_i = (commit_loss_i * mask).mean()
|
||||
codebook_loss_i = (codebook_loss_i * mask).mean()
|
||||
|
||||
all_commit_losses.append(commit_loss_i)
|
||||
all_codebook_losses.append(codebook_loss_i)
|
||||
all_indices.append(indices_i)
|
||||
all_quantized.append(z_q_i)
|
||||
|
||||
all_commit_losses, all_codebook_losses, all_indices, all_quantized = map(
|
||||
torch.stack,
|
||||
(all_commit_losses, all_codebook_losses, all_indices, all_quantized),
|
||||
)
|
||||
|
||||
return (
|
||||
quantized_out,
|
||||
all_indices,
|
||||
all_commit_losses,
|
||||
all_codebook_losses,
|
||||
all_quantized,
|
||||
)
|
||||
|
||||
def vq2emb(self, vq, n_quantizers=None):
|
||||
quantized_out = 0.0
|
||||
if n_quantizers is None:
|
||||
n_quantizers = self.num_quantizers
|
||||
for idx, quantizer in enumerate(self.quantizers):
|
||||
if idx >= n_quantizers:
|
||||
break
|
||||
quantized_out += quantizer.vq2emb(vq[idx])
|
||||
return quantized_out
|
||||
|
||||
def latent2dist(self, z, n_quantizers=None):
|
||||
quantized_out = 0.0
|
||||
residual = z
|
||||
|
||||
all_dists = []
|
||||
all_indices = []
|
||||
|
||||
if n_quantizers is None:
|
||||
n_quantizers = self.num_quantizers
|
||||
|
||||
for i, quantizer in enumerate(self.quantizers):
|
||||
if self.training is False and i >= n_quantizers:
|
||||
break
|
||||
dist_i, indices_i, z_q_i = quantizer.latent2dist(residual)
|
||||
all_dists.append(dist_i)
|
||||
all_indices.append(indices_i)
|
||||
|
||||
quantized_out = quantized_out + z_q_i
|
||||
residual = residual - z_q_i
|
||||
|
||||
all_dists = torch.stack(all_dists)
|
||||
all_indices = torch.stack(all_indices)
|
||||
|
||||
return all_dists, all_indices
|
||||
|
||||
|
||||
@@ -1,404 +0,0 @@
|
||||
# TTS Audio Suite patch: Bundled from the official IndexTTS 2.5 runtime to avoid upstream dependency pins conflicting with ComfyUI.
|
||||
# Copyright (c) 2024 Amphion.
|
||||
#
|
||||
# This source code is licensed under the MIT license found in the
|
||||
# LICENSE file in the root directory of this source tree.
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
from einops import rearrange, repeat
|
||||
from torch.nn.utils import weight_norm
|
||||
|
||||
|
||||
def WNConv1d(*args, **kwargs):
|
||||
return weight_norm(nn.Conv1d(*args, **kwargs))
|
||||
|
||||
|
||||
def WNConvTranspose1d(*args, **kwargs):
|
||||
return weight_norm(nn.ConvTranspose1d(*args, **kwargs))
|
||||
|
||||
|
||||
def l2norm(t):
|
||||
return F.normalize(t, p=2, dim=-1)
|
||||
|
||||
|
||||
def ema_inplace(moving_avg, new, decay):
|
||||
moving_avg.data.mul_(decay).add_(new, alpha=(1 - decay))
|
||||
|
||||
|
||||
def laplace_smoothing(x, n_categories, eps=1e-5):
|
||||
return (x + eps) / (x.sum() + n_categories * eps)
|
||||
|
||||
|
||||
def sample_vectors(samples, num):
|
||||
num_samples, device = samples.shape[0], samples.device
|
||||
|
||||
if num_samples >= num:
|
||||
indices = torch.randperm(num_samples, device=device)[:num]
|
||||
else:
|
||||
indices = torch.randint(0, num_samples, (num,), device=device)
|
||||
|
||||
return samples[indices]
|
||||
|
||||
|
||||
def kmeans(samples, num_clusters, num_iters=10, use_cosine_sim=False):
|
||||
dim, dtype, device = samples.shape[-1], samples.dtype, samples.device
|
||||
|
||||
means = sample_vectors(samples, num_clusters)
|
||||
|
||||
for _ in range(num_iters):
|
||||
if use_cosine_sim:
|
||||
dists = samples @ means.t()
|
||||
else:
|
||||
diffs = rearrange(samples, "n d -> n () d") - rearrange(
|
||||
means, "c d -> () c d"
|
||||
)
|
||||
dists = -(diffs**2).sum(dim=-1)
|
||||
|
||||
buckets = dists.max(dim=-1).indices
|
||||
bins = torch.bincount(buckets, minlength=num_clusters)
|
||||
zero_mask = bins == 0
|
||||
bins_min_clamped = bins.masked_fill(zero_mask, 1)
|
||||
|
||||
new_means = buckets.new_zeros(num_clusters, dim, dtype=dtype)
|
||||
new_means.scatter_add_(0, repeat(buckets, "n -> n d", d=dim), samples)
|
||||
new_means = new_means / bins_min_clamped[..., None]
|
||||
|
||||
if use_cosine_sim:
|
||||
new_means = l2norm(new_means)
|
||||
|
||||
means = torch.where(zero_mask[..., None], means, new_means)
|
||||
|
||||
return means, bins
|
||||
|
||||
|
||||
class EuclideanCodebook(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
dim,
|
||||
codebook_size,
|
||||
kmeans_init=False,
|
||||
kmeans_iters=10,
|
||||
decay=0.8,
|
||||
eps=1e-5,
|
||||
threshold_ema_dead_code=2,
|
||||
weight_init=False,
|
||||
):
|
||||
super().__init__()
|
||||
|
||||
self.decay = decay
|
||||
init_fn = torch.randn if not weight_init else torch.zeros
|
||||
embed = init_fn(codebook_size, dim)
|
||||
|
||||
if weight_init:
|
||||
nn.init.uniform_(embed, -1 / codebook_size, 1 / codebook_size)
|
||||
|
||||
self.codebook_size = codebook_size
|
||||
self.kmeans_iters = kmeans_iters
|
||||
self.eps = eps
|
||||
self.threshold_ema_dead_code = threshold_ema_dead_code
|
||||
|
||||
self.register_buffer(
|
||||
"initted", torch.Tensor([not kmeans_init])
|
||||
) # if kmeans_init is True, then initted is False; otherwise, initted is True
|
||||
self.register_buffer("cluster_size", torch.zeros(codebook_size))
|
||||
self.register_buffer("embed", embed)
|
||||
self.register_buffer("embed_avg", embed.clone())
|
||||
|
||||
def init_embed_(self, data):
|
||||
embed, cluster_size = kmeans(data, self.codebook_size, self.kmeans_iters)
|
||||
self.embed.data.copy_(embed)
|
||||
self.embed_avg.data.copy_(embed)
|
||||
self.cluster_size.data.copy_(cluster_size)
|
||||
self.initted.data.copy_(torch.Tensor([True]))
|
||||
|
||||
def replace(self, samples, mask):
|
||||
modified_codebook = torch.where(
|
||||
mask[..., None], sample_vectors(samples, self.codebook_size), self.embed
|
||||
)
|
||||
self.embed.data.copy_(modified_codebook)
|
||||
|
||||
def expire_codes_(self, batch_samples):
|
||||
if self.threshold_ema_dead_code == 0:
|
||||
return
|
||||
|
||||
expired_codes = self.cluster_size < self.threshold_ema_dead_code
|
||||
if not torch.any(expired_codes):
|
||||
return
|
||||
batch_samples = rearrange(batch_samples, "... d -> (...) d")
|
||||
self.replace(batch_samples, mask=expired_codes)
|
||||
|
||||
def forward(self, x):
|
||||
shape, dtype = x.shape, x.dtype
|
||||
flatten = rearrange(x, "... d -> (...) d")
|
||||
embed = self.embed.t() # (codebook_size, dim) -> (dim, codebook_size)
|
||||
|
||||
if not self.initted:
|
||||
self.init_embed_(flatten)
|
||||
|
||||
dist = -(
|
||||
flatten.pow(2).sum(1, keepdim=True)
|
||||
- 2 * flatten @ embed
|
||||
+ embed.pow(2).sum(0, keepdim=True)
|
||||
)
|
||||
|
||||
embed_ind = dist.max(dim=-1).indices
|
||||
embed_onehot = F.one_hot(embed_ind, self.codebook_size).type(dtype)
|
||||
embed_ind = embed_ind.view(*shape[:-1])
|
||||
quantize = F.embedding(embed_ind, self.embed)
|
||||
|
||||
if self.training:
|
||||
ema_inplace(self.cluster_size, embed_onehot.sum(0), self.decay)
|
||||
embed_sum = (
|
||||
flatten.t() @ embed_onehot
|
||||
) # (dim, ...) @ (..., codebook_size) -> (dim, codebook_size)
|
||||
ema_inplace(self.embed_avg, embed_sum.t(), self.decay)
|
||||
cluster_size = (
|
||||
laplace_smoothing(self.cluster_size, self.codebook_size, self.eps)
|
||||
* self.cluster_size.sum()
|
||||
)
|
||||
embed_normalized = self.embed_avg / cluster_size.unsqueeze(1)
|
||||
self.embed.data.copy_(embed_normalized)
|
||||
self.expire_codes_(x)
|
||||
|
||||
return quantize, embed_ind
|
||||
|
||||
def vq2emb(self, vq):
|
||||
quantize = F.embedding(vq, self.embed)
|
||||
return quantize
|
||||
|
||||
def latent2dist(self, x):
|
||||
shape, dtype = x.shape, x.dtype
|
||||
flatten = rearrange(x, "... d -> (...) d")
|
||||
embed = self.embed.t() # (codebook_size, dim) -> (dim, codebook_size)
|
||||
|
||||
if not self.initted:
|
||||
self.init_embed_(flatten)
|
||||
|
||||
dist = -(
|
||||
flatten.pow(2).sum(1, keepdim=True)
|
||||
- 2 * flatten @ embed
|
||||
+ embed.pow(2).sum(0, keepdim=True)
|
||||
)
|
||||
|
||||
embed_ind = dist.max(dim=-1).indices
|
||||
embed_ind = embed_ind.view(*shape[:-1])
|
||||
quantize = F.embedding(embed_ind, self.embed)
|
||||
|
||||
dist = dist.view(*shape[:-1], -1)
|
||||
|
||||
return dist, embed_ind, quantize
|
||||
|
||||
|
||||
class SimpleCodebook(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
dim,
|
||||
codebook_size,
|
||||
use_l2_normlize=False,
|
||||
):
|
||||
super().__init__()
|
||||
|
||||
self.dim = dim
|
||||
self.codebook_size = codebook_size
|
||||
self.use_l2_normlize = use_l2_normlize
|
||||
|
||||
self.embed = nn.Embedding(self.codebook_size, self.dim)
|
||||
|
||||
def forward(self, x):
|
||||
shape, dtype = x.shape, x.dtype
|
||||
flatten = rearrange(x, "... d -> (...) d")
|
||||
embed = self.embed.weight.t() # (codebook_size, dim) -> (dim, codebook_size)
|
||||
|
||||
if self.use_l2_normlize:
|
||||
flatten = F.normalize(flatten)
|
||||
embed = F.normalize(embed)
|
||||
|
||||
dist = -(
|
||||
flatten.pow(2).sum(1, keepdim=True)
|
||||
- 2 * flatten @ embed
|
||||
+ embed.pow(2).sum(0, keepdim=True)
|
||||
)
|
||||
|
||||
embed_ind = dist.max(dim=-1).indices
|
||||
embed_ind = embed_ind.view(*shape[:-1])
|
||||
quantize = F.embedding(embed_ind, self.embed)
|
||||
|
||||
return quantize, embed_ind
|
||||
|
||||
def vq2emb(self, vq):
|
||||
quantize = F.embedding(vq, self.embed.weight)
|
||||
return quantize
|
||||
|
||||
def latent2dist(self, x):
|
||||
shape, dtype = x.shape, x.dtype
|
||||
flatten = rearrange(x, "... d -> (...) d")
|
||||
embed = self.embed.weight.t() # (codebook_size, dim) -> (dim, codebook_size)
|
||||
|
||||
if self.use_l2_normlize:
|
||||
flatten = F.normalize(flatten)
|
||||
embed = F.normalize(embed)
|
||||
|
||||
dist = -(
|
||||
flatten.pow(2).sum(1, keepdim=True)
|
||||
- 2 * flatten @ embed
|
||||
+ embed.pow(2).sum(0, keepdim=True)
|
||||
)
|
||||
|
||||
embed_ind = dist.max(dim=-1).indices
|
||||
embed_ind = embed_ind.view(*shape[:-1])
|
||||
quantize = F.embedding(embed_ind, self.embed)
|
||||
|
||||
dist = dist.view(*shape[:-1], -1)
|
||||
|
||||
return dist, embed_ind, quantize
|
||||
|
||||
|
||||
class VectorQuantize(nn.Module):
|
||||
"""Vector quantization and factorized vecotor quantization implementation
|
||||
Args:
|
||||
input_dim (int): Dimension of input.
|
||||
codebook_size (int): Codebook size.
|
||||
codebook_dim (int): Codebook dimension. We suggest use codebook_dim = input_dim
|
||||
if use codebook_type == "euclidean", otherwise, if you want to use
|
||||
factorized vector quantization, use codebook_dim as small number (e.g. 8 or 32).
|
||||
commitment (float): Weight for commitment loss.
|
||||
use_l2_normlize (bool): Whether to use l2 normlized codes for factorized vecotor quantization,
|
||||
we suggest use it as True if you want to use factorized vector quantization
|
||||
kmeans_init (bool): Whether to use kmeans to initialize the codebooks.
|
||||
kmeans_iters (int): Number of iterations used for kmeans initialization.
|
||||
decay (float): Decay for exponential moving average over the codebooks.
|
||||
epsilon (float): Epsilon value for numerical stability.
|
||||
threshold_ema_dead_code (int): Threshold for dead code expiration. Replace any codes
|
||||
that have an exponential moving average cluster size less than the specified threshold with
|
||||
randomly selected vector from the current batch.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
input_dim,
|
||||
codebook_size,
|
||||
codebook_dim,
|
||||
commitment=0.005,
|
||||
codebook_loss_weight=1.0,
|
||||
use_l2_normlize=False,
|
||||
codebook_type="euclidean", # "euclidean" or "simple"
|
||||
kmeans_init=False,
|
||||
kmeans_iters=10,
|
||||
decay=0.8,
|
||||
eps=1e-5,
|
||||
threshold_ema_dead_code=2,
|
||||
weight_init=False,
|
||||
):
|
||||
super().__init__()
|
||||
self.input_dim = input_dim
|
||||
self.codebook_size = codebook_size
|
||||
self.codebook_dim = codebook_dim
|
||||
self.commitment = commitment
|
||||
self.codebook_loss_weight = codebook_loss_weight
|
||||
self.use_l2_normlize = use_l2_normlize
|
||||
self.codebook_type = codebook_type
|
||||
self.kmeans_init = kmeans_init
|
||||
self.kmeans_iters = kmeans_iters
|
||||
self.decay = decay
|
||||
self.eps = eps
|
||||
self.threshold_ema_dead_code = threshold_ema_dead_code
|
||||
self.weight_init = weight_init
|
||||
|
||||
if self.input_dim != self.codebook_dim:
|
||||
self.in_project = WNConv1d(self.input_dim, self.codebook_dim, kernel_size=1)
|
||||
self.out_project = WNConv1d(
|
||||
self.codebook_dim, self.input_dim, kernel_size=1
|
||||
)
|
||||
|
||||
else:
|
||||
self.in_project = nn.Identity()
|
||||
self.out_project = nn.Identity()
|
||||
|
||||
if self.codebook_type == "euclidean":
|
||||
self.codebook = EuclideanCodebook(
|
||||
self.codebook_dim,
|
||||
codebook_size=self.codebook_size,
|
||||
kmeans_init=self.kmeans_init,
|
||||
kmeans_iters=self.kmeans_iters,
|
||||
decay=self.decay,
|
||||
eps=self.eps,
|
||||
threshold_ema_dead_code=self.threshold_ema_dead_code,
|
||||
weight_init=self.weight_init,
|
||||
)
|
||||
elif self.codebook_type == "simple":
|
||||
self.codebook = SimpleCodebook(
|
||||
self.codebook_dim,
|
||||
codebook_size=self.codebook_size,
|
||||
use_l2_normlize=self.use_l2_normlize,
|
||||
)
|
||||
else:
|
||||
raise NotImplementedError(
|
||||
f"codebook_type {self.codebook_type} is not implemented!"
|
||||
)
|
||||
|
||||
def forward(self, z):
|
||||
"""
|
||||
Parameters
|
||||
----------
|
||||
z: torch.Tensor[B x D x T]
|
||||
|
||||
Returns
|
||||
-------
|
||||
z_q: torch.Tensor[B x D x T]
|
||||
Quantized continuous representation of input
|
||||
commit_loss: Tensor[B]
|
||||
Commitment loss to train encoder to predict vectors closer to codebook entries
|
||||
codebook_loss: Tensor[B]
|
||||
Codebook loss to update the codebook
|
||||
indices: torch.Tensor[B x T]
|
||||
Codebook indices (quantized discrete representation of input)
|
||||
z_e: torch.Tensor[B x D x T]
|
||||
Projected latents (continuous representation of input before quantization)
|
||||
"""
|
||||
|
||||
# Factorized codes project input into low-dimensional space if self.input_dim != self.codebook_dim
|
||||
z_e = self.in_project(z)
|
||||
z_q, indices = self.decode_latents(z_e)
|
||||
|
||||
# Compute commitment loss and codebook loss
|
||||
if self.training:
|
||||
commit_loss = (
|
||||
F.mse_loss(z_e, z_q.detach(), reduction="none").mean([1, 2])
|
||||
* self.commitment
|
||||
)
|
||||
codebook_loss = (
|
||||
F.mse_loss(z_q, z_e.detach(), reduction="none").mean([1, 2])
|
||||
* self.codebook_loss_weight
|
||||
)
|
||||
else:
|
||||
commit_loss = torch.zeros(z.shape[0], device=z.device)
|
||||
codebook_loss = torch.zeros(z.shape[0], device=z.device)
|
||||
|
||||
z_q = z_e + (z_q - z_e).detach()
|
||||
|
||||
z_q = self.out_project(z_q)
|
||||
|
||||
return z_q, commit_loss, codebook_loss, indices, z_e
|
||||
|
||||
def decode_latents(self, latents):
|
||||
encodings = rearrange(latents, "b d t -> b t d")
|
||||
z_q, indices = self.codebook(encodings)
|
||||
z_q = z_q.transpose(1, 2)
|
||||
return z_q, indices
|
||||
|
||||
def vq2emb(self, vq, out_proj=True):
|
||||
emb = self.codebook.vq2emb(vq)
|
||||
emb = emb.transpose(1, 2)
|
||||
if out_proj:
|
||||
emb = self.out_project(emb)
|
||||
return emb
|
||||
|
||||
def latent2dist(self, latents):
|
||||
latents = rearrange(latents, "b d t -> b t d")
|
||||
dist, embed_ind, quantize = self.codebook.latent2dist(latents)
|
||||
return dist, embed_ind, quantize.transpose(1, 2)
|
||||
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
# TTS Audio Suite patch: Package marker for the bundled IndexTTS 2.5 Vocos codec.
|
||||
@@ -1,853 +0,0 @@
|
||||
# TTS Audio Suite patch: Bundled from the official IndexTTS 2.5 runtime to avoid upstream dependency pins conflicting with ComfyUI.
|
||||
# Copyright (c) 2024 Amphion.
|
||||
#
|
||||
# This source code is licensed under the MIT license found in the
|
||||
# LICENSE file in the root directory of this source tree.
|
||||
|
||||
from typing import Optional, Tuple
|
||||
|
||||
import numpy as np
|
||||
import scipy
|
||||
import torch
|
||||
from torch import nn, view_as_real, view_as_complex
|
||||
from torch import nn
|
||||
from torch.nn.utils import weight_norm, remove_weight_norm
|
||||
from torchaudio.functional.functional import _hz_to_mel, _mel_to_hz
|
||||
|
||||
|
||||
def safe_log(x: torch.Tensor, clip_val: float = 1e-7) -> torch.Tensor:
|
||||
"""
|
||||
Computes the element-wise logarithm of the input tensor with clipping to avoid near-zero values.
|
||||
|
||||
Args:
|
||||
x (Tensor): Input tensor.
|
||||
clip_val (float, optional): Minimum value to clip the input tensor. Defaults to 1e-7.
|
||||
|
||||
Returns:
|
||||
Tensor: Element-wise logarithm of the input tensor with clipping applied.
|
||||
"""
|
||||
return torch.log(torch.clip(x, min=clip_val))
|
||||
|
||||
|
||||
def symlog(x: torch.Tensor) -> torch.Tensor:
|
||||
return torch.sign(x) * torch.log1p(x.abs())
|
||||
|
||||
|
||||
def symexp(x: torch.Tensor) -> torch.Tensor:
|
||||
return torch.sign(x) * (torch.exp(x.abs()) - 1)
|
||||
|
||||
|
||||
class STFT(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
n_fft: int,
|
||||
hop_length: int,
|
||||
win_length: int,
|
||||
center=True,
|
||||
):
|
||||
super().__init__()
|
||||
self.center = center
|
||||
self.n_fft = n_fft
|
||||
self.hop_length = hop_length
|
||||
self.win_length = win_length
|
||||
window = torch.hann_window(win_length)
|
||||
self.register_buffer("window", window)
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
# x: (B, T * hop_length)
|
||||
|
||||
if not self.center:
|
||||
pad = self.win_length - self.hop_length
|
||||
x = torch.nn.functional.pad(x, (pad // 2, pad // 2), mode="reflect")
|
||||
|
||||
stft_spec = torch.stft(
|
||||
x,
|
||||
self.n_fft,
|
||||
hop_length=self.hop_length,
|
||||
win_length=self.win_length,
|
||||
window=self.window,
|
||||
center=self.center,
|
||||
return_complex=False,
|
||||
) # (B, n_fft // 2 + 1, T, 2)
|
||||
|
||||
rea = stft_spec[:, :, :, 0] # (B, n_fft // 2 + 1, T, 2)
|
||||
imag = stft_spec[:, :, :, 1] # (B, n_fft // 2 + 1, T, 2)
|
||||
|
||||
log_mag = torch.log(
|
||||
torch.abs(torch.sqrt(torch.pow(rea, 2) + torch.pow(imag, 2))) + 1e-5
|
||||
) # (B, n_fft // 2 + 1, T)
|
||||
phase = torch.atan2(imag, rea) # (B, n_fft // 2 + 1, T)
|
||||
|
||||
return log_mag, phase
|
||||
|
||||
|
||||
class ISTFT(nn.Module):
|
||||
"""
|
||||
Custom implementation of ISTFT since torch.istft doesn't allow custom padding (other than `center=True`) with
|
||||
windowing. This is because the NOLA (Nonzero Overlap Add) check fails at the edges.
|
||||
See issue: https://github.com/pytorch/pytorch/issues/62323
|
||||
Specifically, in the context of neural vocoding we are interested in "same" padding analogous to CNNs.
|
||||
The NOLA constraint is met as we trim padded samples anyway.
|
||||
|
||||
Args:
|
||||
n_fft (int): Size of Fourier transform.
|
||||
hop_length (int): The distance between neighboring sliding window frames.
|
||||
win_length (int): The size of window frame and STFT filter.
|
||||
padding (str, optional): Type of padding. Options are "center" or "same". Defaults to "same".
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self, n_fft: int, hop_length: int, win_length: int, padding: str = "same"
|
||||
):
|
||||
super().__init__()
|
||||
if padding not in ["center", "same"]:
|
||||
raise ValueError("Padding must be 'center' or 'same'.")
|
||||
self.padding = padding
|
||||
self.n_fft = n_fft
|
||||
self.hop_length = hop_length
|
||||
self.win_length = win_length
|
||||
window = torch.hann_window(win_length)
|
||||
self.register_buffer("window", window)
|
||||
|
||||
def forward(self, spec: torch.Tensor) -> torch.Tensor:
|
||||
"""
|
||||
Compute the Inverse Short Time Fourier Transform (ISTFT) of a complex spectrogram.
|
||||
|
||||
Args:
|
||||
spec (Tensor): Input complex spectrogram of shape (B, N, T), where B is the batch size,
|
||||
N is the number of frequency bins, and T is the number of time frames.
|
||||
|
||||
Returns:
|
||||
Tensor: Reconstructed time-domain signal of shape (B, L), where L is the length of the output signal.
|
||||
"""
|
||||
if self.padding == "center":
|
||||
# Fallback to pytorch native implementation
|
||||
return torch.istft(
|
||||
spec,
|
||||
self.n_fft,
|
||||
self.hop_length,
|
||||
self.win_length,
|
||||
self.window,
|
||||
center=True,
|
||||
)
|
||||
elif self.padding == "same":
|
||||
pad = (self.win_length - self.hop_length) // 2
|
||||
else:
|
||||
raise ValueError("Padding must be 'center' or 'same'.")
|
||||
|
||||
assert spec.dim() == 3, "Expected a 3D tensor as input"
|
||||
B, N, T = spec.shape
|
||||
|
||||
# Inverse FFT
|
||||
ifft = torch.fft.irfft(spec, self.n_fft, dim=1, norm="backward")
|
||||
ifft = ifft * self.window[None, :, None]
|
||||
|
||||
# Overlap and Add
|
||||
output_size = (T - 1) * self.hop_length + self.win_length
|
||||
y = torch.nn.functional.fold(
|
||||
ifft,
|
||||
output_size=(1, output_size),
|
||||
kernel_size=(1, self.win_length),
|
||||
stride=(1, self.hop_length),
|
||||
)[:, 0, 0, pad:-pad]
|
||||
|
||||
# Window envelope
|
||||
window_sq = self.window.square().expand(1, T, -1).transpose(1, 2)
|
||||
window_envelope = torch.nn.functional.fold(
|
||||
window_sq,
|
||||
output_size=(1, output_size),
|
||||
kernel_size=(1, self.win_length),
|
||||
stride=(1, self.hop_length),
|
||||
).squeeze()[pad:-pad]
|
||||
|
||||
# Normalize
|
||||
assert (window_envelope > 1e-11).all()
|
||||
y = y / window_envelope
|
||||
|
||||
return y
|
||||
|
||||
|
||||
class MDCT(nn.Module):
|
||||
"""
|
||||
Modified Discrete Cosine Transform (MDCT) module.
|
||||
|
||||
Args:
|
||||
frame_len (int): Length of the MDCT frame.
|
||||
padding (str, optional): Type of padding. Options are "center" or "same". Defaults to "same".
|
||||
"""
|
||||
|
||||
def __init__(self, frame_len: int, padding: str = "same"):
|
||||
super().__init__()
|
||||
if padding not in ["center", "same"]:
|
||||
raise ValueError("Padding must be 'center' or 'same'.")
|
||||
self.padding = padding
|
||||
self.frame_len = frame_len
|
||||
N = frame_len // 2
|
||||
n0 = (N + 1) / 2
|
||||
window = torch.from_numpy(scipy.signal.cosine(frame_len)).float()
|
||||
self.register_buffer("window", window)
|
||||
|
||||
pre_twiddle = torch.exp(-1j * torch.pi * torch.arange(frame_len) / frame_len)
|
||||
post_twiddle = torch.exp(-1j * torch.pi * n0 * (torch.arange(N) + 0.5) / N)
|
||||
# view_as_real: NCCL Backend does not support ComplexFloat data type
|
||||
# https://github.com/pytorch/pytorch/issues/71613
|
||||
self.register_buffer("pre_twiddle", view_as_real(pre_twiddle))
|
||||
self.register_buffer("post_twiddle", view_as_real(post_twiddle))
|
||||
|
||||
def forward(self, audio: torch.Tensor) -> torch.Tensor:
|
||||
"""
|
||||
Apply the Modified Discrete Cosine Transform (MDCT) to the input audio.
|
||||
|
||||
Args:
|
||||
audio (Tensor): Input audio waveform of shape (B, T), where B is the batch size
|
||||
and T is the length of the audio.
|
||||
|
||||
Returns:
|
||||
Tensor: MDCT coefficients of shape (B, L, N), where L is the number of output frames
|
||||
and N is the number of frequency bins.
|
||||
"""
|
||||
if self.padding == "center":
|
||||
audio = torch.nn.functional.pad(
|
||||
audio, (self.frame_len // 2, self.frame_len // 2)
|
||||
)
|
||||
elif self.padding == "same":
|
||||
# hop_length is 1/2 frame_len
|
||||
audio = torch.nn.functional.pad(
|
||||
audio, (self.frame_len // 4, self.frame_len // 4)
|
||||
)
|
||||
else:
|
||||
raise ValueError("Padding must be 'center' or 'same'.")
|
||||
|
||||
x = audio.unfold(-1, self.frame_len, self.frame_len // 2)
|
||||
N = self.frame_len // 2
|
||||
x = x * self.window.expand(x.shape)
|
||||
X = torch.fft.fft(
|
||||
x * view_as_complex(self.pre_twiddle).expand(x.shape), dim=-1
|
||||
)[..., :N]
|
||||
res = X * view_as_complex(self.post_twiddle).expand(X.shape) * np.sqrt(1 / N)
|
||||
return torch.real(res) * np.sqrt(2)
|
||||
|
||||
|
||||
class IMDCT(nn.Module):
|
||||
"""
|
||||
Inverse Modified Discrete Cosine Transform (IMDCT) module.
|
||||
|
||||
Args:
|
||||
frame_len (int): Length of the MDCT frame.
|
||||
padding (str, optional): Type of padding. Options are "center" or "same". Defaults to "same".
|
||||
"""
|
||||
|
||||
def __init__(self, frame_len: int, padding: str = "same"):
|
||||
super().__init__()
|
||||
if padding not in ["center", "same"]:
|
||||
raise ValueError("Padding must be 'center' or 'same'.")
|
||||
self.padding = padding
|
||||
self.frame_len = frame_len
|
||||
N = frame_len // 2
|
||||
n0 = (N + 1) / 2
|
||||
window = torch.from_numpy(scipy.signal.cosine(frame_len)).float()
|
||||
self.register_buffer("window", window)
|
||||
|
||||
pre_twiddle = torch.exp(1j * torch.pi * n0 * torch.arange(N * 2) / N)
|
||||
post_twiddle = torch.exp(1j * torch.pi * (torch.arange(N * 2) + n0) / (N * 2))
|
||||
self.register_buffer("pre_twiddle", view_as_real(pre_twiddle))
|
||||
self.register_buffer("post_twiddle", view_as_real(post_twiddle))
|
||||
|
||||
def forward(self, X: torch.Tensor) -> torch.Tensor:
|
||||
"""
|
||||
Apply the Inverse Modified Discrete Cosine Transform (IMDCT) to the input MDCT coefficients.
|
||||
|
||||
Args:
|
||||
X (Tensor): Input MDCT coefficients of shape (B, L, N), where B is the batch size,
|
||||
L is the number of frames, and N is the number of frequency bins.
|
||||
|
||||
Returns:
|
||||
Tensor: Reconstructed audio waveform of shape (B, T), where T is the length of the audio.
|
||||
"""
|
||||
B, L, N = X.shape
|
||||
Y = torch.zeros((B, L, N * 2), dtype=X.dtype, device=X.device)
|
||||
Y[..., :N] = X
|
||||
Y[..., N:] = -1 * torch.conj(torch.flip(X, dims=(-1,)))
|
||||
y = torch.fft.ifft(
|
||||
Y * view_as_complex(self.pre_twiddle).expand(Y.shape), dim=-1
|
||||
)
|
||||
y = (
|
||||
torch.real(y * view_as_complex(self.post_twiddle).expand(y.shape))
|
||||
* np.sqrt(N)
|
||||
* np.sqrt(2)
|
||||
)
|
||||
result = y * self.window.expand(y.shape)
|
||||
output_size = (1, (L + 1) * N)
|
||||
audio = torch.nn.functional.fold(
|
||||
result.transpose(1, 2),
|
||||
output_size=output_size,
|
||||
kernel_size=(1, self.frame_len),
|
||||
stride=(1, self.frame_len // 2),
|
||||
)[:, 0, 0, :]
|
||||
|
||||
if self.padding == "center":
|
||||
pad = self.frame_len // 2
|
||||
elif self.padding == "same":
|
||||
pad = self.frame_len // 4
|
||||
else:
|
||||
raise ValueError("Padding must be 'center' or 'same'.")
|
||||
|
||||
audio = audio[:, pad:-pad]
|
||||
return audio
|
||||
|
||||
|
||||
class FourierHead(nn.Module):
|
||||
"""Base class for inverse fourier modules."""
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
"""
|
||||
Args:
|
||||
x (Tensor): Input tensor of shape (B, L, H), where B is the batch size,
|
||||
L is the sequence length, and H denotes the model dimension.
|
||||
|
||||
Returns:
|
||||
Tensor: Reconstructed time-domain audio signal of shape (B, T), where T is the length of the output signal.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement the forward method.")
|
||||
|
||||
|
||||
class ISTFTHead(FourierHead):
|
||||
"""
|
||||
ISTFT Head module for predicting STFT complex coefficients.
|
||||
|
||||
Args:
|
||||
dim (int): Hidden dimension of the model.
|
||||
n_fft (int): Size of Fourier transform.
|
||||
hop_length (int): The distance between neighboring sliding window frames, which should align with
|
||||
the resolution of the input features.
|
||||
padding (str, optional): Type of padding. Options are "center" or "same". Defaults to "same".
|
||||
"""
|
||||
|
||||
def __init__(self, dim: int, n_fft: int, hop_length: int, padding: str = "same"):
|
||||
super().__init__()
|
||||
out_dim = n_fft + 2
|
||||
self.out = torch.nn.Linear(dim, out_dim)
|
||||
self.istft = ISTFT(
|
||||
n_fft=n_fft, hop_length=hop_length, win_length=n_fft, padding=padding
|
||||
)
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
"""
|
||||
Forward pass of the ISTFTHead module.
|
||||
|
||||
Args:
|
||||
x (Tensor): Input tensor of shape (B, L, H), where B is the batch size,
|
||||
L is the sequence length, and H denotes the model dimension.
|
||||
|
||||
Returns:
|
||||
Tensor: Reconstructed time-domain audio signal of shape (B, T), where T is the length of the output signal.
|
||||
"""
|
||||
x = self.out(x).transpose(1, 2)
|
||||
mag, p = x.chunk(2, dim=1)
|
||||
mag = torch.exp(mag)
|
||||
mag = torch.clip(
|
||||
mag, max=1e2
|
||||
) # safeguard to prevent excessively large magnitudes
|
||||
# wrapping happens here. These two lines produce real and imaginary value
|
||||
x = torch.cos(p)
|
||||
y = torch.sin(p)
|
||||
# recalculating phase here does not produce anything new
|
||||
# only costs time
|
||||
# phase = torch.atan2(y, x)
|
||||
# S = mag * torch.exp(phase * 1j)
|
||||
# better directly produce the complex value
|
||||
S = mag * (x + 1j * y)
|
||||
audio = self.istft(S)
|
||||
return audio
|
||||
|
||||
|
||||
class IMDCTSymExpHead(FourierHead):
|
||||
"""
|
||||
IMDCT Head module for predicting MDCT coefficients with symmetric exponential function
|
||||
|
||||
Args:
|
||||
dim (int): Hidden dimension of the model.
|
||||
mdct_frame_len (int): Length of the MDCT frame.
|
||||
padding (str, optional): Type of padding. Options are "center" or "same". Defaults to "same".
|
||||
sample_rate (int, optional): The sample rate of the audio. If provided, the last layer will be initialized
|
||||
based on perceptual scaling. Defaults to None.
|
||||
clip_audio (bool, optional): Whether to clip the audio output within the range of [-1.0, 1.0]. Defaults to False.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dim: int,
|
||||
mdct_frame_len: int,
|
||||
padding: str = "same",
|
||||
sample_rate: Optional[int] = None,
|
||||
clip_audio: bool = False,
|
||||
):
|
||||
super().__init__()
|
||||
out_dim = mdct_frame_len // 2
|
||||
self.out = nn.Linear(dim, out_dim)
|
||||
self.imdct = IMDCT(frame_len=mdct_frame_len, padding=padding)
|
||||
self.clip_audio = clip_audio
|
||||
|
||||
if sample_rate is not None:
|
||||
# optionally init the last layer following mel-scale
|
||||
m_max = _hz_to_mel(sample_rate // 2)
|
||||
m_pts = torch.linspace(0, m_max, out_dim)
|
||||
f_pts = _mel_to_hz(m_pts)
|
||||
scale = 1 - (f_pts / f_pts.max())
|
||||
|
||||
with torch.no_grad():
|
||||
self.out.weight.mul_(scale.view(-1, 1))
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
"""
|
||||
Forward pass of the IMDCTSymExpHead module.
|
||||
|
||||
Args:
|
||||
x (Tensor): Input tensor of shape (B, L, H), where B is the batch size,
|
||||
L is the sequence length, and H denotes the model dimension.
|
||||
|
||||
Returns:
|
||||
Tensor: Reconstructed time-domain audio signal of shape (B, T), where T is the length of the output signal.
|
||||
"""
|
||||
x = self.out(x)
|
||||
x = symexp(x)
|
||||
x = torch.clip(
|
||||
x, min=-1e2, max=1e2
|
||||
) # safeguard to prevent excessively large magnitudes
|
||||
audio = self.imdct(x)
|
||||
if self.clip_audio:
|
||||
audio = torch.clip(x, min=-1.0, max=1.0)
|
||||
|
||||
return audio
|
||||
|
||||
|
||||
class IMDCTCosHead(FourierHead):
|
||||
"""
|
||||
IMDCT Head module for predicting MDCT coefficients with parametrizing MDCT = exp(m) · cos(p)
|
||||
|
||||
Args:
|
||||
dim (int): Hidden dimension of the model.
|
||||
mdct_frame_len (int): Length of the MDCT frame.
|
||||
padding (str, optional): Type of padding. Options are "center" or "same". Defaults to "same".
|
||||
clip_audio (bool, optional): Whether to clip the audio output within the range of [-1.0, 1.0]. Defaults to False.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dim: int,
|
||||
mdct_frame_len: int,
|
||||
padding: str = "same",
|
||||
clip_audio: bool = False,
|
||||
):
|
||||
super().__init__()
|
||||
self.clip_audio = clip_audio
|
||||
self.out = nn.Linear(dim, mdct_frame_len)
|
||||
self.imdct = IMDCT(frame_len=mdct_frame_len, padding=padding)
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
"""
|
||||
Forward pass of the IMDCTCosHead module.
|
||||
|
||||
Args:
|
||||
x (Tensor): Input tensor of shape (B, L, H), where B is the batch size,
|
||||
L is the sequence length, and H denotes the model dimension.
|
||||
|
||||
Returns:
|
||||
Tensor: Reconstructed time-domain audio signal of shape (B, T), where T is the length of the output signal.
|
||||
"""
|
||||
x = self.out(x)
|
||||
m, p = x.chunk(2, dim=2)
|
||||
m = torch.exp(m).clip(
|
||||
max=1e2
|
||||
) # safeguard to prevent excessively large magnitudes
|
||||
audio = self.imdct(m * torch.cos(p))
|
||||
if self.clip_audio:
|
||||
audio = torch.clip(x, min=-1.0, max=1.0)
|
||||
return audio
|
||||
|
||||
|
||||
class ConvNeXtBlock(nn.Module):
|
||||
"""ConvNeXt Block adapted from https://github.com/facebookresearch/ConvNeXt to 1D audio signal.
|
||||
|
||||
Args:
|
||||
dim (int): Number of input channels.
|
||||
intermediate_dim (int): Dimensionality of the intermediate layer.
|
||||
layer_scale_init_value (float, optional): Initial value for the layer scale. None means no scaling.
|
||||
Defaults to None.
|
||||
adanorm_num_embeddings (int, optional): Number of embeddings for AdaLayerNorm.
|
||||
None means non-conditional LayerNorm. Defaults to None.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dim: int,
|
||||
intermediate_dim: int,
|
||||
layer_scale_init_value: float,
|
||||
adanorm_num_embeddings: Optional[int] = None,
|
||||
):
|
||||
super().__init__()
|
||||
self.dwconv = nn.Conv1d(
|
||||
dim, dim, kernel_size=7, padding=3, groups=dim
|
||||
) # depthwise conv
|
||||
self.adanorm = adanorm_num_embeddings is not None
|
||||
if adanorm_num_embeddings:
|
||||
self.norm = AdaLayerNorm(adanorm_num_embeddings, dim, eps=1e-6)
|
||||
else:
|
||||
self.norm = nn.LayerNorm(dim, eps=1e-6)
|
||||
self.pwconv1 = nn.Linear(
|
||||
dim, intermediate_dim
|
||||
) # pointwise/1x1 convs, implemented with linear layers
|
||||
self.act = nn.GELU()
|
||||
self.pwconv2 = nn.Linear(intermediate_dim, dim)
|
||||
self.gamma = (
|
||||
nn.Parameter(layer_scale_init_value * torch.ones(dim), requires_grad=True)
|
||||
if layer_scale_init_value > 0
|
||||
else None
|
||||
)
|
||||
|
||||
def forward(
|
||||
self, x: torch.Tensor, cond_embedding_id: Optional[torch.Tensor] = None
|
||||
) -> torch.Tensor:
|
||||
residual = x
|
||||
x = self.dwconv(x)
|
||||
x = x.transpose(1, 2) # (B, C, T) -> (B, T, C)
|
||||
if self.adanorm:
|
||||
assert cond_embedding_id is not None
|
||||
x = self.norm(x, cond_embedding_id)
|
||||
else:
|
||||
x = self.norm(x)
|
||||
x = self.pwconv1(x)
|
||||
x = self.act(x)
|
||||
x = self.pwconv2(x)
|
||||
if self.gamma is not None:
|
||||
x = self.gamma * x
|
||||
x = x.transpose(1, 2) # (B, T, C) -> (B, C, T)
|
||||
|
||||
x = residual + x
|
||||
return x
|
||||
|
||||
|
||||
class AdaLayerNorm(nn.Module):
|
||||
"""
|
||||
Adaptive Layer Normalization module with learnable embeddings per `num_embeddings` classes
|
||||
|
||||
Args:
|
||||
num_embeddings (int): Number of embeddings.
|
||||
embedding_dim (int): Dimension of the embeddings.
|
||||
"""
|
||||
|
||||
def __init__(self, num_embeddings: int, embedding_dim: int, eps: float = 1e-6):
|
||||
super().__init__()
|
||||
self.eps = eps
|
||||
self.dim = embedding_dim
|
||||
self.scale = nn.Embedding(
|
||||
num_embeddings=num_embeddings, embedding_dim=embedding_dim
|
||||
)
|
||||
self.shift = nn.Embedding(
|
||||
num_embeddings=num_embeddings, embedding_dim=embedding_dim
|
||||
)
|
||||
torch.nn.init.ones_(self.scale.weight)
|
||||
torch.nn.init.zeros_(self.shift.weight)
|
||||
|
||||
def forward(self, x: torch.Tensor, cond_embedding_id: torch.Tensor) -> torch.Tensor:
|
||||
scale = self.scale(cond_embedding_id)
|
||||
shift = self.shift(cond_embedding_id)
|
||||
x = nn.functional.layer_norm(x, (self.dim,), eps=self.eps)
|
||||
x = x * scale + shift
|
||||
return x
|
||||
|
||||
|
||||
class ResBlock1(nn.Module):
|
||||
"""
|
||||
ResBlock adapted from HiFi-GAN V1 (https://github.com/jik876/hifi-gan) with dilated 1D convolutions,
|
||||
but without upsampling layers.
|
||||
|
||||
Args:
|
||||
dim (int): Number of input channels.
|
||||
kernel_size (int, optional): Size of the convolutional kernel. Defaults to 3.
|
||||
dilation (tuple[int], optional): Dilation factors for the dilated convolutions.
|
||||
Defaults to (1, 3, 5).
|
||||
lrelu_slope (float, optional): Negative slope of the LeakyReLU activation function.
|
||||
Defaults to 0.1.
|
||||
layer_scale_init_value (float, optional): Initial value for the layer scale. None means no scaling.
|
||||
Defaults to None.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dim: int,
|
||||
kernel_size: int = 3,
|
||||
dilation: Tuple[int, int, int] = (1, 3, 5),
|
||||
lrelu_slope: float = 0.1,
|
||||
layer_scale_init_value: Optional[float] = None,
|
||||
):
|
||||
super().__init__()
|
||||
self.lrelu_slope = lrelu_slope
|
||||
self.convs1 = nn.ModuleList(
|
||||
[
|
||||
weight_norm(
|
||||
nn.Conv1d(
|
||||
dim,
|
||||
dim,
|
||||
kernel_size,
|
||||
1,
|
||||
dilation=dilation[0],
|
||||
padding=self.get_padding(kernel_size, dilation[0]),
|
||||
)
|
||||
),
|
||||
weight_norm(
|
||||
nn.Conv1d(
|
||||
dim,
|
||||
dim,
|
||||
kernel_size,
|
||||
1,
|
||||
dilation=dilation[1],
|
||||
padding=self.get_padding(kernel_size, dilation[1]),
|
||||
)
|
||||
),
|
||||
weight_norm(
|
||||
nn.Conv1d(
|
||||
dim,
|
||||
dim,
|
||||
kernel_size,
|
||||
1,
|
||||
dilation=dilation[2],
|
||||
padding=self.get_padding(kernel_size, dilation[2]),
|
||||
)
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
self.convs2 = nn.ModuleList(
|
||||
[
|
||||
weight_norm(
|
||||
nn.Conv1d(
|
||||
dim,
|
||||
dim,
|
||||
kernel_size,
|
||||
1,
|
||||
dilation=1,
|
||||
padding=self.get_padding(kernel_size, 1),
|
||||
)
|
||||
),
|
||||
weight_norm(
|
||||
nn.Conv1d(
|
||||
dim,
|
||||
dim,
|
||||
kernel_size,
|
||||
1,
|
||||
dilation=1,
|
||||
padding=self.get_padding(kernel_size, 1),
|
||||
)
|
||||
),
|
||||
weight_norm(
|
||||
nn.Conv1d(
|
||||
dim,
|
||||
dim,
|
||||
kernel_size,
|
||||
1,
|
||||
dilation=1,
|
||||
padding=self.get_padding(kernel_size, 1),
|
||||
)
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
self.gamma = nn.ParameterList(
|
||||
[
|
||||
(
|
||||
nn.Parameter(
|
||||
layer_scale_init_value * torch.ones(dim, 1), requires_grad=True
|
||||
)
|
||||
if layer_scale_init_value is not None
|
||||
else None
|
||||
),
|
||||
(
|
||||
nn.Parameter(
|
||||
layer_scale_init_value * torch.ones(dim, 1), requires_grad=True
|
||||
)
|
||||
if layer_scale_init_value is not None
|
||||
else None
|
||||
),
|
||||
(
|
||||
nn.Parameter(
|
||||
layer_scale_init_value * torch.ones(dim, 1), requires_grad=True
|
||||
)
|
||||
if layer_scale_init_value is not None
|
||||
else None
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
for c1, c2, gamma in zip(self.convs1, self.convs2, self.gamma):
|
||||
xt = torch.nn.functional.leaky_relu(x, negative_slope=self.lrelu_slope)
|
||||
xt = c1(xt)
|
||||
xt = torch.nn.functional.leaky_relu(xt, negative_slope=self.lrelu_slope)
|
||||
xt = c2(xt)
|
||||
if gamma is not None:
|
||||
xt = gamma * xt
|
||||
x = xt + x
|
||||
return x
|
||||
|
||||
def remove_weight_norm(self):
|
||||
for l in self.convs1:
|
||||
remove_weight_norm(l)
|
||||
for l in self.convs2:
|
||||
remove_weight_norm(l)
|
||||
|
||||
@staticmethod
|
||||
def get_padding(kernel_size: int, dilation: int = 1) -> int:
|
||||
return int((kernel_size * dilation - dilation) / 2)
|
||||
|
||||
|
||||
class Backbone(nn.Module):
|
||||
"""Base class for the generator's backbone. It preserves the same temporal resolution across all layers."""
|
||||
|
||||
def forward(self, x: torch.Tensor, **kwargs) -> torch.Tensor:
|
||||
"""
|
||||
Args:
|
||||
x (Tensor): Input tensor of shape (B, C, L), where B is the batch size,
|
||||
C denotes output features, and L is the sequence length.
|
||||
|
||||
Returns:
|
||||
Tensor: Output of shape (B, L, H), where B is the batch size, L is the sequence length,
|
||||
and H denotes the model dimension.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement the forward method.")
|
||||
|
||||
|
||||
class VocosBackbone(Backbone):
|
||||
"""
|
||||
Vocos backbone module built with ConvNeXt blocks. Supports additional conditioning with Adaptive Layer Normalization
|
||||
|
||||
Args:
|
||||
input_channels (int): Number of input features channels.
|
||||
dim (int): Hidden dimension of the model.
|
||||
intermediate_dim (int): Intermediate dimension used in ConvNeXtBlock.
|
||||
num_layers (int): Number of ConvNeXtBlock layers.
|
||||
layer_scale_init_value (float, optional): Initial value for layer scaling. Defaults to `1 / num_layers`.
|
||||
adanorm_num_embeddings (int, optional): Number of embeddings for AdaLayerNorm.
|
||||
None means non-conditional model. Defaults to None.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
input_channels: int,
|
||||
dim: int,
|
||||
intermediate_dim: int,
|
||||
num_layers: int,
|
||||
layer_scale_init_value: Optional[float] = None,
|
||||
adanorm_num_embeddings: Optional[int] = None,
|
||||
):
|
||||
super().__init__()
|
||||
self.input_channels = input_channels
|
||||
self.embed = nn.Conv1d(input_channels, dim, kernel_size=7, padding=3)
|
||||
self.adanorm = adanorm_num_embeddings is not None
|
||||
if adanorm_num_embeddings:
|
||||
self.norm = AdaLayerNorm(adanorm_num_embeddings, dim, eps=1e-6)
|
||||
else:
|
||||
self.norm = nn.LayerNorm(dim, eps=1e-6)
|
||||
layer_scale_init_value = layer_scale_init_value or 1 / num_layers
|
||||
self.convnext = nn.ModuleList(
|
||||
[
|
||||
ConvNeXtBlock(
|
||||
dim=dim,
|
||||
intermediate_dim=intermediate_dim,
|
||||
layer_scale_init_value=layer_scale_init_value,
|
||||
adanorm_num_embeddings=adanorm_num_embeddings,
|
||||
)
|
||||
for _ in range(num_layers)
|
||||
]
|
||||
)
|
||||
self.final_layer_norm = nn.LayerNorm(dim, eps=1e-6)
|
||||
self.apply(self._init_weights)
|
||||
|
||||
def _init_weights(self, m):
|
||||
if isinstance(m, (nn.Conv1d, nn.Linear)):
|
||||
nn.init.trunc_normal_(m.weight, std=0.02)
|
||||
nn.init.constant_(m.bias, 0)
|
||||
|
||||
def forward(self, x: torch.Tensor, **kwargs) -> torch.Tensor:
|
||||
bandwidth_id = kwargs.get("bandwidth_id", None)
|
||||
x = self.embed(x)
|
||||
if self.adanorm:
|
||||
assert bandwidth_id is not None
|
||||
x = self.norm(x.transpose(1, 2), cond_embedding_id=bandwidth_id)
|
||||
else:
|
||||
x = self.norm(x.transpose(1, 2))
|
||||
x = x.transpose(1, 2)
|
||||
for conv_block in self.convnext:
|
||||
x = conv_block(x, cond_embedding_id=bandwidth_id)
|
||||
x = self.final_layer_norm(x.transpose(1, 2))
|
||||
return x
|
||||
|
||||
|
||||
class VocosResNetBackbone(Backbone):
|
||||
"""
|
||||
Vocos backbone module built with ResBlocks.
|
||||
|
||||
Args:
|
||||
input_channels (int): Number of input features channels.
|
||||
dim (int): Hidden dimension of the model.
|
||||
num_blocks (int): Number of ResBlock1 blocks.
|
||||
layer_scale_init_value (float, optional): Initial value for layer scaling. Defaults to None.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
input_channels,
|
||||
dim,
|
||||
num_blocks,
|
||||
layer_scale_init_value=None,
|
||||
):
|
||||
super().__init__()
|
||||
self.input_channels = input_channels
|
||||
self.embed = weight_norm(
|
||||
nn.Conv1d(input_channels, dim, kernel_size=3, padding=1)
|
||||
)
|
||||
layer_scale_init_value = layer_scale_init_value or 1 / num_blocks / 3
|
||||
self.resnet = nn.Sequential(
|
||||
*[
|
||||
ResBlock1(dim=dim, layer_scale_init_value=layer_scale_init_value)
|
||||
for _ in range(num_blocks)
|
||||
]
|
||||
)
|
||||
|
||||
def forward(self, x: torch.Tensor, **kwargs) -> torch.Tensor:
|
||||
x = self.embed(x)
|
||||
x = self.resnet(x)
|
||||
x = x.transpose(1, 2)
|
||||
return x
|
||||
|
||||
|
||||
class Vocos(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
input_channels: int = 256,
|
||||
dim: int = 384,
|
||||
intermediate_dim: int = 1152,
|
||||
num_layers: int = 8,
|
||||
adanorm_num_embeddings: int = 4,
|
||||
n_fft: int = 800,
|
||||
hop_size: int = 200,
|
||||
padding: str = "same",
|
||||
):
|
||||
super().__init__()
|
||||
|
||||
self.backbone = VocosBackbone(
|
||||
input_channels=input_channels,
|
||||
dim=dim,
|
||||
intermediate_dim=intermediate_dim,
|
||||
num_layers=num_layers,
|
||||
adanorm_num_embeddings=adanorm_num_embeddings,
|
||||
)
|
||||
self.head = ISTFTHead(dim, n_fft, hop_size, padding)
|
||||
|
||||
def forward(self, x):
|
||||
x = self.backbone(x)
|
||||
x = self.head(x)
|
||||
|
||||
return x[:, None, :]
|
||||
|
||||
|
||||
@@ -1,10 +0,0 @@
|
||||
# TTS Audio Suite patch: Bundled from the official IndexTTS 2.5 runtime to avoid upstream dependency pins conflicting with ComfyUI.
|
||||
from indextts.utils.maskgct.models.codec.kmeans.repcodec_model import RepCodec
|
||||
|
||||
|
||||
def build_semantic_codec(cfg):
|
||||
semantic_codec = RepCodec(cfg=cfg)
|
||||
semantic_codec.eval()
|
||||
return semantic_codec
|
||||
|
||||
|
||||
@@ -1,261 +0,0 @@
|
||||
# TTS Audio Suite patch: Bundled from the official IndexTTS 2.5 runtime to avoid upstream dependency pins conflicting with ComfyUI.
|
||||
# Copyright (c) 2024 Amphion.
|
||||
#
|
||||
# This source code is licensed under the MIT license found in the
|
||||
# LICENSE file in the root directory of this source tree.
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
|
||||
from torch.nn import functional as F
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
from indextts.codec.amphion_codec.quantize import ResidualVQ
|
||||
from indextts.codec.kmeans.vocos import VocosBackbone
|
||||
|
||||
|
||||
def init_weights(m):
|
||||
if isinstance(m, nn.Conv1d):
|
||||
nn.init.trunc_normal_(m.weight, std=0.02)
|
||||
nn.init.constant_(m.bias, 0)
|
||||
if isinstance(m, nn.Linear):
|
||||
nn.init.trunc_normal_(m.weight, std=0.02)
|
||||
nn.init.constant_(m.bias, 0)
|
||||
|
||||
|
||||
class EnhancedCodec(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
codebook_size=8192,
|
||||
hidden_size=1024,
|
||||
codebook_dim=8,
|
||||
vocos_dim=384,
|
||||
vocos_intermediate_dim=2048,
|
||||
vocos_num_layers=12,
|
||||
num_quantizers=1,
|
||||
downsample_scale=2,
|
||||
cfg=None,
|
||||
):
|
||||
super().__init__()
|
||||
codebook_size = (
|
||||
cfg.codebook_size
|
||||
if cfg is not None and hasattr(cfg, "codebook_size")
|
||||
else codebook_size
|
||||
)
|
||||
codebook_dim = (
|
||||
cfg.codebook_dim
|
||||
if cfg is not None and hasattr(cfg, "codebook_dim")
|
||||
else codebook_dim
|
||||
)
|
||||
hidden_size = (
|
||||
cfg.hidden_size
|
||||
if cfg is not None and hasattr(cfg, "hidden_size")
|
||||
else hidden_size
|
||||
)
|
||||
vocos_dim = (
|
||||
cfg.vocos_dim
|
||||
if cfg is not None and hasattr(cfg, "vocos_dim")
|
||||
else vocos_dim
|
||||
)
|
||||
vocos_intermediate_dim = (
|
||||
cfg.vocos_intermediate_dim
|
||||
if cfg is not None and hasattr(cfg, "vocos_intermediate_dim")
|
||||
else vocos_intermediate_dim
|
||||
)
|
||||
vocos_num_layers = (
|
||||
cfg.vocos_num_layers
|
||||
if cfg is not None and hasattr(cfg, "vocos_num_layers")
|
||||
else vocos_num_layers
|
||||
)
|
||||
num_quantizers = (
|
||||
cfg.num_quantizers
|
||||
if cfg is not None and hasattr(cfg, "num_quantizers")
|
||||
else num_quantizers
|
||||
)
|
||||
downsample_scale = (
|
||||
cfg.downsample_scale
|
||||
if cfg is not None and hasattr(cfg, "downsample_scale")
|
||||
else downsample_scale
|
||||
)
|
||||
|
||||
self.codebook_size = codebook_size
|
||||
self.codebook_dim = codebook_dim
|
||||
self.hidden_size = hidden_size
|
||||
self.vocos_dim = vocos_dim
|
||||
self.vocos_intermediate_dim = vocos_intermediate_dim
|
||||
self.vocos_num_layers = vocos_num_layers
|
||||
self.num_quantizers = num_quantizers
|
||||
self.downsample_scale = downsample_scale
|
||||
|
||||
if self.downsample_scale != None and self.downsample_scale > 1:
|
||||
self.down = nn.Conv1d(
|
||||
self.hidden_size, self.hidden_size, kernel_size=3, stride=2, padding=1
|
||||
)
|
||||
self.up = nn.Conv1d(
|
||||
self.hidden_size, self.hidden_size, kernel_size=3, stride=1, padding=1
|
||||
)
|
||||
|
||||
self.encoder = nn.Sequential(
|
||||
VocosBackbone(
|
||||
input_channels=self.hidden_size,
|
||||
dim=self.vocos_dim,
|
||||
intermediate_dim=self.vocos_intermediate_dim,
|
||||
num_layers=self.vocos_num_layers,
|
||||
adanorm_num_embeddings=None,
|
||||
),
|
||||
nn.Linear(self.vocos_dim, self.hidden_size),
|
||||
)
|
||||
self.decoder = nn.Sequential(
|
||||
VocosBackbone(
|
||||
input_channels=self.hidden_size,
|
||||
dim=self.vocos_dim,
|
||||
intermediate_dim=self.vocos_intermediate_dim,
|
||||
num_layers=self.vocos_num_layers,
|
||||
adanorm_num_embeddings=None,
|
||||
),
|
||||
nn.Linear(self.vocos_dim, self.hidden_size),
|
||||
)
|
||||
|
||||
self.quantizer = ResidualVQ(
|
||||
input_dim=hidden_size,
|
||||
num_quantizers=num_quantizers,
|
||||
codebook_size=codebook_size,
|
||||
codebook_dim=codebook_dim,
|
||||
quantizer_type="fvq",
|
||||
quantizer_dropout=0.0,
|
||||
commitment=0.15,
|
||||
codebook_loss_weight=1.0,
|
||||
use_l2_normlize=True,
|
||||
)
|
||||
|
||||
self.reset_parameters()
|
||||
|
||||
def forward(self, x):
|
||||
|
||||
# downsample
|
||||
feat = x
|
||||
length = x.size(1)
|
||||
if length % 2 != 0:
|
||||
# 去掉最后一帧
|
||||
x = x[:, :-1, :]
|
||||
feat = feat[:, :-1, :] # 关键：同步裁剪feat
|
||||
if self.downsample_scale != None and self.downsample_scale > 1:
|
||||
x = x.transpose(1, 2)
|
||||
x = self.down(x)
|
||||
x = F.gelu(x)
|
||||
x = x.transpose(1, 2)
|
||||
|
||||
x = self.encoder(x.transpose(1, 2)).transpose(1, 2)
|
||||
|
||||
(
|
||||
quantized_out,
|
||||
all_indices,
|
||||
all_commit_losses,
|
||||
all_codebook_losses,
|
||||
_,
|
||||
) = self.quantizer(x)
|
||||
|
||||
# while 1:
|
||||
# pass
|
||||
# decoder
|
||||
x = self.decoder(quantized_out)
|
||||
x_rec = x
|
||||
|
||||
# up
|
||||
if self.downsample_scale != None and self.downsample_scale > 1:
|
||||
x = x.transpose(1, 2)
|
||||
x = F.interpolate(x, scale_factor=2, mode="nearest")
|
||||
x_rec = self.up(x).transpose(1, 2)
|
||||
|
||||
codebook_loss = (all_codebook_losses + all_commit_losses).mean()
|
||||
all_indices = all_indices
|
||||
reconstruction_loss = F.mse_loss(x_rec, feat)
|
||||
|
||||
return x_rec, codebook_loss, all_indices, reconstruction_loss
|
||||
|
||||
def quantize(self, x):
|
||||
|
||||
if self.downsample_scale != None and self.downsample_scale > 1:
|
||||
x = x.transpose(1, 2)
|
||||
x = self.down(x)
|
||||
x = F.gelu(x)
|
||||
x = x.transpose(1, 2)
|
||||
|
||||
x = self.encoder(x.transpose(1, 2)).transpose(1, 2)
|
||||
|
||||
(
|
||||
quantized_out,
|
||||
all_indices,
|
||||
all_commit_losses,
|
||||
all_codebook_losses,
|
||||
_,
|
||||
) = self.quantizer(x)
|
||||
|
||||
if all_indices.shape[0] == 1:
|
||||
return all_indices.squeeze(0), quantized_out.transpose(1, 2)
|
||||
return all_indices, quantized_out.transpose(1, 2)
|
||||
|
||||
def reset_parameters(self):
|
||||
self.apply(init_weights)
|
||||
|
||||
|
||||
def decode(self, codes):
|
||||
"""
|
||||
通过 codes 恢复quantized_out
|
||||
|
||||
Args:
|
||||
codes: Tensor[N x B x T] or Tensor[B x T] (当N=1时)
|
||||
量化的索引
|
||||
|
||||
Returns:
|
||||
quantized_out: Tensor[B x D x T]
|
||||
重建的量化输出
|
||||
"""
|
||||
# 处理单个量化器的情况
|
||||
if codes.dim() == 2:
|
||||
codes = codes.unsqueeze(0) # [B, T] -> [1, B, T]
|
||||
|
||||
# 使用quantizer的vq2emb方法恢复量化输出
|
||||
quantized_out = self.quantizer.vq2emb(codes)
|
||||
x = self.decoder(quantized_out)
|
||||
|
||||
# 如果有下采样操作，则进行上采样
|
||||
if self.downsample_scale != None and self.downsample_scale > 1:
|
||||
x = x.transpose(1, 2)
|
||||
x = F.interpolate(x, scale_factor=2, mode="nearest")
|
||||
x_rec = self.up(x).transpose(1, 2)
|
||||
|
||||
return x_rec
|
||||
|
||||
def load_checkpoint(self, checkpoint_path):
|
||||
"""Load model weights from a checkpoint file."""
|
||||
assert os.path.isfile(checkpoint_path), f"Checkpoint not found: {checkpoint_path}"
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint_dict = torch.load(checkpoint_path, map_location='cpu', weights_only=True)
|
||||
saved_state_dict = checkpoint_dict['model']
|
||||
state_dict = self.state_dict()
|
||||
new_state_dict = {}
|
||||
for k, v in state_dict.items():
|
||||
if k in saved_state_dict and saved_state_dict[k].shape == v.shape:
|
||||
new_state_dict[k] = saved_state_dict[k]
|
||||
else:
|
||||
logger.warning("%s is not in the checkpoint or shape mismatch", k)
|
||||
new_state_dict[k] = v
|
||||
self.load_state_dict(new_state_dict)
|
||||
logger.info("Loaded codec checkpoint '%s'", checkpoint_path)
|
||||
|
||||
if __name__ == "__main__":
|
||||
repcodec = EnhancedCodec(vocos_dim=1024, downsample_scale=2)
|
||||
print(repcodec)
|
||||
print(sum(p.numel() for p in repcodec.parameters()) / 1e6)
|
||||
x = torch.randn(5, 10, 1024)
|
||||
x_rec, codebook_loss, all_indices = repcodec(x)
|
||||
print(x_rec.shape, codebook_loss, all_indices.shape)
|
||||
vq_id, emb = repcodec.quantize(x)
|
||||
print(vq_id.shape, emb.shape)
|
||||
|
||||
|
||||
@@ -21,7 +21,6 @@ from indextts.gpt.conformer_encoder import ConformerEncoder
|
||||
from indextts.gpt.perceiver import PerceiverResampler
|
||||
from indextts.utils.arch_util import AttentionBlock
|
||||
from indextts.utils.typical_sampling import TypicalLogitsWarper
|
||||
from indextts.utils.tokenizer import LANGUAGE_DICT
|
||||
|
||||
|
||||
def null_position_embeddings(range, dim):
|
||||
@@ -315,8 +314,7 @@ class UnifiedVoice(nn.Module):
|
||||
start_text_token=0, stop_text_token=1, number_mel_codes=8194, start_mel_token=8192, stop_mel_token=8193,
|
||||
train_solo_embeddings=False, use_mel_codes_as_input=True,
|
||||
checkpointing=True, types=1,
|
||||
condition_num_latent=32, condition_type="perceiver", condition_module=None, emo_condition_module=None, use_accel=False,
|
||||
spk_cond_mode="conformer"):
|
||||
condition_num_latent=32, condition_type="perceiver", condition_module=None, emo_condition_module=None, use_accel=False):
|
||||
"""
|
||||
Args:
|
||||
layers: Number of layers in transformer stack.
|
||||
@@ -355,31 +353,23 @@ class UnifiedVoice(nn.Module):
|
||||
self.cond_num = condition_num_latent
|
||||
self.cond_mask_pad = nn.ConstantPad1d((self.cond_num, 0), True)
|
||||
self.emo_cond_mask_pad = nn.ConstantPad1d((1, 0), True)
|
||||
# TTS Audio Suite patch: Keep one Transformers-5-compatible GPT implementation for
|
||||
# both IndexTTS-2 and 2.5 while selecting their different speaker conditioning.
|
||||
self.spk_cond_mode = spk_cond_mode
|
||||
if spk_cond_mode == "campplus":
|
||||
self.spk_emb_proj = nn.Linear(192, model_dim)
|
||||
if condition_type == "perceiver":
|
||||
self.conditioning_encoder = ConditioningEncoder(1024, model_dim, num_attn_heads=heads)
|
||||
self.perceiver_encoder = PerceiverResampler(model_dim, dim_context=model_dim, num_latents=self.cond_num)
|
||||
elif condition_type == "conformer_perceiver" or condition_type == "conformer_encoder":
|
||||
self.conditioning_encoder = ConformerEncoder(input_size=1024,
|
||||
output_size=condition_module['output_size'],
|
||||
linear_units=condition_module['linear_units'],
|
||||
attention_heads=condition_module['attention_heads'],
|
||||
num_blocks=condition_module['num_blocks'],
|
||||
input_layer=condition_module['input_layer'])
|
||||
if condition_type == "conformer_perceiver":
|
||||
self.perceiver_encoder = PerceiverResampler(model_dim, dim_context=condition_module['output_size'],
|
||||
ff_mult=condition_module['perceiver_mult'],
|
||||
heads=condition_module['attention_heads'],
|
||||
num_latents=self.cond_num)
|
||||
else:
|
||||
if condition_type == "perceiver":
|
||||
self.conditioning_encoder = ConditioningEncoder(1024, model_dim, num_attn_heads=heads)
|
||||
self.perceiver_encoder = PerceiverResampler(model_dim, dim_context=model_dim, num_latents=self.cond_num)
|
||||
elif condition_type == "conformer_perceiver" or condition_type == "conformer_encoder":
|
||||
self.conditioning_encoder = ConformerEncoder(input_size=1024,
|
||||
output_size=condition_module['output_size'],
|
||||
linear_units=condition_module['linear_units'],
|
||||
attention_heads=condition_module['attention_heads'],
|
||||
num_blocks=condition_module['num_blocks'],
|
||||
input_layer=condition_module['input_layer'])
|
||||
if condition_type == "conformer_perceiver":
|
||||
self.perceiver_encoder = PerceiverResampler(model_dim, dim_context=condition_module['output_size'],
|
||||
ff_mult=condition_module['perceiver_mult'],
|
||||
heads=condition_module['attention_heads'],
|
||||
num_latents=self.cond_num)
|
||||
else:
|
||||
self.conditioning_encoder = ConditioningEncoder(1024, model_dim, num_attn_heads=heads, mean=True)
|
||||
self.speed_emb = nn.Embedding(2, model_dim)
|
||||
self.speed_emb.weight.data.normal_(mean=0.0, std=0.0)
|
||||
self.conditioning_encoder = ConditioningEncoder(1024, model_dim, num_attn_heads=heads, mean=True)
|
||||
|
||||
self.emo_conditioning_encoder = ConformerEncoder(input_size=1024,
|
||||
output_size=emo_condition_module['output_size'],
|
||||
@@ -395,8 +385,6 @@ class UnifiedVoice(nn.Module):
|
||||
|
||||
|
||||
self.text_embedding = nn.Embedding(self.number_text_tokens * types + 1, model_dim)
|
||||
if spk_cond_mode == "campplus":
|
||||
self.lang_embedding = nn.Embedding(len(LANGUAGE_DICT) + 1, model_dim)
|
||||
self.emo_layer = nn.Linear(model_dim, model_dim)
|
||||
self.emovec_layer = nn.Linear(1024, model_dim)
|
||||
|
||||
@@ -418,6 +406,9 @@ class UnifiedVoice(nn.Module):
|
||||
self.text_head = nn.Linear(model_dim, self.number_text_tokens * types + 1)
|
||||
self.mel_head = nn.Linear(model_dim, self.number_mel_codes)
|
||||
|
||||
self.speed_emb = nn.Embedding(2, model_dim)
|
||||
self.speed_emb.weight.data.normal_(mean=0.0, std=0.0)
|
||||
|
||||
# Initialize the embeddings per the GPT-2 scheme
|
||||
embeddings = [self.text_embedding]
|
||||
if use_mel_codes_as_input:
|
||||
@@ -631,12 +622,7 @@ class UnifiedVoice(nn.Module):
|
||||
"""
|
||||
|
||||
if do_spk_cond:
|
||||
if self.spk_cond_mode == "campplus":
|
||||
speech_conditioning_latent = self.spk_emb_proj(speech_conditioning_latent)
|
||||
if speech_conditioning_latent.ndim != 3:
|
||||
speech_conditioning_latent = speech_conditioning_latent.unsqueeze(1)
|
||||
else:
|
||||
speech_conditioning_latent = self.get_conditioning(speech_conditioning_latent.transpose(1,2), cond_mel_lengths)
|
||||
speech_conditioning_latent = self.get_conditioning(speech_conditioning_latent.transpose(1,2), cond_mel_lengths)
|
||||
else:
|
||||
speech_conditioning_latent = speech_conditioning_latent
|
||||
|
||||
@@ -651,16 +637,9 @@ class UnifiedVoice(nn.Module):
|
||||
mel_codes = self.set_mel_padding(mel_codes, mel_codes_lengths)
|
||||
mel_codes = F.pad(mel_codes, (0, 1), value=self.stop_mel_token)
|
||||
|
||||
if self.spk_cond_mode == "campplus":
|
||||
padding = torch.zeros(
|
||||
speech_conditioning_latent.size(0), 2, speech_conditioning_latent.size(2),
|
||||
device=speech_conditioning_latent.device,
|
||||
)
|
||||
conds = torch.cat((speech_conditioning_latent + emo_vec.unsqueeze(1), padding), 1)
|
||||
else:
|
||||
duration_emb = self.speed_emb(torch.zeros_like(use_speed))
|
||||
duration_emb_half = self.speed_emb(torch.ones_like(use_speed))
|
||||
conds = torch.cat((speech_conditioning_latent + emo_vec.unsqueeze(1), duration_emb_half.unsqueeze(1), duration_emb.unsqueeze(1)), 1)
|
||||
duration_emb = self.speed_emb(torch.zeros_like(use_speed))
|
||||
duration_emb_half = self.speed_emb(torch.ones_like(use_speed))
|
||||
conds = torch.cat((speech_conditioning_latent + emo_vec.unsqueeze(1), duration_emb_half.unsqueeze(1), duration_emb.unsqueeze(1)), 1)
|
||||
text_inputs, text_targets = self.build_aligned_inputs_and_targets(text_inputs, self.start_text_token, self.stop_text_token)
|
||||
text_emb = self.text_embedding(text_inputs) + self.text_pos_embedding(text_inputs)
|
||||
mel_codes, mel_targets = self.build_aligned_inputs_and_targets(mel_codes, self.start_mel_token, self.stop_mel_token)
|
||||
@@ -675,7 +654,6 @@ class UnifiedVoice(nn.Module):
|
||||
self,
|
||||
conditional_latents: torch.Tensor,
|
||||
text_inputs: torch.Tensor,
|
||||
langs: torch.Tensor = None,
|
||||
):
|
||||
|
||||
"""
|
||||
@@ -703,8 +681,6 @@ class UnifiedVoice(nn.Module):
|
||||
text_input = F.pad(text_input, (0, 1), value=self.stop_text_token)
|
||||
text_input_pos = torch.arange(0, text_input.size(-1), device=device)
|
||||
text_emb = self.text_embedding(text_input) + self.text_pos_embedding.emb(text_input_pos)
|
||||
if langs is not None and self.spk_cond_mode == "campplus":
|
||||
text_emb += self.lang_embedding(langs[i])
|
||||
# concatenate [conditional latents][text embeddings]
|
||||
conds_text_emb = [
|
||||
conditional_latents.squeeze(0) if single_cond else conditional_latents[i],
|
||||
@@ -739,10 +715,7 @@ class UnifiedVoice(nn.Module):
|
||||
fake_inputs[:, -1] = self.start_mel_token
|
||||
return fake_inputs, batched_mel_emb, attention_mask
|
||||
|
||||
def inference_speech(self, speech_condition, text_inputs, langs=None,
|
||||
emo_speech_condition=None, cond_lengths=None, emo_cond_lengths=None, emo_vec=None,
|
||||
use_speed=False, campplus_embedding=None, wav=None,
|
||||
input_tokens=None, num_return_sequences=1,
|
||||
def inference_speech(self, speech_condition, text_inputs, emo_speech_condition=None, cond_lengths=None, emo_cond_lengths=None, emo_vec=None, use_speed=False, input_tokens=None, num_return_sequences=1,
|
||||
max_generate_length=None, typical_sampling=False, typical_mass=.9, **hf_generate_kwargs):
|
||||
"""
|
||||
Args:
|
||||
@@ -763,27 +736,7 @@ class UnifiedVoice(nn.Module):
|
||||
if emo_cond_lengths is None:
|
||||
emo_cond_lengths = torch.tensor([emo_speech_condition.shape[-1]], device=speech_condition.device)
|
||||
|
||||
if self.spk_cond_mode == "campplus":
|
||||
if campplus_embedding is not None:
|
||||
speech_conditioning_latent = campplus_embedding
|
||||
elif wav is not None:
|
||||
if not hasattr(self, 'sv_pipeline'):
|
||||
from modelscope.pipelines import pipeline
|
||||
self.sv_pipeline = pipeline(
|
||||
task='speaker-verification',
|
||||
model='iic/speech_campplus_sv_zh-cn_16k-common',
|
||||
device='cpu',
|
||||
)
|
||||
speech_conditioning_latent = torch.tensor(
|
||||
self.sv_pipeline([wav], output_emb=True)['embs']
|
||||
).to(text_inputs.device)
|
||||
else:
|
||||
raise ValueError("campplus mode requires campplus_embedding or wav")
|
||||
speech_conditioning_latent = self.spk_emb_proj(speech_conditioning_latent)
|
||||
if speech_conditioning_latent.ndim != 3:
|
||||
speech_conditioning_latent = speech_conditioning_latent.unsqueeze(0)
|
||||
else:
|
||||
speech_conditioning_latent = self.get_conditioning(speech_condition.transpose(1,2), cond_lengths)
|
||||
speech_conditioning_latent = self.get_conditioning(speech_condition.transpose(1,2), cond_lengths)
|
||||
if emo_vec is None:
|
||||
print('compute emo vec')
|
||||
emo_vec = self.get_emo_conditioning(emo_speech_condition.transpose(1,2), emo_cond_lengths)
|
||||
@@ -792,18 +745,11 @@ class UnifiedVoice(nn.Module):
|
||||
else:
|
||||
print('Use the specified emotion vector')
|
||||
|
||||
if self.spk_cond_mode == "campplus":
|
||||
padding = torch.zeros(
|
||||
speech_conditioning_latent.size(0), 2, speech_conditioning_latent.size(2),
|
||||
device=speech_conditioning_latent.device,
|
||||
)
|
||||
conds_latent = torch.cat((speech_conditioning_latent + emo_vec.unsqueeze(1), padding), 1)
|
||||
else:
|
||||
tmp = torch.zeros(text_inputs.size(0)).to(text_inputs.device)
|
||||
duration_emb = self.speed_emb(torch.zeros_like(tmp).long())
|
||||
duration_emb_half = self.speed_emb(torch.ones_like(tmp).long())
|
||||
conds_latent = torch.cat((speech_conditioning_latent + emo_vec.unsqueeze(1), duration_emb_half.unsqueeze(1), duration_emb.unsqueeze(1)), 1)
|
||||
input_ids, inputs_embeds, attention_mask = self.prepare_gpt_inputs(conds_latent, text_inputs, langs)
|
||||
tmp = torch.zeros(text_inputs.size(0)).to(text_inputs.device)
|
||||
duration_emb = self.speed_emb(torch.zeros_like(tmp).long())
|
||||
duration_emb_half = self.speed_emb(torch.ones_like(tmp).long())
|
||||
conds_latent = torch.cat((speech_conditioning_latent + emo_vec.unsqueeze(1), duration_emb_half.unsqueeze(1), duration_emb.unsqueeze(1)), 1)
|
||||
input_ids, inputs_embeds, attention_mask = self.prepare_gpt_inputs(conds_latent, text_inputs)
|
||||
self.inference_model.store_mel_emb(inputs_embeds)
|
||||
if input_tokens is None:
|
||||
inputs = input_ids
|
||||
|
||||
@@ -662,8 +662,12 @@ def load_state_dict(
|
||||
):
|
||||
extra_args = {"mmap": True}
|
||||
weights_only_kwarg = {"weights_only": weights_only} if is_torch_greater_or_equal_than_1_13 else {}
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
return torch.load(checkpoint_file, map_location=map_location, **{**weights_only_kwarg, **extra_args, "weights_only": True})
|
||||
return torch.load(
|
||||
checkpoint_file,
|
||||
map_location=map_location,
|
||||
**weights_only_kwarg,
|
||||
**extra_args,
|
||||
)
|
||||
except Exception as e:
|
||||
try:
|
||||
with open(checkpoint_file) as f:
|
||||
|
||||
@@ -111,8 +111,7 @@ class IndexTTS:
|
||||
self.use_cuda_kernel = False
|
||||
self.bigvgan = Generator(self.cfg.bigvgan, use_cuda_kernel=self.use_cuda_kernel)
|
||||
self.bigvgan_path = os.path.join(self.model_dir, self.cfg.bigvgan_checkpoint)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
vocoder_dict = torch.load(self.bigvgan_path, map_location="cpu", weights_only=True)
|
||||
vocoder_dict = torch.load(self.bigvgan_path, map_location="cpu")
|
||||
self.bigvgan.load_state_dict(vocoder_dict["generator"])
|
||||
self.bigvgan = self.bigvgan.to(self.device)
|
||||
# remove weight norm on eval mode
|
||||
|
||||
@@ -192,17 +192,6 @@ class IndexTTS2:
|
||||
self.gpt.eval()
|
||||
print(">> GPT weights restored from:", self.gpt_path)
|
||||
|
||||
if use_deepspeed:
|
||||
try:
|
||||
import transformers
|
||||
transformers_major = int(transformers.__version__.split('.', 1)[0])
|
||||
except (ImportError, ValueError, AttributeError):
|
||||
transformers_major = None
|
||||
if transformers_major is not None and transformers_major >= 5:
|
||||
use_deepspeed = False
|
||||
print("⚠️ DeepSpeed kernel injection is not supported by the bundled IndexTTS GPT2 path with Transformers 5.x")
|
||||
print(">> Falling back to standard IndexTTS-2 inference (DeepSpeed disabled)")
|
||||
|
||||
if use_deepspeed:
|
||||
try:
|
||||
import deepspeed
|
||||
@@ -210,15 +199,7 @@ class IndexTTS2:
|
||||
use_deepspeed = False
|
||||
print(f">> Failed to load DeepSpeed. Falling back to normal inference. Error: {e}")
|
||||
|
||||
try:
|
||||
self.gpt.post_init_gpt2_config(use_deepspeed=use_deepspeed, kv_cache=True, half=self.use_fp16)
|
||||
except Exception as e:
|
||||
if not use_deepspeed:
|
||||
raise
|
||||
use_deepspeed = False
|
||||
print(f"⚠️ DeepSpeed initialization failed: {e}")
|
||||
print(">> Falling back to standard IndexTTS-2 inference (DeepSpeed disabled)")
|
||||
self.gpt.post_init_gpt2_config(use_deepspeed=False, kv_cache=True, half=self.use_fp16)
|
||||
self.gpt.post_init_gpt2_config(use_deepspeed=use_deepspeed, kv_cache=True, half=self.use_fp16)
|
||||
|
||||
if self.use_cuda_kernel:
|
||||
# preload the CUDA kernel for BigVGAN
|
||||
@@ -338,8 +319,7 @@ class IndexTTS2:
|
||||
campplus_ckpt_path = os.path.join(campplus_path, "campplus_cn_common.bin")
|
||||
|
||||
campplus_model = CAMPPlus(feat_dim=80, embedding_size=192)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
campplus_model.load_state_dict(torch.load(campplus_ckpt_path, map_location="cpu", weights_only=True))
|
||||
campplus_model.load_state_dict(torch.load(campplus_ckpt_path, map_location="cpu"))
|
||||
self.campplus_model = campplus_model.to(self.load_device)
|
||||
self.campplus_model.eval()
|
||||
print(">> campplus_model weights restored from:", campplus_ckpt_path)
|
||||
@@ -365,13 +345,11 @@ class IndexTTS2:
|
||||
self.tokenizer = TextTokenizer(self.bpe_path, self.normalizer)
|
||||
print(">> bpe model loaded from:", self.bpe_path)
|
||||
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
emo_matrix = torch.load(os.path.join(self.model_dir, self.cfg.emo_matrix), weights_only=True)
|
||||
emo_matrix = torch.load(os.path.join(self.model_dir, self.cfg.emo_matrix))
|
||||
self.emo_matrix = emo_matrix.to(self.load_device)
|
||||
self.emo_num = list(self.cfg.emo_num)
|
||||
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
spk_matrix = torch.load(os.path.join(self.model_dir, self.cfg.spk_matrix), weights_only=True)
|
||||
spk_matrix = torch.load(os.path.join(self.model_dir, self.cfg.spk_matrix))
|
||||
self.spk_matrix = spk_matrix.to(self.load_device)
|
||||
|
||||
self.emo_matrix = torch.split(self.emo_matrix, self.emo_num)
|
||||
@@ -904,8 +882,7 @@ class IndexTTS2:
|
||||
cond_lengths=torch.tensor([spk_cond_emb.shape[-1]], device=text_tokens.device),
|
||||
emo_cond_lengths=torch.tensor([emo_cond_emb.shape[-1]], device=text_tokens.device),
|
||||
emo_vec=emovec,
|
||||
# TTS Audio Suite patch: Honor the engine node's sampling control.
|
||||
do_sample=do_sample,
|
||||
do_sample=True,
|
||||
top_p=top_p,
|
||||
top_k=top_k,
|
||||
temperature=temperature,
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,4 +1,3 @@
|
||||
from utils.security.data_loading import load_numpy_data # TTS Audio Suite patch: NumPy dictionaries must not execute pickle globals.
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
@@ -46,7 +45,7 @@ class DACFile:
|
||||
|
||||
@classmethod
|
||||
def load(cls, path):
|
||||
artifacts = load_numpy_data(path)[()]
|
||||
artifacts = np.load(path, allow_pickle=True)[()]
|
||||
codes = torch.from_numpy(artifacts["codes"].astype(int))
|
||||
if artifacts["metadata"].get("dac_version", None) not in SUPPORTED_VERSIONS:
|
||||
raise RuntimeError(
|
||||
|
||||
@@ -478,8 +478,7 @@ class BigVGAN(
|
||||
local_files_only=local_files_only,
|
||||
)
|
||||
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint_dict = torch.load(model_file, map_location=map_location, weights_only=True)
|
||||
checkpoint_dict = torch.load(model_file, map_location=map_location)
|
||||
|
||||
try:
|
||||
model.load_state_dict(checkpoint_dict["generator"])
|
||||
|
||||
@@ -61,8 +61,7 @@ def get_padding(kernel_size, dilation=1):
|
||||
def load_checkpoint(filepath, device):
|
||||
assert os.path.isfile(filepath)
|
||||
print(f"Loading '{filepath}'")
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint_dict = torch.load(filepath, map_location=device, weights_only=True)
|
||||
checkpoint_dict = torch.load(filepath, map_location=device)
|
||||
print("Complete.")
|
||||
return checkpoint_dict
|
||||
|
||||
|
||||
@@ -220,8 +220,7 @@ def load_F0_models(path):
|
||||
from .JDC.model import JDCNet
|
||||
|
||||
F0_model = JDCNet(num_class=1, seq_len=192)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
params = torch.load(path, map_location="cpu", weights_only=True)["net"]
|
||||
params = torch.load(path, map_location="cpu")["net"]
|
||||
F0_model.load_state_dict(params)
|
||||
_ = F0_model.train()
|
||||
|
||||
@@ -529,8 +528,7 @@ def load_checkpoint(
|
||||
is_distributed=False,
|
||||
load_ema=False,
|
||||
):
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
state = torch.load(path, map_location="cpu", weights_only=True)
|
||||
state = torch.load(path, map_location="cpu")
|
||||
params = state["net"]
|
||||
if load_ema and "ema" in state:
|
||||
print("Loading EMA")
|
||||
@@ -587,8 +585,7 @@ def load_checkpoint2(
|
||||
is_distributed=False,
|
||||
load_ema=False,
|
||||
):
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
state = torch.load(path, map_location="cpu", weights_only=True)
|
||||
state = torch.load(path, map_location="cpu")
|
||||
params = state["net"]
|
||||
if load_ema and "ema" in state:
|
||||
print("Loading EMA")
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user