Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5a28a46010 | ||
|
|
211b192f4a | ||
|
|
4a99f15851 |
@@ -1,14 +0,0 @@
|
||||
# Registry archives contain runtime files and user documentation.
|
||||
# These exclusions do not remove anything from the Git repository.
|
||||
/.github/
|
||||
/tests/
|
||||
/scripts/
|
||||
/PROJECT_INDEX.md
|
||||
/AGENTS.md
|
||||
/CLAUDE.md
|
||||
/docs/BUMP_SCRIPT_INSTRUCTIONS.md
|
||||
|
||||
# Keep the metadata YAML files: the auxiliary registry reads its YAML at runtime.
|
||||
/docs/Dev reports/*
|
||||
!/docs/Dev reports/tts_audio_suite_engines.yaml
|
||||
!/docs/Dev reports/tts_audio_suite_aux_models.yaml
|
||||
@@ -14,9 +14,6 @@ jobs:
|
||||
steps:
|
||||
- name: Check out code
|
||||
uses: actions/checkout@v4
|
||||
- name: Preserve Unicode filenames in Registry archives
|
||||
# comfy-cli reads git ls-files as plain paths; Git must not escape emoji.
|
||||
run: git config core.quotepath false
|
||||
- name: Publish Custom Node
|
||||
uses: Comfy-Org/publish-node-action@main
|
||||
with:
|
||||
|
||||
-168
@@ -5,174 +5,6 @@ All notable changes to this project will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [5.10.0] - 2026-10-09
|
||||
|
||||
### Added
|
||||
|
||||
- Shared Runtime installation controls and suite security improvements
|
||||
- Add automatic Shared Runtime setup during TTS Audio Suite installation and updates, with a ComfyUI setting and file option to skip it.
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve runtime behavior so workflows use prepared environments without installing Python packages.
|
||||
- Improve file access and model loading across TTS, voice conversion, audio tools, and training to reject unsafe paths and executable checkpoint objects while keeping supported weight formats.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix Registry packages missing guides and workflows with Unicode filenames, and exclude development-only files from published packages.
|
||||
|
||||
### Removed
|
||||
|
||||
- Remove Dedicated Runtime choices from Qwen3-TTS and Step Audio EditX; existing workflows migrate to Shared Runtime without moving other settings.
|
||||
## [5.9.3] - 2026-10-08
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix isolated TTS worker startup failures from conflicting Python modules
|
||||
- Fix Qwen3-TTS and Voice Design failing on first use or after Step Audio EditX with utils is not a package
|
||||
- Fix conflicting module paths affecting all isolated TTS and ASR workers
|
||||
- Fix isolated workers losing ComfyUI imports when their working directory changes
|
||||
- Fix Step Audio EditX importing bundled modules during model discovery
|
||||
- Fix Step Audio EditX CPU and MPS device patch not targeting the bundled encoder
|
||||
## [5.9.2] - 2026-10-04
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix Classic ChatterBox crashing during short or consecutive speech segments (PR #368, thanks @melayyoub)
|
||||
- Fix Classic ChatterBox short segments being replaced with silence by the old workaround
|
||||
- Fix ChatterBox Classic and Official 23-Lang unclear failures when generation produces no usable speech; report a clear error instead
|
||||
|
||||
### Removed
|
||||
|
||||
- Remove Classic ChatterBox's old protection template and automatic short-text padding from Text and SRT; existing workflows remain compatible
|
||||
## [5.9.1] - 2026-10-04
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix ChatterBox Classic SRT cache reuse during character and language switching
|
||||
- Fix unnecessary language model loading when ChatterBox Classic SRT speech is already cached
|
||||
- Fix ChatterBox Classic pause processing ignoring the disabled audio-cache setting
|
||||
## [5.9.0] - 2026-10-03
|
||||
|
||||
### Added
|
||||
|
||||
- Add Step Audio EditX checkpoint selection and expanded sound editing
|
||||
- Add dated 2026-01-23 and 2025-11-28 Step Audio EditX model choices with separate downloads
|
||||
- Add expanded Step Audio EditX sound tags, including giggle, chuckle, clears throat, and new vocal reactions
|
||||
- Add new Step Audio EditX sounds plus fear and humour emotions to the Multiline TTS Tag Editor
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve Step Audio EditX model documentation and editor help while preserving existing workflow widget order and installed weights
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix Step Audio EditX Audio Editor ignoring the connected engine's selected model
|
||||
- Fix stale Step Audio EditX edits when changing models or generation settings
|
||||
- Fix ChatterBox v2/v3 native sound tags being treated as Step Audio EditX edits
|
||||
## [5.8.11] - 2026-09-19
|
||||
|
||||
### Added
|
||||
|
||||
- Add Multiline TTS Tag Editor globe language picker
|
||||
- Add an animated globe for choosing and swapping languages
|
||||
- Add realistic simplified continent outlines and smooth travel between language locations
|
||||
- Add square language buttons with a resizable layout that remembers its size
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve language selection from the node's left panel
|
||||
## [5.8.10] - 2026-09-04
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix Step Audio EditX failing when its speech recognition model is missing
|
||||
- Remove unused dependency constraints that could break installation on newer Python versions
|
||||
- Improve Echo-TTS and FAISS dependency handling through the extension installer
|
||||
## [5.8.9] - 2026-09-03
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix TTS model discovery when extra_model_paths.yaml defines a shared TTS directory
|
||||
- Fix local ComfyUI models/TTS installations being hidden by shared storage configuration.
|
||||
- Fix download path ordering so the configured default directory remains preferred.
|
||||
## [5.8.8] - 2026-08-31
|
||||
|
||||
### Added
|
||||
|
||||
- Clarify that parallel workers are available only for ChatterBox Classic
|
||||
- Prevent unsupported TTS engines from entering experimental parallel processing
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve legacy ChatterBox parallel-worker guidance
|
||||
- Explain that the option is not true model batching and may not improve speed
|
||||
## [5.8.7] - 2026-08-28
|
||||
|
||||
### Added
|
||||
|
||||
- Add native Transformers 5 support for VibeVoice and KugelAudio
|
||||
- Run VibeVoice and KugelAudio directly in the main ComfyUI environment
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix VibeVoice 1.5B and KugelAudio model loading and generation on current ComfyUI installations
|
||||
- Remove misleading compatibility warnings during VibeVoice loading and generation
|
||||
|
||||
### Removed
|
||||
|
||||
- Remove the VibeVoice runtime isolation setting
|
||||
## [5.8.6] - 2026-08-28
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve Vocal Removal cache hashing without temporary WAV encoding
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix Vocal Removal crashing before audio separation on systems with incompatible SoundFile virtual I/O
|
||||
## [5.8.5] - 2026-08-26
|
||||
|
||||
### Added
|
||||
|
||||
- Add support for loading RVC models from nested folders
|
||||
- Add automatic matching for index files stored beside nested RVC models
|
||||
- Preserve support for the existing shared RVC index folder
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve RVC character model organization
|
||||
## [5.8.4] - 2026-08-21
|
||||
|
||||
### Added
|
||||
|
||||
- Keep the example workflow reliable by disabling DeepSpeed by default
|
||||
- Automatically use standard inference when the optional DeepSpeed path is unsupported
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix IndexTTS DeepSpeed failures
|
||||
## [5.8.3] - 2026-08-18
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve long-running ChatterBox session stability
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix ChatterBox generations degrading after extended use
|
||||
- Fix English ChatterBox returning very short audio after many generations
|
||||
- Fix Official 23-Lang v1/v2 accumulating background processing hooks
|
||||
## [5.8.2] - 2026-08-18
|
||||
|
||||
### Added
|
||||
|
||||
- Keep trained adapters visible and selectable with the matching community model
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix MOSS Voice Acting LoRA selection
|
||||
- Fix LoRA controls appearing unavailable when the LAION Voice Acting 8B model is selected
|
||||
## [5.8.1] - 2026-08-11
|
||||
|
||||
### Added
|
||||
|
||||
+6
-21
@@ -17,7 +17,7 @@
|
||||
**Key architectural rules:**
|
||||
- Chunking happens in the **processor**, not the adapter (`generate_single()` on adapter = raw single call)
|
||||
- Runtime routing happens through `ModelLoadConfig.runtime_mode` + `runtime_profile`, not ad-hoc subprocess calls
|
||||
- Shared runtime workers are currently used for fragile engine families such as Qwen3-TTS / ASR, Granite forced alignment, and Higgs Audio 2. VibeVoice/Kugel and other engines that support the modern stack run natively in the main Transformers 5 environment.
|
||||
- Shared runtime workers are currently used for fragile engine families such as VibeVoice, Qwen3-TTS / ASR, Granite forced alignment, and Higgs Audio 2. Engines that support the modern stack run natively in the main Transformers 5 environment.
|
||||
- YAML (`docs/Dev reports/tts_audio_suite_engines.yaml`) is source of truth for engine doc tables → run `python3 scripts/generate_engine_tables.py --readme` to regenerate
|
||||
- Auxiliary YAML (`docs/Dev reports/tts_audio_suite_aux_models.yaml`) is source of truth for helper/post-process model docs → run `python3 scripts/generate_aux_model_docs.py`
|
||||
- All models download to `ComfyUI/models/TTS/<model-name>/`
|
||||
@@ -78,7 +78,6 @@
|
||||
- `ENGINE_COMPARISON.md`, `LANGUAGE_SUPPORT.md`, `FEATURE_COMPARISON.md` - Auto-generated tables
|
||||
|
||||
### Dev Docs (`docs/Dev reports/`)
|
||||
- `REGISTRY_COMPATIBILITY_PLAN.md` - File access, safe loading, config parsing, runtime installation/preferences, archive fixes, verification and Registry approval evidence
|
||||
- `tts_audio_suite_engines.yaml` - **Source of truth** for all engine metadata
|
||||
- `tts_audio_suite_aux_models.yaml` - **Source of truth** for helper/post-process model metadata
|
||||
- `SRT_IMPLEMENTATION.md` - SRT timing technical details
|
||||
@@ -120,7 +119,7 @@
|
||||
- `nodes/subtitles/text_to_srt_builder_node.py` - Build SRT from transcript text plus timing data, or estimate timings from plain text
|
||||
- `nodes/subtitles/srt_advanced_options_node.py` - Subtitle readability / segmentation policy options
|
||||
- `nodes/text/tts_tag_editor_node.py` - 🏷️ Multiline TTS Tag Editor: rich text editor with character/language/parameter dropdowns, preset system, syntax highlighting, undo/redo — pairs with `web/string_multiline_tag_editor.js`
|
||||
- `nodes/step_audio_editx_special/step_audio_editx_audio_editor_node.py` - 🎨 Audio Editor: post-process ANY engine's audio with Step Audio EditX (emotion/style, paralinguistic effects, speed); dated checkpoint choices and tags documented in `docs/STEP_AUDIO_EDITX_CHECKPOINTS.md`
|
||||
- `nodes/step_audio_editx_special/step_audio_editx_audio_editor_node.py` - 🎨 Audio Editor: post-process ANY engine's audio with Step Audio EditX (14 emotions, 32 styles, paralinguistic effects like `<Laughter>`, speed control) — universal, not just for Step Audio EditX engine
|
||||
- `nodes/engines/index_tts_emotion_options_node.py` - IndexTTS-2 emotion radar chart
|
||||
|
||||
### Audio / Video Nodes
|
||||
@@ -137,19 +136,19 @@
|
||||
- `unified_model_interface.py` - Universal factory pattern for all engines
|
||||
- `engine_registry.py` - Engine capability definitions
|
||||
- `factory_config.py` - standardized model load config, runtime mode/profile normalization
|
||||
- `step_audio_editx_checkpoints.py` - Dated Step Audio EditX choices and pinned Hugging Face revisions
|
||||
- `manager.py` - Model discovery and caching
|
||||
- `comfyui_model_wrapper/` - ComfyUI native model management integration
|
||||
- `extra_paths.py` - extra_model_paths.yaml support
|
||||
|
||||
### Isolated Runtimes (`utils/runtimes/`)
|
||||
- `profiles.py` - named runtime profiles (shared/dedicated legacy Transformers 4 environments and engine-specific runtimes)
|
||||
- `launcher.py` - worker environment construction, suite-first import paths, Windows toolchain env setup
|
||||
- `profiles.py` - named runtime profiles (`vibevoice_transformers4_shared`, dedicated variants, etc.)
|
||||
- `launcher.py` - runtime bootstrap, venv creation, Windows toolchain env setup
|
||||
- `session.py`, `protocol.py` - JSONL worker transport and message protocol
|
||||
- `bootstrap.py` - shared runtime bootstrap helpers
|
||||
- `vibevoice_proxy.py`, `qwen3_tts_proxy.py`, `qwen3_asr_proxy.py`, `higgs_audio_proxy.py` - parent-process proxies
|
||||
- `workers/` - worker subprocess entrypoints for VibeVoice, Qwen3-TTS, Qwen3-ASR/aligner, Higgs Audio, Fish Audio S2, and Step Audio EditX; each prioritizes suite imports before third-party dependencies
|
||||
- `workers/` - worker subprocess entrypoints for VibeVoice, Qwen3-TTS, Qwen3-ASR/aligner, Higgs Audio
|
||||
- Current shared legacy T4 runtime profile is reused by:
|
||||
- VibeVoice / Kugel
|
||||
- Qwen3-TTS
|
||||
- Qwen3-ASR and Granite's optional Qwen forced aligner
|
||||
- Higgs Audio 2
|
||||
@@ -204,20 +203,6 @@
|
||||
- `web/asr_srt_preset_widgets.js` - ASR SRT preset locking
|
||||
|
||||
## Scripts & Config
|
||||
- `.comfyignore` - Registry archive exclusions for development tools and reports; keeps runtime metadata YAML, user documentation, and engine/training code
|
||||
- `scripts/bump_version_enhanced.py` - Version bump with changelog (use `patch`/`minor`/`major`)
|
||||
- `scripts/generate_engine_tables.py` - Regenerate all docs from YAML (`--readme` flag for README too)
|
||||
- `requirements.txt`, `pyproject.toml` - Dependencies and project metadata
|
||||
|
||||
## Registry compatibility helpers
|
||||
|
||||
- `utils/security/path_access.py` - Configured data-folder containment, link checks and safe output paths
|
||||
- `utils/security/data_loading.py` - Restricted checkpoint/dictionary/NumPy loading and scoped dependency adapters
|
||||
- `utils/security/demucs_loading.py`, `audio_separator_loading.py` - Preserve legacy separator metadata with restricted dependency loaders
|
||||
- `utils/downloads/unified_downloader.py` - Model download destinations contained in registered data folders
|
||||
- `utils/security/config_values.py` - Data-only config expressions with bounded list arithmetic
|
||||
- `utils/runtimes/bootstrap.py` - Runtime lookup/readiness only; generation never installs packages
|
||||
- `utils/runtimes/installation.py` - Installation-time preparation invoked by `install.py`
|
||||
- `utils/runtimes/settings.py`, `settings_api.py` - One persisted installation preference and same-origin UI writes
|
||||
- `web/runtime_installation_settings.js` - Shared runtime installation setting and status/cleanup details
|
||||
- `web/runtime_mode_compatibility.js` - Dedicated-to-shared migration for older workflows
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
[![Dynamic TOML Badge][version-shield]][version-url]
|
||||
[](https://ko-fi.com/diogogo)
|
||||
|
||||
# TTS Audio Suite v5.10.0
|
||||
# TTS Audio Suite v5.8.1
|
||||
|
||||
[](https://ko-fi.com/diogogo)
|
||||
|
||||
@@ -246,8 +246,8 @@ For comprehensive technical information, refer to the [SRT_IMPLEMENTATION.md](do
|
||||
This is the new architectural baseline for the suite.
|
||||
|
||||
* **Main environment moved forward**: the primary ComfyUI environment is now meant to run on **Transformers 5**
|
||||
* **Isolation for fragile engines**: engines that still behave better on the older stack can use **a shared** legacy runtime instead of forcing the whole suite backward
|
||||
* **Cleaner engine strategy**: modern engines such as **Higgs Audio v3**, **MOSS-TTS**, and other compatible stacks can stay native in the main environment
|
||||
* **Isolation for fragile engines**: engines that still behave better on the older stack can use **shared** or **dedicated** legacy runtimes instead of forcing the whole suite backward
|
||||
* **Cleaner engine strategy**: modern engines such as **Higgs Audio v3**, **Step Audio EditX**, **MOSS-TTS**, and other compatible stacks can stay native in the main environment
|
||||
* **Less dependency deadlock**: adding new engines no longer has to mean globally freezing the entire project to one old Transformers version
|
||||
|
||||
This matters because the suite now has a clearer split:
|
||||
@@ -429,7 +429,7 @@ dataset formats, training workflow, adapter loading, and CPU-safe preflight.
|
||||
3. Choose between Custom Character Switching (recommended) or Native Multi-Speaker mode
|
||||
4. Generate long-form content with automatic voice cloning from your voices folder
|
||||
|
||||
VibeVoice and KugelAudio run directly in the suite's Transformers 5 main environment.
|
||||
**Isolation note:** On current ComfyUI stacks, **Kugel/VibeVoice is usually meant to run with `⚠️ Runtime Isolation = Shared Runtime`**. If you force `Main Environment`, you are opting back into the dependency-conflict path.
|
||||
|
||||
**Perfect for:**
|
||||
|
||||
@@ -811,10 +811,9 @@ Welcome to our show! [Alice:happy_sarah] I'm so excited to be here!
|
||||
**NEW in v4.15**: Revolutionary LLM-based audio post-processing with emotion, style, and paralinguistic control!
|
||||
|
||||
* **🎨 Step Audio EditX - Audio Editor Node**: Post-process ANY TTS audio with advanced editing capabilities
|
||||
* **📅 Dated Checkpoints**: Choose `2026-01-23` (default for new engine nodes) or `2025-11-28` in the existing model dropdown. Downloads use pinned revisions and separate folders; existing unversioned/local selections keep their files. [Checkpoint guide](docs/STEP_AUDIO_EDITX_CHECKPOINTS.md).
|
||||
* **🗣️ Paralinguistic Effects**: Original tags plus January sounds such as `<giggle>`, `<chuckle>`, `<inhale>`, `<exhale>`, `<cough>`, and `<clears_throat>`; iteration syntax remains available, e.g. `<giggle:2>`.
|
||||
* **😊 Emotion Controls**: happy, sad, angry, excited, fear, humour, surprised, disgusted, confusion, empathy, embarrass, depressed, coldness, admiration; older `calm` and `fearful` options remain accepted
|
||||
* **🎭 Speaking Styles**: whisper, serious, child, older, girl, pure, sister, sweet, exaggerated, ethereal, generous, recite, act_coy, warm, shy, comfort, authority, chat, radio, soulful, gentle, story, vivid, program, news, advertising, roar, murmur, shout, deeply, loudly, arrogant, friendly
|
||||
* **🗣️ Paralinguistic Effects**: Insert natural sounds - Laughter, Breathing, Sigh, Uhm, Surprise (oh/ah/wa), Confirmation, Question, Dissatisfaction
|
||||
* **😊 14 Emotion Controls**: happy, sad, angry, excited, calm, fearful, surprised, disgusted, confusion, empathy, embarrass, depressed, coldness, admiration
|
||||
* **🎭 32 Speaking Styles**: whisper, serious, child, older, girl, pure, sister, sweet, exaggerated, ethereal, generous, recite, act_coy, warm, shy, comfort, authority, chat, radio, soulful, gentle, story, vivid, program, news, advertising, roar, murmur, shout, deeply, loudly, arrogant, friendly
|
||||
* **⚡ Speed Control**: faster, slower, more_faster, more_slower with multi-iteration support
|
||||
* **🔊 Voice Restoration**: ChatterBox VC integration to restore original voice resemblance after editing
|
||||
* **🏷️ Inline Edit Tags**: Apply effects directly in text using `<Laughter:2>`, `<emotion:happy>`, `<style:whisper>` tags
|
||||
@@ -1218,31 +1217,6 @@ Perfect for:
|
||||
- 🟢 **OpenSeeFace mouth movement**: ✅ Working (experimental)
|
||||
- 🔴 **MediaPipe mouth movement**: ❌ Incompatible (use OpenSeeFace)
|
||||
|
||||
### Shared Runtime installation
|
||||
|
||||
**Most users can keep the defaults.** Manager prepares one Shared Runtime during installation or updates. Qwen3-TTS/ASR, Step Audio EditX, Higgs Audio 2, and forced alignment reuse it. Model weights download only when needed. Running a workflow or restarting ComfyUI does not install Python packages.
|
||||
|
||||
If an engine reports that Shared Runtime is missing or needs repair, enable **Install shared runtime automatically** under **Settings > TTS Audio Suite > Runtime installation**, then repair/reinstall TTS Audio Suite through Manager. Restart when Manager requests it.
|
||||
|
||||
<details>
|
||||
<summary>Advanced: skip installation or remove an unused shared runtime</summary>
|
||||
|
||||
Turn off **Install shared runtime automatically** in the setting above to skip preparation during future installs and updates. Engines still set to Shared Runtime can use an existing environment; the setting does not switch engines to Main Environment.
|
||||
|
||||
Open **TTS Audio Suite > Runtime installation details** from ComfyUI's menu to see the configuration file, runtime folder, and installation status. To reclaim the runtime's space, turn off automatic installation, close ComfyUI, and manually delete the displayed **shared_legacy_t4** folder. Model weights and voice files are separate. Engines set to Shared Runtime need that environment restored before they can run again.
|
||||
|
||||
To skip the first installation, create this file before installing the suite:
|
||||
|
||||
**ComfyUI/user/__tts_audio_suite/runtime_settings.json**
|
||||
|
||||
```json
|
||||
{"install_shared_runtime": false}
|
||||
```
|
||||
|
||||
The UI and installer read the same file. With a custom ComfyUI user directory, use its **__tts_audio_suite** folder. After the suite has loaded, it remembers that location for Manager updates and reinstalls. Before the first installation with a custom user directory, set **TTS_AUDIO_SUITE_USER_DIRECTORY** to that directory in the installer's environment.
|
||||
|
||||
</details>
|
||||
|
||||
### Option 2: Manual Installation
|
||||
|
||||
**Same intelligent installer, manual setup:**
|
||||
@@ -1442,12 +1416,7 @@ Core TTS installation will continue; only the listed features may be unavailable
|
||||
|
||||
A common problem is installing dependencies in the wrong Python environment. Always ensure you are installing dependencies within your ComfyUI's Python environment.
|
||||
|
||||
Where an engine offers **⚠️ Runtime Isolation**, there are two choices:
|
||||
|
||||
- **Main Environment** uses ComfyUI's installed Python packages.
|
||||
- **⚠️ Shared Runtime** uses one support environment shared by compatible engines, helping avoid dependency conflicts. Keep the engine's default unless you need to change it.
|
||||
|
||||
Older workflows that selected Dedicated Runtime now use Shared Runtime. Their other settings stay in place. Existing dedicated folders are kept; after closing ComfyUI, you can delete an unused dedicated folder to reclaim its space. Keep the **shared_legacy_t4** folder if you use Shared Runtime.
|
||||
If the engine comparison table shows **`Shared`** or **`Dedicated`** in the **Isolation** column, that engine has its own secondary-environment path for dependency conflicts. Configure that on the engine node with `⚠️ Runtime Isolation` instead of trying to downgrade your main ComfyUI environment.
|
||||
|
||||
* **Verify your Python environment:** After activating your venv or navigating to your portable ComfyUI installation, check the Python executable being used:
|
||||
|
||||
@@ -1565,7 +1534,7 @@ For offline/manual setup:
|
||||
| VibeVoice | `ComfyUI/models/TTS/VibeVoice/` | ✅ | 1.5B and 7B variants |
|
||||
| RVC | `ComfyUI/models/TTS/RVC/` | ✅* | Base models auto; character `.pth` can be user-provided |
|
||||
| IndexTTS-2 | `ComfyUI/models/TTS/IndexTTS/` | ✅ | Emotion components included |
|
||||
| Step Audio EditX | `ComfyUI/models/TTS/step_audio_editx/` | ✅ | Lazy dated checkpoints + tokenizer stack; unversioned installs retained |
|
||||
| Step Audio EditX | `ComfyUI/models/TTS/step_audio_editx/` | ✅ | Main model + tokenizer stack |
|
||||
| CosyVoice3 | `ComfyUI/models/TTS/CosyVoice/` | ✅ | Variant-specific lazy downloads |
|
||||
| Qwen3-TTS / ASR | `ComfyUI/models/TTS/qwen3_tts/` | ✅ | Per-variant download + shared tokenizer |
|
||||
| MOSS-TTS | `ComfyUI/models/TTS/moss_tts/` | ✅ | Local/Delay/VoiceGenerator/SoundEffect v1/TTSD models plus shared MOSS-Audio-Tokenizer codec |
|
||||
|
||||
+6
-10
@@ -344,7 +344,6 @@ def setup_api_routes():
|
||||
import folder_paths
|
||||
from server import PromptServer
|
||||
from aiohttp import web
|
||||
from utils.security.path_access import resolve_input_path, filename_component, child_path
|
||||
|
||||
def _get_ui_data_dir():
|
||||
base_dir = os.path.join(folder_paths.get_system_user_directory("tts_audio_suite"), "ui")
|
||||
@@ -356,8 +355,8 @@ def setup_api_routes():
|
||||
|
||||
from utils.voice.alias_api import register_character_alias_routes
|
||||
register_character_alias_routes(PromptServer.instance.routes, web)
|
||||
from utils.runtimes.settings_api import register_runtime_settings_routes
|
||||
register_runtime_settings_routes(PromptServer.instance.routes, web)
|
||||
from utils.audio_cpp.capability_api import register_audio_cpp_capability_routes
|
||||
register_audio_cpp_capability_routes(PromptServer.instance.routes, web)
|
||||
|
||||
@PromptServer.instance.routes.get("/api/tts-audio-suite/index-tts-emotion-presets")
|
||||
async def get_index_tts_emotion_presets_endpoint(request):
|
||||
@@ -644,8 +643,7 @@ print(json.dumps({"devices": devices}))
|
||||
if not audio_file:
|
||||
return web.json_response({"error": "audio_file is required for preview analysis"}, status=400)
|
||||
|
||||
audio_file = resolve_input_path(audio_file)
|
||||
node_id = filename_component(data.get("node_id") or "preview")
|
||||
node_id = str(data.get("node_id") or "preview")
|
||||
|
||||
analyzer_node_path = os.path.join(os.path.dirname(__file__), "nodes", "audio", "analyzer_node.py")
|
||||
spec = importlib.util.spec_from_file_location("tts_audio_suite_audio_analyzer_node", analyzer_node_path)
|
||||
@@ -653,7 +651,7 @@ print(json.dumps({"devices": devices}))
|
||||
spec.loader.exec_module(analyzer_node_module)
|
||||
|
||||
analyzer_node = analyzer_node_module.AudioAnalyzerNode()
|
||||
analysis_result = analyzer_node.analyze_audio(
|
||||
analyzer_node.analyze_audio(
|
||||
audio_file=audio_file,
|
||||
analysis_method=data.get("analysis_method", "silence"),
|
||||
precision_level=data.get("precision_level", "milliseconds"),
|
||||
@@ -667,9 +665,7 @@ print(json.dumps({"devices": devices}))
|
||||
)
|
||||
|
||||
import folder_paths
|
||||
if analysis_result[1].startswith("Error:"):
|
||||
return web.json_response({"error": analysis_result[1]}, status=400)
|
||||
cache_file = child_path(folder_paths.get_output_directory(), f"audio_analyzer_cache_{node_id}.json")
|
||||
cache_file = os.path.join(folder_paths.get_output_directory(), f"audio_analyzer_cache_{node_id}.json")
|
||||
with open(cache_file, "r", encoding="utf-8") as f:
|
||||
payload = json.load(f)
|
||||
|
||||
@@ -678,7 +674,7 @@ print(json.dumps({"devices": devices}))
|
||||
return response
|
||||
except Exception as e:
|
||||
print(f"⚠️ Audio analyzer preview failed: {e}")
|
||||
return web.json_response({"error": str(e)}, status=400 if isinstance(e, (ValueError, FileNotFoundError)) else 500)
|
||||
return web.json_response({"error": str(e)}, status=500)
|
||||
|
||||
@PromptServer.instance.routes.get("/api/tts-audio-suite/training-progress")
|
||||
async def get_training_progress_endpoint(request):
|
||||
|
||||
@@ -1,9 +1,5 @@
|
||||
# Isolated Runtimes Plan
|
||||
|
||||
Historical design. The current implementation uses Main Environment or one Shared Runtime,
|
||||
prepared by install.py. Dedicated profiles and first-use package installation were removed;
|
||||
see [Registry compatibility work](REGISTRY_COMPATIBILITY_PLAN.md).
|
||||
|
||||
## Goal
|
||||
|
||||
Keep ComfyUI in one modern main runtime while routing fragile engines into
|
||||
|
||||
@@ -1,248 +0,0 @@
|
||||
# Registry compatibility work
|
||||
|
||||
Scope implemented locally on 2026-10-09: file containment, restricted data loading,
|
||||
configuration parsing, installation-time Shared Runtime setup, user preferences,
|
||||
dedicated-runtime migration, and Registry archive cleanup. Local verification is recorded below.
|
||||
These verification records do not constitute Registry approval.
|
||||
|
||||
## 1. File access
|
||||
|
||||
- Restrict workflow and HTTP-supplied file paths to the configured ComfyUI
|
||||
input/output/temp or explicitly registered model/voice folders, as appropriate
|
||||
for the operation. Resolve links before checking containment.
|
||||
- Keep the audio analyzer's existing widget and browser upload/drag-and-drop.
|
||||
Uploaded audio is already placed in input by ComfyUI's upload endpoint.
|
||||
- Reject paths outside the permitted folders. Copying an arbitrary server-side
|
||||
path into input would still read that path and expose its contents.
|
||||
- Audit the analyzer preview route and node execution together, plus the suite's
|
||||
other file-backed widgets and routes. Protect generated cache/output names
|
||||
from raw node IDs and other request values.
|
||||
- Preserve explicitly configured external model and voice directories; do not
|
||||
turn their normal use into an unrestricted path exception.
|
||||
- Restrict model download destinations before network access, writes, or replacement;
|
||||
reject traversal and links escaping the permitted data roots. Custom-node code
|
||||
folders are excluded from data roots even though ComfyUI registers them.
|
||||
|
||||
## 2. Checkpoint and dictionary loading
|
||||
|
||||
- Prefer restricted PyTorch loading for state dictionaries; keep existing model
|
||||
formats where safe loading supports them. Do not require conversion or copies
|
||||
of upstream weights as the default solution.
|
||||
- The local format audit covered 52 PyTorch files and one plain-pickle language
|
||||
dictionary: 50 checkpoints opened directly with weights_only=True, two Dots
|
||||
latent-statistics files needed a scoped NumPy type allowlist, and the dictionary
|
||||
loaded with a data-only unpickler that refuses class/global lookups.
|
||||
- The audit used the configured test interpreter with Torch 2.10.0+cu130. It did
|
||||
not establish full engine inference, every remote variant, TorchScript safety,
|
||||
or compatibility with old whole-model HuBERT files and all training resumes.
|
||||
- Review RVC inference/separation/training loaders, DramaBox training, Dots
|
||||
integration, optional Russian stress dictionaries, and remaining unsafe
|
||||
helpers. No automatic fallback to unrestricted loading for an untrusted file.
|
||||
- Keep dependency patches scoped to suite integration rather than changing
|
||||
unrelated installed packages or globally weakening deserialization.
|
||||
|
||||
## 3. Configuration expressions
|
||||
|
||||
- Replace the Qwen tokenizer's constant eval with its literal value.
|
||||
- Replace bundled FunASR configuration eval calls with restricted data parsing.
|
||||
Preserve required list/tuple expressions, including existing list addition and
|
||||
repetition; literal_eval alone does not cover those expressions.
|
||||
- Keep neural-network model.eval() calls. Add the required adjacent TTS Audio
|
||||
Suite patch comments for bundled-code changes.
|
||||
|
||||
## 4. Optional isolated environments
|
||||
|
||||
The maintainer rejected a manual setup command as the normal user experience.
|
||||
Required behavior: users get automatic setup or a clear install action inside
|
||||
ComfyUI, without editing files, running terminal commands, or learning about
|
||||
Python environments. Keep existing workflow widgets and their order. Generation now checks prepared-runtime metadata and never installs packages.
|
||||
The installer prepares or reuses Shared Runtime independently of its main-dependency fast path.
|
||||
|
||||
Implemented: prepare the single shared legacy environment automatically from the existing install.py hook during normal
|
||||
Manager installation/update. Default Qwen3-TTS, Qwen ASR, Step Audio EditX, and
|
||||
Higgs Audio 2 already select this environment. Users keep the existing experience
|
||||
without terminal setup or additional support packages. Model weights still
|
||||
download only when the corresponding engine is used.
|
||||
|
||||
The local shared environment measured 139.1 MB (132.7 MiB), including 6,429
|
||||
files. It inherits the main environment's heavy packages rather than installing
|
||||
another PyTorch copy. This measures the current machine, not download size or
|
||||
a universal upper bound; missing/incompatible host dependencies can change it.
|
||||
|
||||
Keep an advanced opt-out for automatic shared-environment preparation. A local
|
||||
installer configuration can apply before first installation; a ComfyUI setting
|
||||
can control subsequent installation/update/repair runs. Skipping preparation
|
||||
must neither delete an existing environment nor silently switch engines to an
|
||||
incompatible main environment. A missing required environment should report
|
||||
that support must be enabled and installation repaired through Manager.
|
||||
|
||||
### Implemented advanced controls
|
||||
|
||||
- Add a default-on "Install shared runtime automatically" setting under
|
||||
ComfyUI Settings > TTS Audio Suite > Runtime installation.
|
||||
- Store the preference in one server-side JSON file read by the installer.
|
||||
The UI edits that same file and shows its actual location; a hand-edited
|
||||
preference must not be overwritten with stale browser defaults on page load.
|
||||
Prefer the suite's existing ComfyUI System User storage, outside package files
|
||||
replaced during updates. Resolve the configured user directory consistently
|
||||
in the running suite and the separate installer process.
|
||||
Default location: ComfyUI/user/__tts_audio_suite/runtime_settings.json.
|
||||
Document the equivalent location under a custom user directory so pre-install
|
||||
opt-out does not depend on opening a UI that is not installed yet.
|
||||
- File schema:
|
||||
|
||||
```json
|
||||
{
|
||||
"install_shared_runtime": false
|
||||
}
|
||||
```
|
||||
|
||||
- Only accept the defined boolean. Do not
|
||||
accept executable paths, package names, URLs, or shell commands from this file,
|
||||
a workflow, or a management request. Protect settings-management writes.
|
||||
- Turning shared preparation off affects later installer runs. It does not
|
||||
remove the installed environment, stop its use by existing workflows, or
|
||||
select Main Environment instead. The file option supports pre-install opt-out;
|
||||
the UI becomes available after the suite is installed.
|
||||
- Do not add automatic deletion on restart. Show whether the environment is
|
||||
installed, its folder location, and instructions to close ComfyUI before
|
||||
manually deleting runtimes/shared_legacy_t4. Explain that engines still set
|
||||
to Shared Runtime then need that environment restored or another prepared,
|
||||
compatible runtime selected. Model weights and voice files are separate.
|
||||
|
||||
### Runtime choices implemented locally
|
||||
|
||||
- Remove Dedicated Runtime from Qwen3-TTS and Step Audio EditX. Runtime widgets
|
||||
stay in their existing positions with Main Environment and Shared Runtime.
|
||||
- Old Dedicated Runtime widget/API values and old dedicated profile names map
|
||||
to Shared Runtime. Frontend migration also updates saved workflow widgets.
|
||||
- Remove the separate Qwen, Step, and VibeVoice legacy profile definitions; keep
|
||||
the shared profile unchanged so existing installations can be reused.
|
||||
- Leave existing dedicated environment folders on disk. Do not delete user files
|
||||
as part of this change. Users can remove unused folders after closing ComfyUI.
|
||||
- No dedicated installation controls or file settings are needed. Reuse valid
|
||||
shared runtime metadata on later updates, checking profile changes separately
|
||||
from the main dependency installer's fast path.
|
||||
|
||||
Manager can defer its installation jobs until restart, including on Windows.
|
||||
That is still Manager-requested package installation. A workflow node recording
|
||||
flags and our prestartup/import code calling pip on every boot is a different
|
||||
boundary; timing alone does not make it acceptable. No new installer node is
|
||||
needed for the recommended default. The implementation still requires Registry review; it does not assert approval under the latest policy.
|
||||
|
||||
Reuse existing compatible environments. An update must not silently delete a
|
||||
working environment during generation. Model downloads remain separate from
|
||||
installing executable Python dependencies. An HTTP route or workflow node that
|
||||
invokes pip after a click would still be runtime installation; renaming pip or
|
||||
hiding it in a helper is not a policy fix.
|
||||
|
||||
Evidence checked:
|
||||
|
||||
- [Official standards](https://docs.comfy.org/registry/standards) prohibit runtime
|
||||
package installation through subprocess calls.
|
||||
- [Manager's install queue](https://github.com/Comfy-Org/ComfyUI-Manager/blob/main/glob/manager_server.py)
|
||||
accepts registered node IDs/versions and delegates to its package installer.
|
||||
Its [special-purpose files documentation](https://github.com/Comfy-Org/ComfyUI-Manager#custom-node-support)
|
||||
documents automatic install.py execution during installation.
|
||||
- [Manager's deferred installer](https://github.com/Comfy-Org/ComfyUI-Manager/blob/main/glob/manager_core.py#L2070)
|
||||
schedules installation jobs for restart when appropriate, rather than requiring
|
||||
each node pack to create an independent startup installer.
|
||||
- [SAM3 Registry versions](https://api.comfy.org/nodes/comfyui-sam3/versions?include_status_reason=true):
|
||||
0.1.21 is Active, with a manual SAFE decision by drltdata@comfy.org under
|
||||
policy-v0.1. Its [published archive](https://cdn.comfy.org/pznodes/comfyui-sam3/0.1.21/node.zip)
|
||||
contains install.py calling comfy_env.install(), plus isolated-environment
|
||||
dependency configuration. This is an accepted installation-time precedent,
|
||||
not a blanket exception for our current first-use bootstrap or newer policies.
|
||||
- [comfy-env installation documentation](https://docs.comfy-forge.org/comfy-env/install/)
|
||||
describes installation-time creation rather than environment creation during
|
||||
node execution. Adoption of that library is not required for this plan.
|
||||
- Registry review requests [226](https://github.com/Comfy-Org/registry-backend/issues/226),
|
||||
[261](https://github.com/Comfy-Org/registry-backend/issues/261), and
|
||||
[210](https://github.com/Comfy-Org/registry-backend/issues/210) are unanswered;
|
||||
they do not prove approval of optional runtime installers.
|
||||
|
||||
### User documentation
|
||||
|
||||
README.md now describes automatic installation, the UI/file opt-out, custom user
|
||||
directories, repair through Manager, and manual cleanup. The details command shows
|
||||
the actual preference file and runtime folder; no UI route or node runs pip.
|
||||
|
||||
## 5. Registry archive contents
|
||||
|
||||
- Added .comfyignore to exclude root tests/scripts, GitHub automation, the project
|
||||
index, and development reports from Registry archives. Repository contents
|
||||
remain available for development.
|
||||
- Keep the two metadata YAML files in docs/Dev reports. The auxiliary model
|
||||
registry reads its YAML at runtime. Keep user guides, example workflows,
|
||||
frontend assets, and all engine/training code.
|
||||
- The publisher's current Git filename handling skips quoted emoji paths.
|
||||
Configure core.quotepath=false in publishing so those guides/examples ship.
|
||||
|
||||
## Verification for the implementation
|
||||
|
||||
- Compare an archive with an independently collected, NUL-delimited Git source
|
||||
manifest (git ls-files -z), not the packer's own parsed filename list. Require
|
||||
every missing file to have an intentional exclusion and verify retained bytes.
|
||||
This catches silent omissions from quoting, decoding, and path parsing.
|
||||
- Exercise uploads, allowed typed paths, rejected traversal/link escapes, and
|
||||
malformed node IDs through both node execution and HTTP preview routes.
|
||||
- Check restricted loaders against the actual local formats, then exercise the
|
||||
affected engine inference and training paths using the configured test setup.
|
||||
- Check existing prepared environments and missing/stale profiles. Confirm that
|
||||
generation cannot install packages or replace an environment.
|
||||
- Recheck the published Registry status separately from successful upload;
|
||||
packaging cleanup by itself does not fix the runtime findings.
|
||||
|
||||
## Local checks for dedicated runtime removal
|
||||
|
||||
- The existing runtime unit test module passed all 27 cases, including legacy
|
||||
mode/profile migration and rejection of unsupported values.
|
||||
- JavaScript migration checks covered 20 hook/value combinations and confirmed
|
||||
other widget positions/values and unrelated nodes remain unchanged.
|
||||
- The restarted ComfyUI server exposes only Main Environment and Shared Runtime
|
||||
for Qwen3-TTS and Step Audio EditX, with the original input order. Its prompt
|
||||
validation accepts old dedicated values and rejects unknown modes. Requests
|
||||
deliberately omitted required downstream inputs, so no generation was queued.
|
||||
- ComfyUI serves the new frontend extension. Full-fix checks are recorded below.
|
||||
|
||||
- Source rewrites must match AST call identity. Compare preserved attribute-call
|
||||
identities against the original source, independently of edit selection; builtin
|
||||
eval removal must not rename model.eval() or other unrelated methods.
|
||||
|
||||
## Completed local verification
|
||||
|
||||
- 59 targeted existing unit cases passed, covering runtime/profile migration,
|
||||
installer repairs, audio processing, voice compatibility, and dependency guidance.
|
||||
- An empty temporary Shared Runtime was created by the real installer using the
|
||||
configured test Python. It installed Transformers 4.57.3, inherited host PyTorch,
|
||||
passed import/readiness/reuse checks, and left the host packages unchanged. The
|
||||
temporary environment was removed. Windows console logging uses ASCII output.
|
||||
- Installer hook checks covered enabled preparation and file-based opt-out.
|
||||
Missing/stale runtime checks cannot run subprocess installers during generation;
|
||||
simulated failed installation restores the previous runtime.
|
||||
- Live API checks passed for uploaded audio analysis, permitted paths, blocked
|
||||
traversal/external paths/node IDs, strict settings values, same-origin writes,
|
||||
preference persistence, and preservation of installed runtime files.
|
||||
- The real browser showed the default-on setting and the details menu with the
|
||||
correct file/folder paths and manual cleanup instructions. Backend file edits
|
||||
override stale browser preferences. Frontend checks also covered rapid toggles
|
||||
and restoring the checkbox after a failed save.
|
||||
- Live Step TTS followed by a giggle edit succeeded through the Shared Runtime,
|
||||
including migration from the old dedicated API value. The final run produced
|
||||
non-silent 24 kHz audio lasting 3.12 seconds.
|
||||
- Restricted Dots latent statistics, the 3,060,182-entry Russian dictionary, NumPy
|
||||
data/DAC formats, and Demucs architecture metadata passed. A small real Demucs
|
||||
model restored its state through audio-separator's existing load_model entry
|
||||
point. Malicious pickle/NumPy/Torch payloads and model-constructor REDUCE calls
|
||||
were rejected. Dependency patches preserve global torch/importlib references.
|
||||
- Python source compilation and independent AST comparison preserved every
|
||||
existing neural model.eval() call. Required config list arithmetic remains usable.
|
||||
- The candidate archive includes intended uncommitted source without staging it:
|
||||
1,936 packaged files and 104 intentional exclusions. Runtime files, Unicode
|
||||
paths, user guides, and both metadata YAML files retain exact bytes. The auxiliary
|
||||
model registry reads its YAML successfully from the packaged layout.
|
||||
|
||||
Limits: Manager is absent from the test installation, so its real queue/restart
|
||||
flow was not exercised; the install.py hook and fresh runtime installation were
|
||||
tested directly. This does not establish every engine's inference/training resume,
|
||||
support arbitrary whole-object checkpoints, or constitute Registry approval.
|
||||
@@ -109,7 +109,7 @@ Decision
|
||||
|
||||
Operational Recommendation
|
||||
- Treat Qwen3-TTS as a legacy-stack engine for now.
|
||||
- Route it into the shared legacy Transformers 4 runtime used by compatible engines (the same runtime family already proven for Qwen in prior testing).
|
||||
- Route it into the shared legacy Transformers 4 runtime used by compatible engines (same family already proven for VibeVoice/Qwen in prior testing).
|
||||
- Keep the main environment on Transformers 5.
|
||||
- Revisit native Transformers 5 support only if Qwen3-TTS becomes strategically important enough to justify deeper behavior-level compatibility work.
|
||||
|
||||
|
||||
@@ -12,6 +12,10 @@ runtime_isolation_modes:
|
||||
label: "Shared Runtime"
|
||||
short_label: "Shared"
|
||||
description: "Use a secondary shared runtime for engines with dependency conflicts. This runtime reuses heavy base packages like PyTorch from the main environment when configured."
|
||||
dedicated_runtime:
|
||||
label: "Dedicated Runtime"
|
||||
short_label: "Dedicated"
|
||||
description: "Use a dedicated secondary runtime only for one engine when the shared runtime is not enough."
|
||||
|
||||
engines:
|
||||
- id: f5-tts
|
||||
@@ -372,6 +376,17 @@ engines:
|
||||
- "Multilingual (KugelAudio variants)"
|
||||
- "4-bit quantization"
|
||||
|
||||
runtime_isolation:
|
||||
default_mode: "shared_runtime"
|
||||
supported_modes:
|
||||
- "main_environment"
|
||||
- "shared_runtime"
|
||||
- "dedicated_runtime"
|
||||
status: "implemented"
|
||||
runtime_profile: "vibevoice_transformers4_shared"
|
||||
dedicated_profile: "vibevoice_transformers4_dedicated"
|
||||
notes: "Recommended on modern ComfyUI stacks because VibeVoice/Kugel currently needs a secondary dependency layer. The shared runtime reuses heavy base packages like PyTorch from the main environment to avoid full duplication."
|
||||
|
||||
model_sources:
|
||||
- component: "vibevoice-1.5B"
|
||||
source_name: "microsoft/VibeVoice-1.5B"
|
||||
@@ -850,8 +865,10 @@ engines:
|
||||
supported_modes:
|
||||
- "main_environment"
|
||||
- "shared_runtime"
|
||||
- "dedicated_runtime"
|
||||
status: "implemented"
|
||||
runtime_profile: "vibevoice_transformers4_shared"
|
||||
dedicated_profile: "qwen3_tts_transformers4_dedicated"
|
||||
notes: "Recommended on modern ComfyUI stacks because Qwen3-TTS, Qwen3-ASR, and the reused Qwen forced aligner currently rely on the shared legacy T4 runtime for stable operation."
|
||||
|
||||
model_sources:
|
||||
@@ -1044,35 +1061,19 @@ engines:
|
||||
asr: false
|
||||
training: false
|
||||
|
||||
runtime_isolation:
|
||||
default_mode: "shared_runtime"
|
||||
supported_modes:
|
||||
- "main_environment"
|
||||
- "shared_runtime"
|
||||
status: "implemented"
|
||||
runtime_profile: "vibevoice_transformers4_shared"
|
||||
notes: "Shared Transformers 4 is recommended; the main Transformers 5 stack can produce invalid audio tokens."
|
||||
|
||||
special_features:
|
||||
- "Second Pass Speech Editing Node: 14 emotions"
|
||||
- "32 speaking styles"
|
||||
- "Dated checkpoints: 2026-01-23 / 2025-11-28"
|
||||
- "Expanded January paralinguistic effects with legacy tag compatibility"
|
||||
- "Selectable main or shared Python runtime (shared Transformers 4 runtime recommended)"
|
||||
- "Paralinguistic effects"
|
||||
- "Selectable main, shared, or dedicated Python runtime (shared Transformers 4 runtime recommended)"
|
||||
|
||||
model_sources:
|
||||
- component: "Step-Audio-EditX-2026-01-23"
|
||||
- component: "Step-Audio-EditX"
|
||||
source_name: "stepfun-ai/Step-Audio-EditX"
|
||||
source_url: "https://huggingface.co/stepfun-ai/Step-Audio-EditX"
|
||||
size: "~7GB"
|
||||
auto_download: true
|
||||
notes: "Default 3B checkpoint; pinned to 5fe2f8a05c2353301ad47d3c1747b262115da138"
|
||||
- component: "Step-Audio-EditX-2025-11-28"
|
||||
source_name: "stepfun-ai/Step-Audio-EditX"
|
||||
source_url: "https://huggingface.co/stepfun-ai/Step-Audio-EditX/tree/7f3de603ae46c96dff6f06f47b1d5a45aabd34fe"
|
||||
size: "~7GB"
|
||||
auto_download: true
|
||||
notes: "Legacy checkpoint; pinned November weights in a separate folder"
|
||||
notes: "Main 3B audio editing model"
|
||||
- component: "Step-Audio-Tokenizer"
|
||||
source_name: "stepfun-ai/Step-Audio-Tokenizer"
|
||||
source_url: "https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer"
|
||||
@@ -1216,6 +1217,7 @@ engines:
|
||||
default_mode: "main_environment"
|
||||
main_environment: true
|
||||
shared_runtime: false
|
||||
dedicated_runtime: false
|
||||
|
||||
special_features:
|
||||
- "Free-form sub-word emotion/prosody tags"
|
||||
@@ -1966,7 +1968,7 @@ readme_model_download_table:
|
||||
- engine: "Step Audio EditX"
|
||||
primary_model_path: "ComfyUI/models/TTS/step_audio_editx/"
|
||||
auto_download: "✅"
|
||||
notes: "Lazy dated checkpoints + tokenizer stack; unversioned installs retained"
|
||||
notes: "Main model + tokenizer stack"
|
||||
- engine: "CosyVoice3"
|
||||
primary_model_path: "ComfyUI/models/TTS/CosyVoice/"
|
||||
auto_download: "✅"
|
||||
@@ -2205,20 +2207,14 @@ model_layouts_markdown: |
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/step_audio_editx/
|
||||
├── Step-Audio-EditX-2026-01-23/
|
||||
│ └── CosyVoice-300M-25Hz/
|
||||
├── Step-Audio-EditX-2025-11-28/ # downloaded only when selected
|
||||
│ └── CosyVoice-300M-25Hz/
|
||||
├── Step-Audio-EditX/ # existing unversioned installations
|
||||
├── Step-Audio-EditX/
|
||||
│ └── CosyVoice-300M-25Hz/
|
||||
└── FunASR-Paraformer/
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Selected checkpoints and tokenizer assets auto-download at pinned revisions.
|
||||
- Existing unversioned/local weights are retained and folders are not relabeled; known config compatibility fields may be normalized.
|
||||
- See [Step Audio EditX checkpoints](STEP_AUDIO_EDITX_CHECKPOINTS.md) for selection and expanded tags.
|
||||
- Main model, tokenizer assets, and speech stack auto-download.
|
||||
|
||||
## Higgs Audio v3
|
||||
|
||||
|
||||
@@ -7,14 +7,14 @@
|
||||
| **F5-TTS** | Main | Base, v1, E2TTS + 8 lang models | ~1.2GB each | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | CC-BY-NC-4.0 | Targeted Word/Speech Editing, Speed control | 10 |
|
||||
| **ChatterBox** | Main | EN, DE×3, IT, FR, RU, HY, KA, JA, KO, NO | ~4.3GB | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | MIT | Expressiveness slider | 10 |
|
||||
| **ChatterBox 23L** | Main | v1, v2, v3, Vietnamese (Viterbox), Egyptian Arabic (oddadmix) | ~4.3GB | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | MIT | V1, V2, and V3 official checkpoints, Emotion tokens (v2; currently ineffective), V3 skips the legacy alignment analyzer and trims the final token artifact | 25 |
|
||||
| **VibeVoice** | Main | 1.5B, 7B, KugelAudio-0 (7B), kugel-2 (7B), Hindi-1.5B/7B | 5.4GB / 18GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | MIT (research-only per model card) | 90-min long-form, Native 4-speaker (Base models), Multilingual (KugelAudio variants), 4-bit quantization | 27 |
|
||||
| **VibeVoice** | Shared | 1.5B, 7B, KugelAudio-0 (7B), kugel-2 (7B), Hindi-1.5B/7B | 5.4GB / 18GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | MIT (research-only per model card) | 90-min long-form, Native 4-speaker (Base models), Multilingual (KugelAudio variants), 4-bit quantization | 27 |
|
||||
| **Higgs Audio 2** | Shared | 3B | ~9GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Boson Higgs Audio 2 Community License | 3 multi-speaker, CUDA graphs (55+ tokens/sec) | 5 |
|
||||
| **Higgs Audio v3** | Main | 4B | ~8GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Boson Higgs Audio v3 Research and Non-Commercial License | Native inline emotion/style/prosody/SFX tags, Zero-shot voice cloning, 100+ language support | 100+ |
|
||||
| **IndexTTS 2 / 2.5** | Main | IndexTTS-2, IndexTTS-2.5 | ~4.7GB / ~5.49GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | bilibili Model Use License | Emotion Control: 8 vectors, Text as reference, Audio as reference, IndexTTS-2.5 official internal feature-duration scaling (not prosody planning), IndexTTS-2.5 pronunciation annotations | 5 |
|
||||
| **CosyVoice3** | Main | 0.5B, 0.5B-RL | ~5.4GB | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | Apache-2.0 | Paralinguistic tags | 4 |
|
||||
| **Qwen3-TTS** | Shared | 0.6B, 1.7B (CustomVoice/VoiceDesign/Base) | ~3-6GB | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | Apache-2.0 | Voice design, ASR (Automatic Speech Recognition) | 10 |
|
||||
| **Granite ASR** | Main | granite-4.0-1b-speech, granite-speech-4.1-2b, granite-speech-4.1-2b-plus | ~4.6GB | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | Apache-2.0 | Native speaker attribution / diarization (plus model variant), Native word-level timestamps (plus model variant), ASR (Automatic Speech Recognition), Custom timestamps/SRT via reused Qwen forced aligner, Speech translation (experimental), Optional forced aligner auto-routed through shared legacy T4 runtime | 6 |
|
||||
| **Step Audio EditX** | Shared | 3B LLM + CosyVoice | ~7GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 (verify before commercial use) | Second Pass Speech Editing Node: 14 emotions, 32 speaking styles, Dated checkpoints: 2026-01-23 / 2025-11-28, Expanded January paralinguistic effects with legacy tag compatibility, Selectable main or shared Python runtime (shared Transformers 4 runtime recommended) | 4 |
|
||||
| **Step Audio EditX** | Main | 3B LLM + CosyVoice | ~7GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 (verify before commercial use) | Second Pass Speech Editing Node: 14 emotions, 32 speaking styles, Paralinguistic effects, Selectable main, shared, or dedicated Python runtime (shared Transformers 4 runtime recommended) | 4 |
|
||||
| **Echo-TTS** | Main | echo-tts-base + fish-s1-dac-min | ~5.3GB + ~1.8GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | CC-BY-NC-SA-4.0 | Diffusion-based (~30s best), Force Speaker KV (speaker drift control) | 1 |
|
||||
| **Fish Audio S2 Pro** | Main | S2 Pro 4B / FP8 | ~10.3GB / ~8.0GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Fish Audio Research License | Free-form sub-word emotion/prosody tags, Native multi-speaker and multi-turn dialogue with dynamic speaker references, Zero-shot voice cloning, Optional per-segment custom character switching, Configurable 4K-32K native context with reduced KV-cache VRAM, Optional community FP8 weight-only checkpoint with BF16 activations, Optional on-the-fly BitsAndBytes INT8/NF4 for the official checkpoint | 80+ languages |
|
||||
| **Dots TTS** | Main | dots.tts-base, dots.tts-soar, dots.tts-mf | ~6GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 | Official auto language detect / language control, SOAR and MeanFlow distilled variants | 19 |
|
||||
@@ -24,4 +24,4 @@
|
||||
| **MOSS-SoundEffect v2** | Main | MOSS-SoundEffect-v2.0 | ~11.2GB | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | Apache-2.0 | Durations up to 30 seconds, Native negative prompting, CFG, flow shift, and diffusion-step controls, Prompt-only text-to-sound generation, 48 kHz mono output, Seeded generation | 2 |
|
||||
| **RVC** | Main | Community .pth | 100-300MB | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | MIT (framework); community models vary | Real-time VC, Integrated training workflow, Pitch shift (±14), 6 HuBERT models, Language-independent | Any |
|
||||
|
||||
*Isolation column: `Main` runs in the main ComfyUI environment. `Shared` uses a shared secondary runtime reused by multiple engines.*
|
||||
*Isolation column: `Main` runs in the main ComfyUI environment. `Shared` uses a shared secondary runtime reused by multiple engines. `Dedicated` uses an engine-specific secondary runtime.*
|
||||
@@ -11,8 +11,6 @@ The multiline TTS tag editor now has an engine-aware `Inline Tags` panel with se
|
||||
|
||||
Use the Step mode when you want post-process editing tags. Use the Higgs or Cosy modes when you want those engines' native generation tags.
|
||||
|
||||
The existing **model_path** dropdown now offers `Step-Audio-EditX-2026-01-23` and `Step-Audio-EditX-2025-11-28`. The January checkpoint is the default for new engine nodes and supports the expanded sound vocabulary below. Existing unversioned and local selections keep their files. See [checkpoint selection and compatibility](STEP_AUDIO_EDITX_CHECKPOINTS.md).
|
||||
|
||||
## 💡 Inline vs Manual Node Workflow
|
||||
|
||||
**ComfyUI is modular** - you can manually chain **TTS → 🎨 Step Audio EditX - Audio Editor** nodes for full control.
|
||||
@@ -61,19 +59,6 @@ Insert non-verbal sounds at specific positions in the audio:
|
||||
| `<Question-ei>` | Question sound | `<Question-ei> Really?` |
|
||||
| `<Dissatisfaction-hnn>` | Dissatisfied "hnn" | `<Dissatisfaction-hnn> Not good` |
|
||||
|
||||
The **2026-01-23** checkpoint additionally supports:
|
||||
|
||||
| Tag | Effect |
|
||||
|---|---|
|
||||
| `<inhale>`, `<exhale>`, `<breath>` | Breathing sounds |
|
||||
| `<laugh>`, `<chuckle>`, `<giggle>` | Laughter variants |
|
||||
| `<clears throat>` or `<clears_throat>` | Throat clearing |
|
||||
| `<snort>`, `<cough>` | Snorting and coughing |
|
||||
| `<Surprise-yo>` | Additional surprise vocalization |
|
||||
| `<Question-ah>`, `<Question-en>`, `<Question-yi>`, `<Question-oh>` | Additional questioning vocalizations |
|
||||
|
||||
These support the same iteration and pipe syntax: `<giggle:2>` and `<clears_throat:1|style:serious>`. Tags are case-insensitive. For ChatterBox v2/v3, specify an iteration for Step effects (`<giggle:1>`); bare overlapping tags retain their native meaning.
|
||||
|
||||
**Position matters:** The sound is inserted where you place the tag.
|
||||
|
||||
**💡 Pro Tip - Stronger Effects:** For more reliable and pronounced effects, include relevant text alongside the tag:
|
||||
@@ -93,7 +78,7 @@ Apply emotional tone to the entire audio segment:
|
||||
|
||||
`<emotion:VALUE>` or `<emotion:VALUE:ITERATIONS>`
|
||||
|
||||
**Available emotions:** happy, sad, angry, excited, fear, humour, surprised, disgusted, confusion, empathy, embarrass, depressed, coldness, admiration. Older suite options `calm` and `fearful` remain accepted.
|
||||
**Available emotions:** happy, sad, angry, excited, calm, fearful, surprised, disgusted, confusion, empathy, embarrass, depressed, coldness, admiration
|
||||
|
||||
**Examples:**
|
||||
```
|
||||
@@ -638,13 +623,9 @@ Choose appropriate effects for the situation:
|
||||
### All Paralinguistic Tags
|
||||
`<Breathing>`, `<Laughter>`, `<Sigh>`, `<Uhm>`, `<Surprise-oh>`, `<Surprise-ah>`, `<Surprise-wa>`, `<Confirmation-en>`, `<Question-ei>`, `<Dissatisfaction-hnn>`
|
||||
|
||||
January checkpoint vocabulary: `<inhale>`, `<exhale>`, `<laugh>`, `<chuckle>`, `<clears throat>` (alias `<clears_throat>`), `<snort>`, `<giggle>`, `<cough>`, `<breath>`, `<Surprise-yo>`, `<Question-ah>`, `<Question-en>`, `<Question-yi>`, `<Question-oh>`.
|
||||
|
||||
### All Emotions
|
||||
`<emotion:happy>`, `<emotion:sad>`, `<emotion:angry>`, `<emotion:excited>`, `<emotion:calm>`, `<emotion:fearful>`, `<emotion:surprised>`, `<emotion:disgusted>`, `<emotion:confusion>`, `<emotion:empathy>`, `<emotion:embarrass>`, `<emotion:depressed>`, `<emotion:coldness>`, `<emotion:admiration>`
|
||||
|
||||
Also available: `<emotion:fear>` and `<emotion:humour>`.
|
||||
|
||||
### All Styles
|
||||
`<style:whisper>`, `<style:serious>`, `<style:child>`, `<style:older>`, `<style:girl>`, `<style:pure>`, `<style:sister>`, `<style:sweet>`, `<style:exaggerated>`, `<style:ethereal>`, `<style:generous>`, `<style:recite>`, `<style:act_coy>`, `<style:warm>`, `<style:shy>`, `<style:comfort>`, `<style:authority>`, `<style:chat>`, `<style:radio>`, `<style:soulful>`, `<style:gentle>`, `<style:story>`, `<style:vivid>`, `<style:program>`, `<style:news>`, `<style:advertising>`, `<style:roar>`, `<style:murmur>`, `<style:shout>`, `<style:deeply>`, `<style:loudly>`, `<style:arrogant>`, `<style:friendly>`
|
||||
|
||||
@@ -668,7 +649,7 @@ A: Slightly. The TTS generates normally, then Step Audio EditX loads once and pr
|
||||
A: Yes! Batch processing is automatically applied - all subtitles are generated first, then all edits are applied at once.
|
||||
|
||||
**Q: What if I make a typo in a tag?**
|
||||
A: The Step parser leaves unknown tags in the text and does not create an edit for them. Correct the spelling before generation; the TTS engine may interpret or speak the remaining text.
|
||||
A: Unknown tags are ignored and stripped from the text. Check console for warnings.
|
||||
|
||||
**Q: Can I nest tags?**
|
||||
A: No. Use pipe syntax or separate tags: `<tag1|tag2>` or `<tag1><tag2>`
|
||||
|
||||
@@ -105,8 +105,7 @@ Use this as the canonical list of model repositories/links for offline setup.
|
||||
|
||||
| Component | Source | Size | Auto-Download | Notes |
|
||||
|---|---|---|---|---|
|
||||
| Step-Audio-EditX-2026-01-23 | [stepfun-ai/Step-Audio-EditX](https://huggingface.co/stepfun-ai/Step-Audio-EditX) | ~7GB | ✅ | Default 3B checkpoint; pinned to 5fe2f8a05c2353301ad47d3c1747b262115da138 |
|
||||
| Step-Audio-EditX-2025-11-28 | [stepfun-ai/Step-Audio-EditX](https://huggingface.co/stepfun-ai/Step-Audio-EditX/tree/7f3de603ae46c96dff6f06f47b1d5a45aabd34fe) | ~7GB | ✅ | Legacy checkpoint; pinned November weights in a separate folder |
|
||||
| Step-Audio-EditX | [stepfun-ai/Step-Audio-EditX](https://huggingface.co/stepfun-ai/Step-Audio-EditX) | ~7GB | ✅ | Main 3B audio editing model |
|
||||
| Step-Audio-Tokenizer | [stepfun-ai/Step-Audio-Tokenizer](https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer) | Included | ✅ | Tokenizer bundle used by Step EditX |
|
||||
|
||||
## Echo-TTS
|
||||
|
||||
@@ -194,20 +194,14 @@ Notes:
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/step_audio_editx/
|
||||
├── Step-Audio-EditX-2026-01-23/
|
||||
│ └── CosyVoice-300M-25Hz/
|
||||
├── Step-Audio-EditX-2025-11-28/ # downloaded only when selected
|
||||
│ └── CosyVoice-300M-25Hz/
|
||||
├── Step-Audio-EditX/ # existing unversioned installations
|
||||
├── Step-Audio-EditX/
|
||||
│ └── CosyVoice-300M-25Hz/
|
||||
└── FunASR-Paraformer/
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Selected checkpoints and tokenizer assets auto-download at pinned revisions.
|
||||
- Existing unversioned/local weights are retained and folders are not relabeled; known config compatibility fields may be normalized.
|
||||
- See [Step Audio EditX checkpoints](STEP_AUDIO_EDITX_CHECKPOINTS.md) for selection and expanded tags.
|
||||
- Main model, tokenizer assets, and speech stack auto-download.
|
||||
|
||||
## Higgs Audio v3
|
||||
|
||||
|
||||
@@ -63,9 +63,6 @@ Step Audio EditX examples:
|
||||
|
||||
```text
|
||||
<Laughter>
|
||||
<giggle>
|
||||
<clears throat:2>
|
||||
<Question-oh>
|
||||
<emotion:happy>
|
||||
<style:whisper:2>
|
||||
<restore:1@2>
|
||||
@@ -96,12 +93,6 @@ OmniVoice examples:
|
||||
|
||||
Use the dedicated inline tag controls in the sidebar when you do not want to type these by hand.
|
||||
|
||||
The Step sound menu includes the original sounds plus the expanded sounds for the `2026-01-23` checkpoint: `inhale`, `exhale`, `laugh`, `chuckle`, `clears throat`, `snort`, `giggle`, `cough`, `breath`, `Surprise-yo`, `Question-ah`, `Question-en`, `Question-yi`, and `Question-oh`. Select `Step-Audio-EditX-2026-01-23` in the engine node to use these sounds. The `2025-11-28` checkpoint supports the original vocabulary; see the [checkpoint guide](STEP_AUDIO_EDITX_CHECKPOINTS.md).
|
||||
|
||||
Validation, highlighting, and the tag-swap menu recognize these sounds. Names are case-insensitive, and `<clears_throat>` is accepted as an alias for `<clears throat>`. The insertion controls preserve the existing iteration syntax, such as `<giggle:2>`, and pipe combinations, such as `<giggle:2|emotion:happy>`. The emotion menu also includes `fear` and `humour`.
|
||||
|
||||
The editor checks syntax for the chosen tag engine but does not inspect the engine node's selected model checkpoint. A valid new sound still requires the January weights.
|
||||
|
||||
Important differences:
|
||||
|
||||
- `Step Audio EditX` tags are post-process controls
|
||||
|
||||
@@ -93,7 +93,7 @@ Manual tests:
|
||||
|
||||
Runtime plan:
|
||||
- Main Environment only
|
||||
- or Shared Runtime if dependency conflicts are expected
|
||||
- or Shared Runtime / Dedicated Runtime if dependency conflicts are expected
|
||||
```
|
||||
|
||||
If the scope is vague, do not code yet.
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
# Step Audio EditX checkpoints
|
||||
|
||||
Choose the checkpoint in the existing **model_path** dropdown of **⚙️ Step Audio EditX Engine**. No widget was added or reordered, so saved device, precision, quantization, generation, and runtime values keep their positions.
|
||||
|
||||
| Model choice | Checkpoint | Download revision |
|
||||
|---|---|---|
|
||||
| `Step-Audio-EditX-2026-01-23` | Updated weights and expanded sound tags; default for new engine nodes | `5fe2f8a05c2353301ad47d3c1747b262115da138` |
|
||||
| `Step-Audio-EditX-2025-11-28` | Legacy weights | `7f3de603ae46c96dff6f06f47b1d5a45aabd34fe` |
|
||||
| `Step-Audio-EditX` | Compatibility choice for an existing unversioned installation | Existing files are retained; a missing installation downloads the January checkpoint |
|
||||
| `local:...` | Explicitly selected local model | Existing files are retained |
|
||||
|
||||
Dates identify the weight uploads. The January release was announced on January 29; its pinned revision also includes subsequent configuration fixes. These are versions of the same 3B model architecture, not official “v1/v2” names.
|
||||
|
||||
Sources: [official model](https://huggingface.co/stepfun-ai/Step-Audio-EditX), [November snapshot](https://huggingface.co/stepfun-ai/Step-Audio-EditX/tree/7f3de603ae46c96dff6f06f47b1d5a45aabd34fe), [upstream release notes](https://github.com/stepfun-ai/Step-Audio-EditX#readme).
|
||||
|
||||
## Existing workflows and storage
|
||||
|
||||
- Existing `Step-Audio-EditX` and `local:...` selections remain valid. The suite does not replace their existing weights, rename their folders, or infer their checkpoint. The existing compatibility normalization of `config.json` still applies.
|
||||
- Dated models download lazily into separate folders under `ComfyUI/models/TTS/step_audio_editx/`, respecting `extra_model_paths.yaml`.
|
||||
- Selecting both checkpoints stores both sets of model files. The second LLM checkpoint requires approximately another 7 GB.
|
||||
- Model and edit-result caches distinguish the selected checkpoint.
|
||||
- The shared Transformers 4 runtime remains recommended. Selecting the updated standard checkpoint does not switch to vLLM or to the separate upstream AWQ checkpoint. The suite's `int4` option remains bitsandbytes NF4.
|
||||
|
||||
## Sound tags
|
||||
|
||||
Use the **2026-01-23** checkpoint for expanded sound tags. The November checkpoint is available to preserve older behavior; selecting it does not add the new learned sounds.
|
||||
|
||||
Existing spellings remain supported:
|
||||
|
||||
`<Breathing>`, `<Laughter>`, `<Sigh>`, `<Uhm>`, `<Surprise-oh>`, `<Surprise-ah>`, `<Surprise-wa>`, `<Confirmation-en>`, `<Question-ei>`, `<Dissatisfaction-hnn>`.
|
||||
|
||||
The January vocabulary also exposes:
|
||||
|
||||
`<inhale>`, `<exhale>`, `<laugh>`, `<chuckle>`, `<clears throat>`, `<snort>`, `<giggle>`, `<cough>`, `<breath>`, `<Surprise-yo>`, `<Question-ah>`, `<Question-en>`, `<Question-yi>`, `<Question-oh>`.
|
||||
|
||||
Input is case-insensitive. `<clears_throat>` is an alias for `<clears throat>`; both become `[clears throat]` in the model instruction. Expanded tags support the existing iteration and pipe syntax, for example:
|
||||
|
||||
```text
|
||||
[Alice] That was funny <giggle:2>.
|
||||
[Bob] Let me explain <clears_throat:1|style:serious>.
|
||||
```
|
||||
|
||||
For ChatterBox v2/v3, use an explicit iteration to request Step editing: `<giggle:1>`. Bare `<giggle>`, `<inhale>`, `<exhale>`, and `<cough>` retain their ChatterBox-native meaning. CosyVoice's native sound tags remain native as well.
|
||||
|
||||
## Audio Editor and other engines
|
||||
|
||||
Connect the dated **Step Audio EditX Engine** output to **🎨 Step Audio EditX - Audio Editor** to choose a checkpoint explicitly. In the Audio Editor's transcript, use bare sound tags such as `<giggle>` or `<clears throat>`; its **n_edit_iterations** input controls the number of passes.
|
||||
|
||||
Without a connected engine, the Audio Editor prefers an installed January checkpoint, then an existing unversioned installation. If neither exists, it downloads the dated January checkpoint. This same fallback applies to automatic Step editing of other engines' audio. An existing unversioned installation may contain older weights; use the manual Audio Editor with a dated engine for explicit control.
|
||||
|
||||
See [the inline tag guide](INLINE_EDIT_TAGS_USER_GUIDE.md) for position, iteration, restoration, language, and duration behavior.
|
||||
@@ -56,7 +56,7 @@ python -c "import triton; print(triton.__version__)"
|
||||
|
||||
**Main environment**: no special configuration beyond a working compiler toolchain.
|
||||
|
||||
**Shared Runtime isolation**:
|
||||
**Shared / Dedicated Runtime isolation**:
|
||||
- TTS Audio Suite now tries to detect Visual Studio automatically and inject a proper MSVC build environment into the isolated worker
|
||||
- This is meant to preserve `torch.compile` parity with old direct Transformers 4 environments
|
||||
- It still depends on the Build Tools actually being installed
|
||||
@@ -112,7 +112,7 @@ pip install --upgrade torchvision --index-url https://download.pytorch.org/whl/c
|
||||
**Solution**:
|
||||
1. Install or repair Visual Studio Build Tools with the C++ workload
|
||||
2. Restart ComfyUI so the new environment is visible
|
||||
3. If using Shared Runtime, let TTS Audio Suite auto-detect the toolchain
|
||||
3. If using Shared/Dedicated Runtime, let TTS Audio Suite auto-detect the toolchain
|
||||
|
||||
If detection still fails, your machine state is broken, not just the Python package stack.
|
||||
|
||||
|
||||
@@ -0,0 +1,486 @@
|
||||
"""audio.cpp adapter for the Suite's unified ASR pipeline."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from typing import Any, Dict, Iterable, Mapping, Optional
|
||||
|
||||
import torch
|
||||
|
||||
from utils.asr.types import ASRRequest, ASRResult, ASRSegment, ASRWord
|
||||
from utils.audio.processing import AudioProcessingUtils
|
||||
|
||||
|
||||
_NATIVE_CHUNK_FAMILIES = {
|
||||
"fun_asr_nano",
|
||||
"higgs_audio_stt",
|
||||
"hviske_asr",
|
||||
"qwen3_asr",
|
||||
"vibevoice_asr",
|
||||
"voxtral_realtime",
|
||||
}
|
||||
|
||||
# audio.cpp release-0.5.1 keeps stale offline decoder state for these loaders:
|
||||
# the first request transcribes normally and later requests return empty text.
|
||||
# A fresh owned process is currently the only reliable reset contract.
|
||||
_RESTART_BETWEEN_CHUNKS_FAMILIES = {"nemotron_asr", "voxtral_realtime"}
|
||||
|
||||
|
||||
def _session(config: Mapping[str, Any]):
|
||||
from utils.audio_cpp.session import get_audio_cpp_session
|
||||
|
||||
return get_audio_cpp_session(dict(config))
|
||||
|
||||
|
||||
def _advanced_options(config: Mapping[str, Any]) -> Dict[str, Any]:
|
||||
value = config.get("advanced_options", config.get("request_options", {}))
|
||||
if value in (None, ""):
|
||||
return {}
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
value = json.loads(value)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"Invalid audio.cpp advanced JSON: {exc.msg}") from exc
|
||||
if not isinstance(value, Mapping):
|
||||
raise ValueError("audio.cpp advanced options must be a JSON object")
|
||||
return dict(value)
|
||||
|
||||
|
||||
def _audio_path(audio: Mapping[str, Any]) -> str:
|
||||
waveform = audio.get("waveform")
|
||||
sample_rate = audio.get("sample_rate")
|
||||
if not torch.is_tensor(waveform):
|
||||
raise TypeError("audio.cpp ASR input must contain a waveform tensor")
|
||||
if sample_rate is None or int(sample_rate) <= 0:
|
||||
raise ValueError("audio.cpp ASR input must contain a positive sample_rate")
|
||||
return os.path.abspath(
|
||||
AudioProcessingUtils.save_audio_to_temp_file(waveform, int(sample_rate))
|
||||
)
|
||||
|
||||
|
||||
def _waveform_3d(audio: Mapping[str, Any]) -> tuple[torch.Tensor, int]:
|
||||
waveform = audio.get("waveform")
|
||||
sample_rate = int(audio.get("sample_rate") or 0)
|
||||
if not torch.is_tensor(waveform):
|
||||
raise TypeError("audio.cpp ASR input must contain a waveform tensor")
|
||||
if sample_rate <= 0:
|
||||
raise ValueError("audio.cpp ASR input must contain a positive sample_rate")
|
||||
if waveform.ndim == 1:
|
||||
waveform = waveform.unsqueeze(0).unsqueeze(0)
|
||||
elif waveform.ndim == 2:
|
||||
waveform = waveform.unsqueeze(0)
|
||||
elif waveform.ndim != 3:
|
||||
raise ValueError(
|
||||
"audio.cpp ASR waveform must have [samples], [channels, samples], or "
|
||||
"[batch, channels, samples] shape"
|
||||
)
|
||||
if waveform.shape[0] != 1:
|
||||
raise ValueError("audio.cpp ASR accepts one audio item at a time")
|
||||
if waveform.shape[-1] <= 0:
|
||||
raise ValueError("audio.cpp ASR input audio is empty")
|
||||
return waveform.detach().cpu(), sample_rate
|
||||
|
||||
|
||||
def _chunk_ranges(
|
||||
total_samples: int,
|
||||
sample_rate: int,
|
||||
chunk_size: int,
|
||||
overlap: int,
|
||||
) -> list[tuple[int, int]]:
|
||||
if chunk_size <= 0:
|
||||
return [(0, total_samples)]
|
||||
if overlap < 0:
|
||||
raise ValueError("ASR overlap must be zero or greater")
|
||||
if overlap >= chunk_size:
|
||||
raise ValueError("ASR overlap must be smaller than chunk_size")
|
||||
|
||||
chunk_samples = chunk_size * sample_rate
|
||||
if total_samples <= chunk_samples:
|
||||
return [(0, total_samples)]
|
||||
step_samples = (chunk_size - overlap) * sample_rate
|
||||
ranges = []
|
||||
start = 0
|
||||
while start < total_samples:
|
||||
end = min(start + chunk_samples, total_samples)
|
||||
ranges.append((start, end))
|
||||
if end >= total_samples:
|
||||
break
|
||||
start += step_samples
|
||||
return ranges
|
||||
|
||||
|
||||
def _normalized_token(value: str) -> str:
|
||||
return re.sub(r"[^\w]+", "", value, flags=re.UNICODE).casefold()
|
||||
|
||||
|
||||
def _merge_transcript(parts: Iterable[str]) -> str:
|
||||
merged: list[str] = []
|
||||
for part in parts:
|
||||
incoming = str(part or "").strip().split()
|
||||
if not incoming:
|
||||
continue
|
||||
if not merged:
|
||||
merged.extend(incoming)
|
||||
continue
|
||||
limit = min(len(merged), len(incoming), 80)
|
||||
duplicate_count = 0
|
||||
for size in range(limit, 0, -1):
|
||||
left = [_normalized_token(token) for token in merged[-size:]]
|
||||
right = [_normalized_token(token) for token in incoming[:size]]
|
||||
if all(left) and left == right:
|
||||
duplicate_count = size
|
||||
break
|
||||
merged.extend(incoming[duplicate_count:])
|
||||
return " ".join(merged).strip()
|
||||
|
||||
|
||||
def _offset_words(
|
||||
words: Iterable[ASRWord], offset: float, unique_after: Optional[float]
|
||||
) -> list[ASRWord]:
|
||||
shifted = []
|
||||
for word in words:
|
||||
item = ASRWord(start=word.start + offset, end=word.end + offset, text=word.text)
|
||||
if unique_after is not None and (item.start + item.end) / 2.0 < unique_after:
|
||||
continue
|
||||
shifted.append(item)
|
||||
return shifted
|
||||
|
||||
|
||||
def _offset_segments(
|
||||
segments: Iterable[ASRSegment], offset: float, unique_after: Optional[float]
|
||||
) -> list[ASRSegment]:
|
||||
shifted = []
|
||||
for segment in segments:
|
||||
item = ASRSegment(
|
||||
start=segment.start + offset,
|
||||
end=segment.end + offset,
|
||||
text=segment.text,
|
||||
speaker=segment.speaker,
|
||||
)
|
||||
if unique_after is not None and (item.start + item.end) / 2.0 < unique_after:
|
||||
continue
|
||||
shifted.append(item)
|
||||
return shifted
|
||||
|
||||
|
||||
def _seconds(value: Any, sample_rate: int) -> float:
|
||||
try:
|
||||
return max(0.0, float(value) / float(sample_rate))
|
||||
except (TypeError, ValueError, ZeroDivisionError):
|
||||
return 0.0
|
||||
|
||||
|
||||
def _words(payload: Mapping[str, Any], sample_rate: int) -> list[ASRWord]:
|
||||
words = []
|
||||
for item in payload.get("words") or []:
|
||||
if not isinstance(item, Mapping):
|
||||
continue
|
||||
text = str(item.get("word", item.get("text", ""))).strip()
|
||||
if not text:
|
||||
continue
|
||||
words.append(
|
||||
ASRWord(
|
||||
start=_seconds(item.get("start_sample"), sample_rate),
|
||||
end=_seconds(item.get("end_sample"), sample_rate),
|
||||
text=text,
|
||||
)
|
||||
)
|
||||
return words
|
||||
|
||||
|
||||
def _plain_segments(payload: Mapping[str, Any], sample_rate: int) -> list[ASRSegment]:
|
||||
segments = []
|
||||
for item in payload.get("segments") or []:
|
||||
if not isinstance(item, Mapping):
|
||||
continue
|
||||
text = str(item.get("text", "")).strip()
|
||||
segments.append(
|
||||
ASRSegment(
|
||||
start=_seconds(item.get("start_sample"), sample_rate),
|
||||
end=_seconds(item.get("end_sample"), sample_rate),
|
||||
text=text,
|
||||
)
|
||||
)
|
||||
return segments
|
||||
|
||||
|
||||
def _speaker_segments(payload: Mapping[str, Any], sample_rate: int) -> list[ASRSegment]:
|
||||
segments = []
|
||||
for item in payload.get("speaker_turns") or []:
|
||||
if not isinstance(item, Mapping):
|
||||
continue
|
||||
speaker = str(item.get("speaker_id", "")).strip()
|
||||
if speaker and not speaker.lower().startswith("speaker"):
|
||||
speaker = f"Speaker {speaker}"
|
||||
segments.append(
|
||||
ASRSegment(
|
||||
start=_seconds(item.get("start_sample"), sample_rate),
|
||||
end=_seconds(item.get("end_sample"), sample_rate),
|
||||
text=str(item.get("text", "")).strip(),
|
||||
speaker=speaker or None,
|
||||
)
|
||||
)
|
||||
return segments
|
||||
|
||||
|
||||
def _attach_words(segments: Iterable[ASRSegment], words: Iterable[ASRWord]) -> None:
|
||||
segment_list = list(segments)
|
||||
for word in words:
|
||||
midpoint = (word.start + word.end) / 2.0
|
||||
target = next(
|
||||
(segment for segment in segment_list if segment.start <= midpoint <= segment.end),
|
||||
None,
|
||||
)
|
||||
if target is not None:
|
||||
target.words.append(word)
|
||||
|
||||
|
||||
class AudioCppASREngineAdapter:
|
||||
"""Normalize audio.cpp transcript/timing output into ``ASRResult``."""
|
||||
|
||||
def __init__(self, engine_data: Dict[str, Any]):
|
||||
self.engine_data = dict(engine_data)
|
||||
self.config = dict(engine_data.get("config", engine_data))
|
||||
|
||||
def _session_config(self) -> Dict[str, Any]:
|
||||
config = dict(self.config)
|
||||
if str(config.get("connection_mode", "auto")).lower() != "external_server":
|
||||
config["requested_task"] = "asr"
|
||||
config["task"] = "asr"
|
||||
return config
|
||||
|
||||
def transcribe(self, req: ASRRequest) -> ASRResult:
|
||||
if req.task != "transcribe":
|
||||
raise ValueError(
|
||||
"audio.cpp release-0.5.1 ASR loaders support transcription, not the "
|
||||
"Unified ASR translate mode"
|
||||
)
|
||||
|
||||
config = self._session_config()
|
||||
family = str(config.get("family", "")).strip()
|
||||
warnings: list[str] = []
|
||||
notes: list[str] = []
|
||||
options = _advanced_options(config)
|
||||
|
||||
# VibeVoice-ASR owns diarization across its full recording. Independent
|
||||
# Suite requests can restart speaker numbering, so preserve its native
|
||||
# chunking only for this mode. All other ASR uses Suite-side windows.
|
||||
native_diarization = (
|
||||
family == "vibevoice_asr" and req.diarization and req.chunk_size > 0
|
||||
)
|
||||
if native_diarization:
|
||||
options.setdefault("audio_chunk_mode", "fixed")
|
||||
options.setdefault("audio_chunk_seconds", int(req.chunk_size))
|
||||
if req.overlap > 0:
|
||||
notes.append(
|
||||
"VibeVoice-ASR diarization uses native chunking to preserve speaker "
|
||||
"identity; the Suite overlap setting is not applied."
|
||||
)
|
||||
elif family in _NATIVE_CHUNK_FAMILIES:
|
||||
options.setdefault("audio_chunk_mode", "none")
|
||||
|
||||
if req.timestamps == "word" and family == "qwen3_asr":
|
||||
session_options = config.get("session_options") or {}
|
||||
aligner = session_options.get("qwen3_asr.forced_aligner_model_path")
|
||||
if aligner:
|
||||
options["return_timestamps"] = True
|
||||
else:
|
||||
warnings.append(
|
||||
"Qwen3-ASR word timestamps require the optional Qwen3 Forced Aligner; "
|
||||
"transcription continued without downloading that auxiliary model."
|
||||
)
|
||||
|
||||
waveform, source_rate = _waveform_3d(req.audio)
|
||||
ranges = (
|
||||
[(0, waveform.shape[-1])]
|
||||
if native_diarization
|
||||
else _chunk_ranges(
|
||||
waveform.shape[-1], source_rate, int(req.chunk_size), int(req.overlap)
|
||||
)
|
||||
)
|
||||
session = _session(config)
|
||||
if str(getattr(session, "task", "asr")) != "asr":
|
||||
raise ValueError(
|
||||
f"audio.cpp model '{session.model_id}' is configured for task "
|
||||
f"'{session.task}', not ASR"
|
||||
)
|
||||
restart_between_chunks = (
|
||||
len(ranges) > 1 and family in _RESTART_BETWEEN_CHUNKS_FAMILIES
|
||||
)
|
||||
if restart_between_chunks and not bool(getattr(session, "owned", False)):
|
||||
raise RuntimeError(
|
||||
f"audio.cpp release-0.5.1 {family} returns empty text after its first "
|
||||
"offline request. Suite-side chunking therefore requires a managed "
|
||||
"audio.cpp server so the Suite can reset it between chunks. Set "
|
||||
"connection_mode to managed, or set ASR chunk_size to 0 when using "
|
||||
"an external server."
|
||||
)
|
||||
if restart_between_chunks:
|
||||
notes.append(
|
||||
f"audio.cpp release-0.5.1 {family} requires a managed server reset "
|
||||
"between Suite chunks to avoid empty repeated-request results."
|
||||
)
|
||||
|
||||
display_family = family or "external model"
|
||||
print(f"🎧 audio.cpp ASR: Transcribing with {display_family}...")
|
||||
if len(ranges) > 1:
|
||||
notes.append(
|
||||
f"Suite-side ASR chunking used {len(ranges)} windows of "
|
||||
f"{int(req.chunk_size)}s with {int(req.overlap)}s overlap."
|
||||
)
|
||||
print(
|
||||
f"🧩 audio.cpp ASR: {len(ranges)} chunks "
|
||||
f"({int(req.chunk_size)}s, {int(req.overlap)}s overlap)"
|
||||
)
|
||||
|
||||
payloads: list[Mapping[str, Any]] = []
|
||||
chunk_timings: list[Mapping[str, Any]] = []
|
||||
chunk_diagnostics: list[Dict[str, Any]] = []
|
||||
started_at = time.time()
|
||||
for index, (start, end) in enumerate(ranges, start=1):
|
||||
if index > 1 and restart_between_chunks:
|
||||
print(
|
||||
f"🔄 audio.cpp ASR: Resetting {family} session for chunk "
|
||||
f"{index}/{len(ranges)}"
|
||||
)
|
||||
session.restart_owned_runtime()
|
||||
chunk_waveform = waveform[..., start:end]
|
||||
chunk_rms = float(torch.sqrt(torch.mean(chunk_waveform.float().square())).item())
|
||||
chunk_peak = float(chunk_waveform.float().abs().max().item())
|
||||
temp_path = _audio_path({
|
||||
"waveform": chunk_waveform,
|
||||
"sample_rate": source_rate,
|
||||
})
|
||||
try:
|
||||
request: Dict[str, Any] = {"audio": temp_path, "options": dict(options)}
|
||||
if req.language:
|
||||
request["language"] = req.language
|
||||
result = session.run(request)
|
||||
payload = result.raw if isinstance(result.raw, Mapping) else {}
|
||||
payloads.append(payload)
|
||||
if isinstance(payload.get("timing"), Mapping):
|
||||
chunk_timings.append(payload["timing"])
|
||||
chunk_diagnostics.append({
|
||||
"index": index,
|
||||
"start": round(start / source_rate, 3),
|
||||
"end": round(end / source_rate, 3),
|
||||
"rms": round(chunk_rms, 6),
|
||||
"peak": round(chunk_peak, 6),
|
||||
"text": str(payload.get("text", "")).strip(),
|
||||
"characters": len(str(payload.get("text", "")).strip()),
|
||||
"upstream_timing": (
|
||||
dict(payload["timing"])
|
||||
if isinstance(payload.get("timing"), Mapping)
|
||||
else None
|
||||
),
|
||||
})
|
||||
finally:
|
||||
try:
|
||||
os.remove(temp_path)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
if len(ranges) > 1:
|
||||
chunk_chars = len(str(payload.get("text", "")).strip())
|
||||
print(
|
||||
f" ASR chunk {index}/{len(ranges)} complete "
|
||||
f"({chunk_chars} chars, RMS {chunk_rms:.4f}, peak {chunk_peak:.4f})"
|
||||
)
|
||||
|
||||
words: list[ASRWord] = []
|
||||
speaker_segments: list[ASRSegment] = []
|
||||
plain_segments: list[ASRSegment] = []
|
||||
overlap_seconds = float(req.overlap) if len(ranges) > 1 else 0.0
|
||||
for index, ((start, _end), payload) in enumerate(zip(ranges, payloads)):
|
||||
offset = start / source_rate
|
||||
unique_after = offset + overlap_seconds if index > 0 else None
|
||||
words.extend(_offset_words(_words(payload, source_rate), offset, unique_after))
|
||||
speaker_segments.extend(
|
||||
_offset_segments(
|
||||
_speaker_segments(payload, source_rate), offset, unique_after
|
||||
)
|
||||
)
|
||||
plain_segments.extend(
|
||||
_offset_segments(_plain_segments(payload, source_rate), offset, unique_after)
|
||||
)
|
||||
|
||||
if req.diarization:
|
||||
segments = speaker_segments
|
||||
if segments:
|
||||
_attach_words(segments, words)
|
||||
else:
|
||||
warnings.append(
|
||||
f"audio.cpp {family or 'ASR model'} returned no speaker-attributed turns."
|
||||
)
|
||||
segments = plain_segments
|
||||
elif req.timestamps == "word" and words:
|
||||
segments = [
|
||||
ASRSegment(start=word.start, end=word.end, text=word.text, words=[word])
|
||||
for word in words
|
||||
]
|
||||
elif req.timestamps == "word":
|
||||
segments = plain_segments
|
||||
else:
|
||||
segments = []
|
||||
|
||||
text = _merge_transcript(payload.get("text", "") for payload in payloads)
|
||||
if req.diarization and speaker_segments:
|
||||
text = " ".join(
|
||||
f"[{segment.speaker}] {segment.text}" if segment.speaker else segment.text
|
||||
for segment in speaker_segments
|
||||
if segment.text
|
||||
).strip()
|
||||
if not text and speaker_segments:
|
||||
text = " ".join(segment.text for segment in speaker_segments if segment.text).strip()
|
||||
if req.timestamps == "word" and not words:
|
||||
warnings.append(f"audio.cpp {family or 'ASR model'} returned no word timestamps.")
|
||||
empty_chunks = sum(
|
||||
1 for payload in payloads if not str(payload.get("text", "")).strip()
|
||||
)
|
||||
if len(payloads) > 1 and empty_chunks:
|
||||
warnings.append(
|
||||
f"audio.cpp {family or 'ASR model'} returned no text for "
|
||||
f"{empty_chunks} of {len(payloads)} Suite chunks."
|
||||
)
|
||||
|
||||
raw: Dict[str, Any] = {}
|
||||
if warnings:
|
||||
raw["warnings"] = warnings
|
||||
if notes:
|
||||
raw["notes"] = notes
|
||||
if len(payloads) == 1 and chunk_timings:
|
||||
raw["timing"] = dict(chunk_timings[0])
|
||||
elif len(payloads) > 1:
|
||||
raw["timing"] = {
|
||||
"wall_ms": round((time.time() - started_at) * 1000.0, 3),
|
||||
"suite_chunks": len(payloads),
|
||||
"suite_chunk_size_seconds": int(req.chunk_size),
|
||||
"suite_overlap_seconds": int(req.overlap),
|
||||
"upstream_wall_ms": round(
|
||||
sum(float(item.get("wall_ms", 0.0)) for item in chunk_timings), 3
|
||||
),
|
||||
}
|
||||
raw["chunks"] = chunk_diagnostics
|
||||
output_language = next(
|
||||
(
|
||||
str(payload.get("language", "")).strip()
|
||||
for payload in payloads
|
||||
if str(payload.get("language", "")).strip()
|
||||
),
|
||||
str(req.language or "").strip(),
|
||||
) or None
|
||||
print(
|
||||
f"✅ audio.cpp ASR: Complete ({len(text)} chars, "
|
||||
f"{len(segments)} timed/speaker segments)"
|
||||
)
|
||||
return ASRResult(
|
||||
text=text,
|
||||
language=output_language,
|
||||
segments=segments,
|
||||
raw=raw or None,
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["AudioCppASREngineAdapter"]
|
||||
@@ -0,0 +1,372 @@
|
||||
"""Adapter between the suite's TTS processors and an audio.cpp session."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import threading
|
||||
from typing import Any, Dict, Mapping, Optional, Tuple
|
||||
|
||||
import torch
|
||||
|
||||
from utils.audio.audio_hash import generate_stable_audio_component
|
||||
from utils.audio.cache import get_audio_cache
|
||||
from utils.audio.processing import AudioProcessingUtils
|
||||
from utils.voice.reference import effective_voice_audio
|
||||
|
||||
|
||||
_CACHE_SAMPLE_RATES: Dict[str, int] = {}
|
||||
_CACHE_SAMPLE_RATES_LOCK = threading.Lock()
|
||||
|
||||
|
||||
def _get_session(config: Mapping[str, Any]):
|
||||
"""Import lazily so the node can still be discovered before optional setup."""
|
||||
from utils.audio_cpp.session import get_audio_cpp_session
|
||||
|
||||
return get_audio_cpp_session(dict(config))
|
||||
|
||||
|
||||
def _canonical_json(value: Mapping[str, Any]) -> str:
|
||||
return json.dumps(value, sort_keys=True, separators=(",", ":"), default=str)
|
||||
|
||||
|
||||
class AudioCppEngineAdapter:
|
||||
"""Build generic ``/v1/tasks/run`` requests and retain their real sample rate."""
|
||||
|
||||
_COMMON_REQUEST_FIELDS = (
|
||||
"temperature",
|
||||
"top_p",
|
||||
"top_k",
|
||||
"repetition_penalty",
|
||||
"max_tokens",
|
||||
"max_steps",
|
||||
"num_inference_steps",
|
||||
"guidance_scale",
|
||||
"speaking_rate",
|
||||
)
|
||||
|
||||
def __init__(self, config: Optional[Dict[str, Any]] = None):
|
||||
self.config = dict(config or {})
|
||||
self.audio_cache = get_audio_cache()
|
||||
self._last_sample_rate: Optional[int] = None
|
||||
self._reference_files: Dict[str, str] = {}
|
||||
self._reference_lock = threading.RLock()
|
||||
|
||||
@property
|
||||
def sample_rate(self) -> Optional[int]:
|
||||
return self._last_sample_rate
|
||||
|
||||
def update_config(self, new_config: Optional[Dict[str, Any]]) -> None:
|
||||
self.config = dict(new_config or {})
|
||||
|
||||
@staticmethod
|
||||
def _reference_text(voice_ref: Any) -> str:
|
||||
if not isinstance(voice_ref, Mapping):
|
||||
return ""
|
||||
return str(
|
||||
voice_ref.get("reference_text")
|
||||
or voice_ref.get("prompt_text")
|
||||
or voice_ref.get("text")
|
||||
or ""
|
||||
).strip()
|
||||
|
||||
def _materialize_reference(self, voice_ref: Any) -> Tuple[Optional[str], str, str, Optional[str]]:
|
||||
"""Return path, transcript, stable hash, and the path that must be removed."""
|
||||
reference_text = AudioCppEngineAdapter._reference_text(voice_ref)
|
||||
if not isinstance(voice_ref, Mapping):
|
||||
return None, reference_text, "default_voice", None
|
||||
|
||||
audio = effective_voice_audio(voice_ref)
|
||||
if audio is None:
|
||||
return None, reference_text, "default_voice", None
|
||||
|
||||
if isinstance(audio, (str, os.PathLike)):
|
||||
path = os.path.abspath(os.path.expanduser(os.fspath(audio)))
|
||||
if not os.path.isfile(path):
|
||||
raise FileNotFoundError(f"audio.cpp reference audio not found: {path}")
|
||||
component = generate_stable_audio_component(audio_file_path=path)
|
||||
return path, reference_text, component, None
|
||||
|
||||
if isinstance(audio, Mapping):
|
||||
waveform = audio.get("waveform")
|
||||
sample_rate = audio.get("sample_rate")
|
||||
audio_dict = dict(audio)
|
||||
elif torch.is_tensor(audio):
|
||||
waveform = audio
|
||||
sample_rate = voice_ref.get("sample_rate")
|
||||
audio_dict = {"waveform": waveform, "sample_rate": sample_rate}
|
||||
else:
|
||||
raise TypeError(f"Unsupported audio.cpp voice reference type: {type(audio).__name__}")
|
||||
|
||||
if not torch.is_tensor(waveform):
|
||||
raise TypeError("audio.cpp reference audio must contain a waveform tensor")
|
||||
if sample_rate is None or int(sample_rate) <= 0:
|
||||
raise ValueError("audio.cpp reference audio must contain a positive sample_rate")
|
||||
|
||||
audio_dict["sample_rate"] = int(sample_rate)
|
||||
component = generate_stable_audio_component(reference_audio=audio_dict)
|
||||
if component not in {"ref_audio_error", "ref_audio_error_not_tensor"}:
|
||||
with self._reference_lock:
|
||||
cached_path = self._reference_files.get(component)
|
||||
if cached_path and os.path.isfile(cached_path):
|
||||
return cached_path, reference_text, component, None
|
||||
temp_path = os.path.abspath(
|
||||
AudioProcessingUtils.save_audio_to_temp_file(waveform, int(sample_rate))
|
||||
)
|
||||
self._reference_files[component] = temp_path
|
||||
return temp_path, reference_text, component, None
|
||||
|
||||
# Hash failures must not make unrelated references share one file.
|
||||
temp_path = os.path.abspath(
|
||||
AudioProcessingUtils.save_audio_to_temp_file(waveform, int(sample_rate))
|
||||
)
|
||||
return temp_path, reference_text, component, temp_path
|
||||
|
||||
def close(self) -> None:
|
||||
with self._reference_lock:
|
||||
paths = list(self._reference_files.values())
|
||||
self._reference_files.clear()
|
||||
for path in paths:
|
||||
try:
|
||||
os.remove(path)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def __del__(self):
|
||||
try:
|
||||
self.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def _advanced_options(self) -> Dict[str, Any]:
|
||||
value = self.config.get(
|
||||
"advanced_options",
|
||||
self.config.get("request_options", self.config.get("advanced_json", {})),
|
||||
)
|
||||
if value in (None, ""):
|
||||
return {}
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
value = json.loads(value)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"Invalid audio.cpp advanced JSON: {exc.msg}") from exc
|
||||
if not isinstance(value, Mapping):
|
||||
raise ValueError("audio.cpp advanced options must be a JSON object")
|
||||
return dict(value)
|
||||
|
||||
def _resolved_task(self, session: Any) -> str:
|
||||
requested = str(self.config.get("task", self.config.get("requested_task", "auto"))).lower()
|
||||
for source in (session, getattr(session, "config", None)):
|
||||
if source is None:
|
||||
continue
|
||||
value = source.get("task") if isinstance(source, Mapping) else getattr(source, "task", None)
|
||||
if str(value).lower() in {"tts", "clon", "vdes"}:
|
||||
return str(value).lower()
|
||||
|
||||
if requested in {"tts", "clon", "vdes"}:
|
||||
return requested
|
||||
if str(self.config.get("connection_mode", "auto")).lower() == "external_server":
|
||||
return "auto"
|
||||
try:
|
||||
from utils.audio_cpp.catalog import resolve_task
|
||||
|
||||
return str(
|
||||
resolve_task(
|
||||
self.config.get("family", ""),
|
||||
self.config.get("package_id", ""),
|
||||
requested="auto",
|
||||
)
|
||||
).lower()
|
||||
except (ImportError, KeyError, TypeError, ValueError):
|
||||
return "tts"
|
||||
|
||||
def _build_request(
|
||||
self,
|
||||
text: str,
|
||||
voice_path: Optional[str],
|
||||
reference_text: str,
|
||||
seed: int,
|
||||
advanced: Dict[str, Any],
|
||||
task: str,
|
||||
) -> Dict[str, Any]:
|
||||
request: Dict[str, Any] = {"text": text, "seed": str(int(seed)), "options": advanced}
|
||||
del task # The persistent session owns its one configured model/task.
|
||||
|
||||
language = str(self.config.get("language", "")).strip()
|
||||
if language and language.lower() not in {"auto", "none"}:
|
||||
request["language"] = language
|
||||
voice_id = str(self.config.get("voice_id", self.config.get("voice", ""))).strip()
|
||||
if voice_id:
|
||||
request["voice_id"] = voice_id
|
||||
if voice_path:
|
||||
request["voice_ref"] = voice_path
|
||||
if reference_text:
|
||||
request["reference_text"] = reference_text
|
||||
instruct = str(self.config.get("instruct", "")).strip()
|
||||
if instruct:
|
||||
request["instruct"] = instruct
|
||||
|
||||
for key in self._COMMON_REQUEST_FIELDS:
|
||||
value = self.config.get(key)
|
||||
if value is not None and value != "":
|
||||
request[key] = value
|
||||
return request
|
||||
|
||||
def _cache_key(
|
||||
self,
|
||||
text: str,
|
||||
audio_component: str,
|
||||
reference_text: str,
|
||||
seed: int,
|
||||
task: str,
|
||||
advanced: Dict[str, Any],
|
||||
character_name: Optional[str],
|
||||
session: Any,
|
||||
) -> str:
|
||||
session_config = getattr(session, "config", {})
|
||||
if not isinstance(session_config, Mapping):
|
||||
session_config = {}
|
||||
session_family = getattr(session, "family", None) or session_config.get(
|
||||
"family", self.config.get("family", "")
|
||||
)
|
||||
session_model_id = getattr(session, "model_id", None) or session_config.get(
|
||||
"model_id", self.config.get("model_id", "")
|
||||
)
|
||||
# Owned servers use a random loopback port on every restart; that port is
|
||||
# transport state, not model identity. External endpoints are stable and
|
||||
# must participate in the cache key.
|
||||
if bool(getattr(session, "owned", False)):
|
||||
session_endpoint = ""
|
||||
else:
|
||||
session_endpoint = getattr(session, "endpoint", None) or self.config.get(
|
||||
"server_url", self.config.get("external_server_url", "")
|
||||
)
|
||||
extra_identity = {
|
||||
"options": advanced,
|
||||
"speaking_rate": self.config.get("speaking_rate"),
|
||||
"connection_mode": self.config.get("connection_mode", "auto"),
|
||||
"server_url": session_endpoint,
|
||||
"binary_path": session_config.get("binary_path", self.config.get("binary_path", "")),
|
||||
"backend": session_config.get("backend", self.config.get("backend", "")),
|
||||
"device": session_config.get("device", self.config.get("device", "")),
|
||||
"load_options": session_config.get("load_options", self.config.get("load_options", {})),
|
||||
"session_options": session_config.get(
|
||||
"session_options", self.config.get("session_options", {})
|
||||
),
|
||||
"default_request_options": session_config.get(
|
||||
"default_request_options", self.config.get("default_request_options", {})
|
||||
),
|
||||
}
|
||||
return self.audio_cache.generate_cache_key(
|
||||
"audio_cpp",
|
||||
text=text,
|
||||
audio_component=audio_component,
|
||||
reference_text=reference_text,
|
||||
family=session_family,
|
||||
package_id=session_config.get("package_id", self.config.get("package_id", "")),
|
||||
model_path=session_config.get("model_path", self.config.get("model_path", "")),
|
||||
model_id=session_model_id,
|
||||
task=task,
|
||||
language=self.config.get("language", ""),
|
||||
voice_id=self.config.get("voice_id", self.config.get("voice", "")),
|
||||
instruct=self.config.get("instruct", ""),
|
||||
temperature=self.config.get("temperature"),
|
||||
top_p=self.config.get("top_p"),
|
||||
top_k=self.config.get("top_k"),
|
||||
repetition_penalty=self.config.get("repetition_penalty"),
|
||||
max_tokens=self.config.get("max_tokens"),
|
||||
max_steps=self.config.get("max_steps"),
|
||||
num_inference_steps=self.config.get("num_inference_steps"),
|
||||
guidance_scale=self.config.get("guidance_scale"),
|
||||
seed=int(seed),
|
||||
request_options=_canonical_json(extra_identity),
|
||||
character=character_name or "narrator",
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _normalize_result(result: Any) -> Tuple[torch.Tensor, int]:
|
||||
waveform = result.get("waveform") if isinstance(result, Mapping) else getattr(result, "waveform", None)
|
||||
sample_rate = result.get("sample_rate") if isinstance(result, Mapping) else getattr(result, "sample_rate", None)
|
||||
|
||||
if waveform is None:
|
||||
named = result.get("named_audio", {}) if isinstance(result, Mapping) else getattr(result, "named_audio", {})
|
||||
values = list(named.values()) if isinstance(named, Mapping) else list(named or [])
|
||||
if len(values) == 1:
|
||||
item = values[0]
|
||||
waveform = item.get("waveform") if isinstance(item, Mapping) else getattr(item, "waveform", None)
|
||||
sample_rate = sample_rate or (item.get("sample_rate") if isinstance(item, Mapping) else getattr(item, "sample_rate", None))
|
||||
|
||||
if waveform is None:
|
||||
raise RuntimeError("audio.cpp returned no primary audio output")
|
||||
if not torch.is_tensor(waveform):
|
||||
waveform = torch.as_tensor(waveform, dtype=torch.float32)
|
||||
waveform = waveform.detach().to(device="cpu", dtype=torch.float32)
|
||||
if waveform.dim() == 1:
|
||||
waveform = waveform.unsqueeze(0)
|
||||
elif waveform.dim() == 3 and waveform.shape[0] == 1:
|
||||
waveform = waveform.squeeze(0)
|
||||
if waveform.dim() != 2:
|
||||
raise ValueError(f"audio.cpp waveform must be [channels, samples], got {tuple(waveform.shape)}")
|
||||
if sample_rate is None or int(sample_rate) <= 0:
|
||||
raise ValueError("audio.cpp returned an invalid sample rate")
|
||||
return waveform.contiguous(), int(sample_rate)
|
||||
|
||||
def generate_single(
|
||||
self,
|
||||
text: str,
|
||||
voice_ref: Optional[Dict[str, Any]] = None,
|
||||
seed: int = 0,
|
||||
enable_audio_cache: bool = True,
|
||||
character_name: Optional[str] = None,
|
||||
) -> Tuple[torch.Tensor, int]:
|
||||
stripped = str(text or "").strip()
|
||||
if not stripped:
|
||||
if self._last_sample_rate is None:
|
||||
raise ValueError("audio.cpp cannot determine a sample rate for empty text")
|
||||
return torch.zeros(1, 0, dtype=torch.float32), self._last_sample_rate
|
||||
|
||||
session = _get_session(self.config)
|
||||
task = self._resolved_task(session)
|
||||
advanced = self._advanced_options()
|
||||
cleanup_path: Optional[str] = None
|
||||
try:
|
||||
voice_path, reference_text, audio_component, cleanup_path = self._materialize_reference(voice_ref)
|
||||
cache_key = self._cache_key(
|
||||
stripped,
|
||||
audio_component,
|
||||
reference_text,
|
||||
seed,
|
||||
task,
|
||||
advanced,
|
||||
character_name,
|
||||
session,
|
||||
)
|
||||
if enable_audio_cache:
|
||||
cached = self.audio_cache.get_cached_audio(cache_key)
|
||||
with _CACHE_SAMPLE_RATES_LOCK:
|
||||
cached_rate = _CACHE_SAMPLE_RATES.get(cache_key)
|
||||
if cached is not None and cached_rate is not None:
|
||||
self._last_sample_rate = cached_rate
|
||||
return cached[0].clone(), cached_rate
|
||||
|
||||
request = self._build_request(stripped, voice_path, reference_text, seed, advanced, task)
|
||||
waveform, sample_rate = self._normalize_result(session.run(request))
|
||||
self._last_sample_rate = sample_rate
|
||||
if enable_audio_cache:
|
||||
duration = waveform.shape[-1] / sample_rate
|
||||
self.audio_cache.cache_audio(cache_key, waveform, duration)
|
||||
with _CACHE_SAMPLE_RATES_LOCK:
|
||||
_CACHE_SAMPLE_RATES[cache_key] = sample_rate
|
||||
return waveform, sample_rate
|
||||
finally:
|
||||
if cleanup_path:
|
||||
try:
|
||||
os.remove(cleanup_path)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
|
||||
# Short alias for callers that do not use the older ``EngineAdapter`` suffix.
|
||||
AudioCppAdapter = AudioCppEngineAdapter
|
||||
@@ -0,0 +1,111 @@
|
||||
"""audio.cpp adapter for the Suite's unified Voice Changer node."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from typing import Any, Dict, Mapping
|
||||
|
||||
import torch
|
||||
|
||||
from engines.adapters.audio_cpp_adapter import AudioCppEngineAdapter
|
||||
from utils.audio.processing import AudioProcessingUtils
|
||||
|
||||
|
||||
def _advanced_options(config: Mapping[str, Any]) -> Dict[str, Any]:
|
||||
value = config.get("advanced_options", config.get("request_options", {}))
|
||||
if value in (None, ""):
|
||||
return {}
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
value = json.loads(value)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"Invalid audio.cpp advanced JSON: {exc.msg}") from exc
|
||||
if not isinstance(value, Mapping):
|
||||
raise ValueError("audio.cpp advanced options must be a JSON object")
|
||||
return dict(value)
|
||||
|
||||
|
||||
def _materialize(audio: Mapping[str, Any], label: str) -> str:
|
||||
waveform = audio.get("waveform")
|
||||
sample_rate = audio.get("sample_rate")
|
||||
if not torch.is_tensor(waveform):
|
||||
raise TypeError(f"audio.cpp {label} must contain a waveform tensor")
|
||||
if sample_rate is None or int(sample_rate) <= 0:
|
||||
raise ValueError(f"audio.cpp {label} must contain a positive sample_rate")
|
||||
return os.path.abspath(
|
||||
AudioProcessingUtils.save_audio_to_temp_file(waveform, int(sample_rate))
|
||||
)
|
||||
|
||||
|
||||
class AudioCppVoiceConversionAdapter:
|
||||
"""Convert source audio toward a target reference using an audio.cpp VC task."""
|
||||
|
||||
def __init__(self, config: Dict[str, Any]):
|
||||
self.config = dict(config)
|
||||
|
||||
def _session_config(self) -> Dict[str, Any]:
|
||||
config = dict(self.config)
|
||||
if str(config.get("connection_mode", "auto")).lower() != "external_server":
|
||||
config["requested_task"] = "vc"
|
||||
config["task"] = "vc"
|
||||
return config
|
||||
|
||||
def convert_voice(
|
||||
self,
|
||||
source_audio: Dict[str, Any],
|
||||
target_audio: Dict[str, Any],
|
||||
refinement_passes: int = 1,
|
||||
) -> tuple[Dict[str, Any], str]:
|
||||
from utils.audio_cpp.session import get_audio_cpp_session
|
||||
|
||||
config = self._session_config()
|
||||
family = str(config.get("family", "")).strip()
|
||||
passes = max(1, int(refinement_passes))
|
||||
current = source_audio
|
||||
output_rate = int(source_audio["sample_rate"])
|
||||
|
||||
session = get_audio_cpp_session(config)
|
||||
if str(getattr(session, "task", "vc")) != "vc":
|
||||
raise ValueError(
|
||||
f"audio.cpp model '{session.model_id}' is configured for task "
|
||||
f"'{session.task}', not voice conversion"
|
||||
)
|
||||
|
||||
for pass_index in range(passes):
|
||||
source_path = _materialize(current, "source audio")
|
||||
target_path = _materialize(target_audio, "target reference audio")
|
||||
try:
|
||||
request = {
|
||||
"audio": source_path,
|
||||
"voice_ref": target_path,
|
||||
"source_audio": source_path,
|
||||
"target_voice": target_path,
|
||||
"options": _advanced_options(config),
|
||||
}
|
||||
print(
|
||||
f"🔄 audio.cpp VC: {family or 'external model'} pass "
|
||||
f"{pass_index + 1}/{passes}..."
|
||||
)
|
||||
result = session.run(request)
|
||||
waveform, output_rate = AudioCppEngineAdapter._normalize_result(result)
|
||||
current = {"waveform": waveform.unsqueeze(0), "sample_rate": output_rate}
|
||||
finally:
|
||||
for path in (source_path, target_path):
|
||||
try:
|
||||
os.remove(path)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
info = (
|
||||
f"Model family: {family or getattr(session, 'family', 'external')}\n"
|
||||
f"Model ID: {session.model_id}\n"
|
||||
f"Task: voice conversion\n"
|
||||
f"Refinement passes: {passes}\n"
|
||||
f"Output sample rate: {output_rate} Hz\n"
|
||||
"Conversion completed successfully"
|
||||
)
|
||||
return current, info
|
||||
|
||||
|
||||
__all__ = ["AudioCppVoiceConversionAdapter"]
|
||||
@@ -67,43 +67,7 @@ class ChatterBoxEngineAdapter:
|
||||
"""
|
||||
self.node.load_tts_model(device, language)
|
||||
|
||||
def create_segment_cache(self, character="narrator", *, cache_model_name=None,
|
||||
cache_probe=False, **params):
|
||||
"""Share cache identity between generation and pre-load probes."""
|
||||
if not params.get("enable_audio_cache", True):
|
||||
return None
|
||||
from utils.audio.cache import create_cache_function
|
||||
|
||||
language = cache_model_name or params.get("current_language", params.get("model", "English"))
|
||||
device = params.get("device", "auto")
|
||||
model_source = params.get("model_source")
|
||||
if not model_source:
|
||||
# Do not borrow the source of a different currently loaded language.
|
||||
sources = getattr(self, "_cache_model_sources", {})
|
||||
if cache_probe:
|
||||
model_source = sources.get((language, device))
|
||||
if model_source is None:
|
||||
return None
|
||||
else:
|
||||
if hasattr(self.node, "model_manager"):
|
||||
model_source = self.node.model_manager.get_model_source("tts")
|
||||
model_source = model_source or "unknown"
|
||||
sources[(language, device)] = model_source
|
||||
self._cache_model_sources = sources
|
||||
|
||||
audio_component = params.get("stable_audio_component", "main_reference")
|
||||
if character != "narrator":
|
||||
audio_component = f"char_file_{character}"
|
||||
return create_cache_function(
|
||||
engine_type="chatterbox", character=character,
|
||||
exaggeration=params.get("exaggeration", 1.0),
|
||||
temperature=params.get("temperature", 0.8),
|
||||
cfg_weight=params.get("cfg_weight", 1.0),
|
||||
seed=params.get("seed", 0), audio_component=audio_component,
|
||||
model_source=model_source, device=device, language=language,
|
||||
)
|
||||
|
||||
def generate_segment_audio(self, text: str, char_audio: str,
|
||||
def generate_segment_audio(self, text: str, char_audio: str,
|
||||
character: str = "narrator", **params) -> torch.Tensor:
|
||||
"""
|
||||
Generate ChatterBox audio for a text segment.
|
||||
@@ -124,7 +88,34 @@ class ChatterBoxEngineAdapter:
|
||||
seed = params.get("seed", 0)
|
||||
enable_cache = params.get("enable_audio_cache", True)
|
||||
|
||||
cache_fn = self.create_segment_cache(character=character, **params)
|
||||
# Create cache function if caching is enabled
|
||||
cache_fn = None
|
||||
if enable_cache:
|
||||
from utils.audio.cache import create_cache_function
|
||||
|
||||
# Get current language/model for cache key
|
||||
current_language = params.get("current_language", params.get("model", "English"))
|
||||
audio_component = params.get("stable_audio_component", "main_reference")
|
||||
if character != "narrator":
|
||||
audio_component = f"char_file_{character}"
|
||||
|
||||
# Get model source
|
||||
model_source = params.get("model_source")
|
||||
if not model_source and hasattr(self.node, 'model_manager'):
|
||||
model_source = self.node.model_manager.get_model_source("tts")
|
||||
|
||||
cache_fn = create_cache_function(
|
||||
engine_type="chatterbox",
|
||||
character=character,
|
||||
exaggeration=exaggeration,
|
||||
temperature=temperature,
|
||||
cfg_weight=cfg_weight,
|
||||
seed=seed,
|
||||
audio_component=audio_component,
|
||||
model_source=model_source or "unknown",
|
||||
device=params.get("device", "auto"),
|
||||
language=current_language
|
||||
)
|
||||
|
||||
# Handle caching externally for consistency with F5-TTS
|
||||
if cache_fn:
|
||||
@@ -151,7 +142,8 @@ class ChatterBoxEngineAdapter:
|
||||
enable_pause_tags=True,
|
||||
character=character,
|
||||
seed=seed,
|
||||
enable_cache=enable_cache,
|
||||
enable_cache=True, # Enable internal caching for pause tag processing
|
||||
crash_protection_template=params.get("crash_protection_template", "hmm ,, {seg} hmm ,,"),
|
||||
stable_audio_component=params.get("stable_audio_component", "main_reference")
|
||||
)
|
||||
# Don't use external cache for pause tag segments
|
||||
@@ -168,6 +160,7 @@ class ChatterBoxEngineAdapter:
|
||||
character=character,
|
||||
seed=seed,
|
||||
enable_cache=False, # Disable internal caching since we handle it externally
|
||||
crash_protection_template=params.get("crash_protection_template", "hmm ,, {seg} hmm ,,"),
|
||||
stable_audio_component=params.get("stable_audio_component", "main_reference")
|
||||
)
|
||||
|
||||
@@ -407,4 +400,4 @@ class ChatterBoxEngineAdapter:
|
||||
else:
|
||||
num_samples = audio_tensor.numel()
|
||||
|
||||
return num_samples / 44100 # ChatterBox sample rate
|
||||
return num_samples / 44100 # ChatterBox sample rate
|
||||
@@ -110,6 +110,7 @@ class ChatterBoxStreamingAdapter(StreamingEngineAdapter):
|
||||
"seed": kwargs.get("seed", 42),
|
||||
"enable_chunking": False, # Don't chunk in streaming - already handled
|
||||
"enable_audio_cache": kwargs.get("enable_audio_cache", True),
|
||||
"crash_protection_template": kwargs.get("crash_protection_template", "hmm ,, {seg} hmm ,,"),
|
||||
"device": kwargs.get("device", "auto"),
|
||||
"reference_audio": kwargs.get("reference_audio", None)
|
||||
}
|
||||
@@ -299,6 +300,7 @@ class ChatterBoxStreamingAdapter(StreamingEngineAdapter):
|
||||
cfg_weight = kwargs.get('cfg_weight', 0.5)
|
||||
seed = kwargs.get('seed', 42)
|
||||
enable_cache = kwargs.get('enable_audio_cache', True)
|
||||
crash_protection = kwargs.get('crash_protection_template', 'hmm ,, {seg} hmm ,,')
|
||||
|
||||
# Check if node has generation method
|
||||
if hasattr(self.node, '_generate_tts_with_pause_tags'):
|
||||
@@ -319,6 +321,7 @@ class ChatterBoxStreamingAdapter(StreamingEngineAdapter):
|
||||
text, voice_path, exaggeration, temperature, cfg_weight,
|
||||
language, True, character=character, seed=seed,
|
||||
enable_cache=enable_cache,
|
||||
crash_protection_template=crash_protection,
|
||||
stable_audio_component=stable_audio_component
|
||||
)
|
||||
elif hasattr(self.node, 'tts_model') and self.node.tts_model:
|
||||
|
||||
@@ -73,30 +73,7 @@ class F5TTSEngineAdapter:
|
||||
|
||||
self.node.load_f5tts_model(model_name, device)
|
||||
|
||||
def create_segment_cache(self, character="narrator", char_text=None, *,
|
||||
cache_model_name=None, cache_probe=False, **params):
|
||||
"""Share cache identity between generation and pre-load probes."""
|
||||
if not params.get("enable_audio_cache", True):
|
||||
return None
|
||||
from utils.audio.cache import create_cache_function
|
||||
|
||||
audio_component = params.get("stable_audio_component", "main_reference")
|
||||
if character != "narrator":
|
||||
audio_component = f"char_file_{character}"
|
||||
model_name = cache_model_name or getattr(
|
||||
self.node, "current_model_name", params.get("model", "F5TTS_Base")
|
||||
)
|
||||
return create_cache_function(
|
||||
engine_type="f5tts", character=character, model_name=model_name,
|
||||
device=params.get("device", "auto"), audio_component=audio_component,
|
||||
ref_text=char_text, temperature=params.get("temperature", 0.8),
|
||||
speed=params.get("speed", 1.0), target_rms=params.get("target_rms", 0.1),
|
||||
cross_fade_duration=params.get("cross_fade_duration", 0.15),
|
||||
nfe_step=max(1, min(params.get("nfe_step", 32), 71)),
|
||||
cfg_strength=params.get("cfg_strength", 2.0), seed=params.get("seed", 0),
|
||||
)
|
||||
|
||||
def generate_segment_audio(self, text: str, char_audio: str, char_text: str,
|
||||
def generate_segment_audio(self, text: str, char_audio: str, char_text: str,
|
||||
character: str = "narrator", **params) -> torch.Tensor:
|
||||
"""
|
||||
Generate F5-TTS audio for a text segment.
|
||||
@@ -126,7 +103,34 @@ class F5TTSEngineAdapter:
|
||||
if safe_nfe_step != nfe_step:
|
||||
print(f"⚠️ F5-TTS: Clamped nfe_step from {nfe_step} to {safe_nfe_step} to prevent ODE solver issues")
|
||||
|
||||
cache_fn = self.create_segment_cache(character=character, char_text=char_text, **params)
|
||||
# Create cache function if caching is enabled
|
||||
cache_fn = None
|
||||
if enable_cache:
|
||||
from utils.audio.cache import create_cache_function
|
||||
|
||||
# Get audio component for cache key
|
||||
audio_component = params.get("stable_audio_component", "main_reference")
|
||||
if character != "narrator":
|
||||
audio_component = f"char_file_{character}"
|
||||
|
||||
# Get current model name for cache key
|
||||
current_model = getattr(self.node, 'current_model_name', params.get("model", "F5TTS_Base"))
|
||||
|
||||
cache_fn = create_cache_function(
|
||||
engine_type="f5tts",
|
||||
character=character,
|
||||
model_name=current_model,
|
||||
device=params.get("device", "auto"),
|
||||
audio_component=audio_component,
|
||||
ref_text=char_text,
|
||||
temperature=temperature,
|
||||
speed=speed,
|
||||
target_rms=target_rms,
|
||||
cross_fade_duration=cross_fade_duration,
|
||||
nfe_step=safe_nfe_step,
|
||||
cfg_strength=cfg_strength,
|
||||
seed=seed
|
||||
)
|
||||
|
||||
# Generate audio using F5-TTS with pause tag support
|
||||
return self.node.generate_f5tts_with_pause_tags(
|
||||
@@ -185,4 +189,4 @@ class F5TTSEngineAdapter:
|
||||
else:
|
||||
num_samples = audio_tensor.numel()
|
||||
|
||||
return num_samples / 24000 # F5-TTS sample rate
|
||||
return num_samples / 24000 # F5-TTS sample rate
|
||||
@@ -74,10 +74,7 @@ class MaskedDiffWithXvec(torch.nn.Module):
|
||||
|
||||
# concat text and prompt_text
|
||||
mask = (~make_pad_mask(token_len)).float().unsqueeze(-1).to(device)
|
||||
# TTS Audio Suite patch: bound speech-token indices to prevent CUDA embedding asserts.
|
||||
token = self.input_embedding(
|
||||
torch.clamp(token, min=0, max=self.input_embedding.num_embeddings - 1)
|
||||
) * mask
|
||||
token = self.input_embedding(torch.clamp(token, min=0)) * mask
|
||||
|
||||
# text encode
|
||||
h, h_lengths = self.encoder(token, token_len)
|
||||
@@ -127,10 +124,7 @@ class MaskedDiffWithXvec(torch.nn.Module):
|
||||
token_len1, token_len2 = prompt_token.shape[1], token.shape[1]
|
||||
token, token_len = torch.concat([prompt_token, token], dim=1), prompt_token_len + token_len
|
||||
mask = (~make_pad_mask(token_len)).unsqueeze(-1).to(embedding)
|
||||
# TTS Audio Suite patch: bound speech-token indices to prevent CUDA embedding asserts.
|
||||
token = self.input_embedding(
|
||||
torch.clamp(token, min=0, max=self.input_embedding.num_embeddings - 1)
|
||||
) * mask
|
||||
token = self.input_embedding(torch.clamp(token, min=0)) * mask
|
||||
|
||||
# text encode
|
||||
h, h_lengths = self.encoder(token, token_len)
|
||||
@@ -221,10 +215,7 @@ class CausalMaskedDiffWithXvec(torch.nn.Module):
|
||||
# concat text and prompt_text
|
||||
token, token_len = torch.concat([prompt_token, token], dim=1), prompt_token_len + token_len
|
||||
mask = (~make_pad_mask(token_len)).unsqueeze(-1).to(embedding)
|
||||
# TTS Audio Suite patch: bound speech-token indices to prevent CUDA embedding asserts.
|
||||
token = self.input_embedding(
|
||||
torch.clamp(token, min=0, max=self.input_embedding.num_embeddings - 1)
|
||||
) * mask
|
||||
token = self.input_embedding(torch.clamp(token, min=0)) * mask
|
||||
|
||||
# text encode
|
||||
h, h_lengths = self.encoder(token, token_len)
|
||||
|
||||
@@ -1,5 +1,3 @@
|
||||
import logging
|
||||
|
||||
from .s3tokenizer import (
|
||||
S3_SR,
|
||||
S3_HOP,
|
||||
@@ -29,20 +27,4 @@ def drop_invalid_tokens(x):
|
||||
e = None
|
||||
|
||||
x = x[s: e]
|
||||
|
||||
# T3's speech head spans `speech_tokens_dict_size` (8194) while S3Gen's
|
||||
# input embedding only has SPEECH_VOCAB_SIZE (6561) rows, so a sampled id
|
||||
# above the vocab that is not exactly SoS/EoS reaches the embedding gather
|
||||
# and trips a device-side "index out of bounds" assert (which surfaces
|
||||
# later, at the next CUDA sync, with a misleading traceback).
|
||||
invalid = x >= SPEECH_VOCAB_SIZE
|
||||
if invalid.any():
|
||||
logging.warning(
|
||||
"Dropping %d out-of-vocabulary speech token(s) (max id %d, vocab %d)",
|
||||
int(invalid.sum()), int(x.max()), SPEECH_VOCAB_SIZE,
|
||||
)
|
||||
x = x[~invalid]
|
||||
# TTS Audio Suite patch: do not pass an empty speech stream to S3Gen.
|
||||
if x.numel() == 0:
|
||||
raise RuntimeError("ChatterBox generated no valid speech tokens. Try another seed or text.")
|
||||
return x
|
||||
|
||||
@@ -16,6 +16,7 @@ from .modules.cond_enc import T3CondEnc, T3Cond
|
||||
from .modules.t3_config import T3Config
|
||||
from .llama_configs import LLAMA_CONFIGS
|
||||
from .inference.t3_hf_backend import T3HuggingfaceBackend
|
||||
from .inference.alignment_stream_analyzer import AlignmentStreamAnalyzer
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -276,9 +277,13 @@ class T3(nn.Module):
|
||||
self.tfmr.use_cache = True
|
||||
self.tfmr.return_dict = True
|
||||
|
||||
# TTS Audio Suite patch: alignment analysis is disabled in the backend, so do
|
||||
# not install its persistent attention hook/forward wrapper on every inference.
|
||||
alignment_stream_analyzer = None
|
||||
alignment_stream_analyzer = AlignmentStreamAnalyzer(
|
||||
self.tfmr,
|
||||
None,
|
||||
text_tokens_slice=(len_cond, len_cond + text_tokens.size(-1)),
|
||||
alignment_layer_idx=9,
|
||||
eos_idx=self.hp.stop_speech_token,
|
||||
)
|
||||
|
||||
# Create backend with updated settings
|
||||
patched_model = T3HuggingfaceBackend(
|
||||
|
||||
@@ -268,7 +268,7 @@ class ChatterboxTTS:
|
||||
if custom_path.suffix == ".safetensors":
|
||||
return load_file(custom_path, device=device)
|
||||
if custom_path.suffix == ".pt":
|
||||
return torch.load(custom_path, map_location=device, weights_only=True)
|
||||
return torch.load(custom_path, map_location=device)
|
||||
|
||||
safetensors_path = ckpt_dir / f"{base_name}.safetensors"
|
||||
pt_path = ckpt_dir / f"{base_name}.pt"
|
||||
@@ -276,7 +276,7 @@ class ChatterboxTTS:
|
||||
if safetensors_path.exists():
|
||||
return load_file(safetensors_path, device=device)
|
||||
elif pt_path.exists():
|
||||
return torch.load(pt_path, map_location=device, weights_only=True)
|
||||
return torch.load(pt_path, map_location=device)
|
||||
elif not required:
|
||||
return None
|
||||
else:
|
||||
@@ -299,7 +299,7 @@ class ChatterboxTTS:
|
||||
return load_file(safetensors_path, device=device)
|
||||
elif pt_path.exists():
|
||||
print(f"📁 Loading {base_name} from local English model: {pt_path}")
|
||||
return torch.load(pt_path, map_location=device, weights_only=True)
|
||||
return torch.load(pt_path, map_location=device)
|
||||
|
||||
# Download English model if not available locally
|
||||
print(f"📦 Downloading English model components for incomplete language model...")
|
||||
@@ -315,7 +315,7 @@ class ChatterboxTTS:
|
||||
return load_file(safetensors_path, device=device)
|
||||
elif pt_path.exists():
|
||||
print(f"📁 Loading {base_name} from downloaded English model: {pt_path}")
|
||||
return torch.load(pt_path, map_location=device, weights_only=True)
|
||||
return torch.load(pt_path, map_location=device)
|
||||
|
||||
raise FileNotFoundError(f"Could not load {base_name} from English fallback")
|
||||
|
||||
@@ -452,7 +452,7 @@ class ChatterboxTTS:
|
||||
warnings.simplefilter("ignore")
|
||||
|
||||
print(f"📦 Loading checkpoint: {unified_model_path.name}")
|
||||
checkpoint = torch.load(unified_model_path, map_location=device, weights_only=True)
|
||||
checkpoint = torch.load(unified_model_path, map_location=device)
|
||||
|
||||
# Extract model configuration
|
||||
model_config = checkpoint.get('model_config', {})
|
||||
|
||||
@@ -98,7 +98,7 @@ class ChatterboxVC:
|
||||
if safetensors_path.exists():
|
||||
return load_file(safetensors_path, device=device)
|
||||
elif pt_path.exists():
|
||||
return torch.load(pt_path, map_location=device, weights_only=True)
|
||||
return torch.load(pt_path, map_location=device)
|
||||
else:
|
||||
raise FileNotFoundError(f"Neither {base_name}.safetensors nor {base_name}.pt found in {ckpt_dir}")
|
||||
|
||||
@@ -107,7 +107,7 @@ class ChatterboxVC:
|
||||
|
||||
ref_dict = None
|
||||
if (builtin_voice := ckpt_dir / "conds.pt").exists():
|
||||
states = torch.load(builtin_voice, map_location=device, weights_only=True)
|
||||
states = torch.load(builtin_voice, map_location=device)
|
||||
ref_dict = states['gen']
|
||||
|
||||
s3gen = S3Gen()
|
||||
|
||||
@@ -1,5 +1,3 @@
|
||||
import logging
|
||||
|
||||
from .s3tokenizer import (
|
||||
S3_SR,
|
||||
S3_HOP,
|
||||
@@ -29,20 +27,4 @@ def drop_invalid_tokens(x):
|
||||
e = None
|
||||
|
||||
x = x[s: e]
|
||||
|
||||
# T3's speech head spans `speech_tokens_dict_size` (8194) while S3Gen's
|
||||
# input embedding only has SPEECH_VOCAB_SIZE (6561) rows, so a sampled id
|
||||
# above the vocab that is not exactly SoS/EoS reaches the embedding gather
|
||||
# and trips a device-side "index out of bounds" assert (which surfaces
|
||||
# later, at the next CUDA sync, with a misleading traceback).
|
||||
invalid = x >= SPEECH_VOCAB_SIZE
|
||||
if invalid.any():
|
||||
logging.warning(
|
||||
"Dropping %d out-of-vocabulary speech token(s) (max id %d, vocab %d)",
|
||||
int(invalid.sum()), int(x.max()), SPEECH_VOCAB_SIZE,
|
||||
)
|
||||
x = x[~invalid]
|
||||
# TTS Audio Suite patch: do not pass an empty speech stream to S3Gen.
|
||||
if x.numel() == 0:
|
||||
raise RuntimeError("ChatterBox generated no valid speech tokens. Try another seed or text.")
|
||||
return x
|
||||
|
||||
+2
-13
@@ -60,10 +60,6 @@ class AlignmentStreamAnalyzer:
|
||||
# using it for all layers slows things down too much. We can apply it to just one layer
|
||||
# by intercepting the kwargs and adding a forward hook (credit: jrm)
|
||||
self.last_aligned_attns = []
|
||||
# TTS Audio Suite patch: retain hook handles so each inference can remove its
|
||||
# analyzer hooks instead of accumulating three more on the shared model.
|
||||
self._hook_handles = []
|
||||
self._original_output_attentions = getattr(getattr(tfmr, "config", None), "output_attentions", None)
|
||||
for i, (layer_idx, head_idx) in enumerate(LLAMA_ALIGNED_HEADS):
|
||||
self.last_aligned_attns += [None]
|
||||
self._add_attention_spy(tfmr, i, layer_idx, head_idx)
|
||||
@@ -85,18 +81,11 @@ class AlignmentStreamAnalyzer:
|
||||
|
||||
target_layer = tfmr.layers[layer_idx].self_attn
|
||||
# Register hook and store the handle
|
||||
self._hook_handles.append(target_layer.register_forward_hook(attention_forward_hook))
|
||||
target_layer.register_forward_hook(attention_forward_hook)
|
||||
if hasattr(tfmr, 'config') and hasattr(tfmr.config, 'output_attentions'):
|
||||
self.original_output_attentions = tfmr.config.output_attentions
|
||||
tfmr.config.output_attentions = True
|
||||
|
||||
def close(self, tfmr):
|
||||
"""Remove inference-scoped hooks and restore the transformer setting."""
|
||||
for hook_handle in self._hook_handles:
|
||||
hook_handle.remove()
|
||||
self._hook_handles.clear()
|
||||
if self._original_output_attentions is not None:
|
||||
tfmr.config.output_attentions = self._original_output_attentions
|
||||
|
||||
def step(self, logits, next_token=None):
|
||||
"""
|
||||
Emits an AlignmentAnalysisResult into the output queue, and potentially modifies the logits to force an EOS.
|
||||
|
||||
@@ -268,12 +268,6 @@ class T3(nn.Module):
|
||||
# TODO? synchronize the expensive compile function
|
||||
# with self.compile_lock:
|
||||
if not self.compiled:
|
||||
# TTS Audio Suite patch: clean up hooks left by an interrupted prior inference.
|
||||
previous_backend = getattr(self, "patched_model", None)
|
||||
previous_analyzer = getattr(previous_backend, "alignment_stream_analyzer", None)
|
||||
if previous_analyzer is not None:
|
||||
previous_analyzer.close(self.tfmr)
|
||||
|
||||
# Default to None for English models, only create for multilingual
|
||||
alignment_stream_analyzer = None
|
||||
if self.hp.is_multilingual and use_alignment_analyzer:
|
||||
@@ -408,7 +402,4 @@ class T3(nn.Module):
|
||||
|
||||
# Concatenate all predicted tokens along the sequence dimension.
|
||||
predicted_tokens = torch.cat(predicted, dim=1) # shape: (B, num_tokens)
|
||||
if self.patched_model.alignment_stream_analyzer is not None:
|
||||
self.patched_model.alignment_stream_analyzer.close(self.tfmr)
|
||||
self.patched_model.alignment_stream_analyzer = None
|
||||
return predicted_tokens
|
||||
|
||||
@@ -432,7 +432,7 @@ class ChatterboxOfficial23LangTTS:
|
||||
warnings.simplefilter("ignore")
|
||||
|
||||
print(f"📦 Loading checkpoint: {unified_model_path.name}")
|
||||
checkpoint = torch.load(unified_model_path, map_location=actual_device, weights_only=True)
|
||||
checkpoint = torch.load(unified_model_path, map_location=actual_device)
|
||||
|
||||
# Extract model configuration
|
||||
model_config = checkpoint.get('model_config', {})
|
||||
|
||||
@@ -114,7 +114,7 @@ class ChatterboxVC:
|
||||
if safetensors_path.exists():
|
||||
return load_file(safetensors_path, device=device)
|
||||
elif pt_path.exists():
|
||||
return torch.load(pt_path, map_location=device, weights_only=True)
|
||||
return torch.load(pt_path, map_location=device)
|
||||
else:
|
||||
raise FileNotFoundError(f"Neither {base_name}.safetensors nor {base_name}.pt found in {ckpt_dir}")
|
||||
|
||||
@@ -123,7 +123,7 @@ class ChatterboxVC:
|
||||
|
||||
ref_dict = None
|
||||
if (builtin_voice := ckpt_dir / "conds.pt").exists():
|
||||
states = torch.load(builtin_voice, weights_only=True)
|
||||
states = torch.load(builtin_voice)
|
||||
ref_dict = states['gen']
|
||||
|
||||
s3gen = S3Gen()
|
||||
|
||||
@@ -73,8 +73,7 @@ def main():
|
||||
assert num == len(path_list)
|
||||
for path in path_list:
|
||||
print('Processing {}'.format(path))
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
states = torch.load(path, map_location=torch.device('cpu'), weights_only=True)
|
||||
states = torch.load(path, map_location=torch.device('cpu'))
|
||||
for k in states.keys():
|
||||
if k not in ['step', 'epoch']:
|
||||
if k not in avg.keys():
|
||||
|
||||
@@ -134,8 +134,7 @@ def main():
|
||||
start_step, start_epoch = 0, -1
|
||||
if args.checkpoint is not None:
|
||||
if os.path.exists(args.checkpoint):
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
state_dict = torch.load(args.checkpoint, map_location='cpu', weights_only=True)
|
||||
state_dict = torch.load(args.checkpoint, map_location='cpu')
|
||||
model.load_state_dict(state_dict, strict=False)
|
||||
if 'step' in state_dict:
|
||||
start_step = state_dict['step']
|
||||
@@ -162,8 +161,7 @@ def main():
|
||||
# DPO related
|
||||
if args.dpo is True:
|
||||
ref_model = deepcopy(configs[args.model])
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
state_dict = torch.load(args.ref_model, map_location='cpu', weights_only=True)
|
||||
state_dict = torch.load(args.ref_model, map_location='cpu')
|
||||
ref_model.load_state_dict(state_dict, strict=False)
|
||||
dpo_loss = DPOLoss(beta=0.01, label_smoothing=0.0, ipo=False)
|
||||
# NOTE maybe it is not needed to wrap ref_model as ddp because its parameter is not updated
|
||||
|
||||
@@ -47,8 +47,7 @@ class CosyVoiceFrontEnd:
|
||||
providers=["CUDAExecutionProvider" if torch.cuda.is_available() else
|
||||
"CPUExecutionProvider"])
|
||||
if os.path.exists(spk2info):
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
self.spk2info = torch.load(spk2info, map_location=self.device, weights_only=True)
|
||||
self.spk2info = torch.load(spk2info, map_location=self.device)
|
||||
else:
|
||||
self.spk2info = {}
|
||||
self.allowed_special = allowed_special
|
||||
|
||||
@@ -64,8 +64,7 @@ class CosyVoiceModel:
|
||||
self.hift_cache_dict = {}
|
||||
|
||||
def load(self, llm_model, flow_model, hift_model):
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
self.llm.load_state_dict(torch.load(llm_model, map_location=self.device, weights_only=True), strict=True)
|
||||
self.llm.load_state_dict(torch.load(llm_model, map_location=self.device), strict=True)
|
||||
self.llm.to(self.device).eval()
|
||||
# FIX: Bundled code fix - Ensure all nested modules are moved to device
|
||||
# Issue: Some nested layers (embed_tokens, etc.) weren't being moved with .to(device)
|
||||
@@ -73,16 +72,14 @@ class CosyVoiceModel:
|
||||
for module in self.llm.modules():
|
||||
module.to(self.device)
|
||||
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
self.flow.load_state_dict(torch.load(flow_model, map_location=self.device, weights_only=True), strict=True)
|
||||
self.flow.load_state_dict(torch.load(flow_model, map_location=self.device), strict=True)
|
||||
self.flow.to(self.device).eval()
|
||||
# FIX: Bundled code fix - Ensure all nested modules are moved to device
|
||||
for module in self.flow.modules():
|
||||
module.to(self.device)
|
||||
|
||||
# in case hift_model is a hifigan model
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
hift_state_dict = {k.replace('generator.', ''): v for k, v in torch.load(hift_model, map_location=self.device, weights_only=True).items()}
|
||||
hift_state_dict = {k.replace('generator.', ''): v for k, v in torch.load(hift_model, map_location=self.device).items()}
|
||||
self.hift.load_state_dict(hift_state_dict, strict=True)
|
||||
self.hift.to(self.device).eval()
|
||||
# FIX: Bundled code fix - Ensure all nested modules are moved to device
|
||||
|
||||
@@ -84,8 +84,7 @@ def assert_required_models_available(args):
|
||||
def load_hifigan(checkpoint_path, device):
|
||||
h = AttrDict(v1)
|
||||
hifigan = HiFiGAN(h).to(device)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
hifigan.load_state_dict(torch.load(checkpoint_path, map_location=device, weights_only=True)["generator"])
|
||||
hifigan.load_state_dict(torch.load(checkpoint_path, map_location=device)["generator"])
|
||||
_ = hifigan.eval()
|
||||
hifigan.remove_weight_norm()
|
||||
return hifigan
|
||||
|
||||
@@ -41,8 +41,7 @@ def get_padding(kernel_size, dilation=1):
|
||||
def load_checkpoint(filepath, device):
|
||||
assert os.path.isfile(filepath)
|
||||
print(f"Loading '{filepath}'")
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint_dict = torch.load(filepath, map_location=device, weights_only=True)
|
||||
checkpoint_dict = torch.load(filepath, map_location=device)
|
||||
print("Complete.")
|
||||
return checkpoint_dict
|
||||
|
||||
|
||||
@@ -15,8 +15,6 @@ from utils.hf_download_logging import quiet_hf_download_logs
|
||||
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
|
||||
|
||||
|
||||
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
|
||||
|
||||
class DotsTTSDownloader:
|
||||
"""Resolve and download official dots.tts model folders."""
|
||||
|
||||
@@ -61,7 +59,6 @@ class DotsTTSDownloader:
|
||||
self.base_path = base_path
|
||||
os.makedirs(self.base_path, exist_ok=True)
|
||||
|
||||
@validate_model_paths
|
||||
def resolve_model_path(self, model_identifier: str) -> str:
|
||||
"""Resolve local: names, absolute paths, or known official model identifiers."""
|
||||
if not model_identifier:
|
||||
@@ -115,7 +112,7 @@ class DotsTTSDownloader:
|
||||
with quiet_hf_download_logs():
|
||||
snapshot_download(
|
||||
repo_id=repo_id,
|
||||
local_dir=allowed_path(model_dir),
|
||||
local_dir=model_dir,
|
||||
local_dir_use_symlinks=False,
|
||||
resume_download=True,
|
||||
force_download=force,
|
||||
|
||||
@@ -4,7 +4,6 @@ Dots TTS engine wrapper.
|
||||
Wraps the official dots.tts runtime with ComfyUI-friendly model lifecycle hooks.
|
||||
"""
|
||||
|
||||
from utils.security.data_loading import RestrictedLoadModule, load_dependency_checkpoint
|
||||
import os
|
||||
import sys
|
||||
import types
|
||||
@@ -109,9 +108,6 @@ class DotsTTSEngine:
|
||||
try:
|
||||
with self._text_normalizer_compat():
|
||||
from dots_tts.runtime import DotsTtsRuntime
|
||||
from dots_tts.models.dots_tts import core
|
||||
if not isinstance(core.torch, RestrictedLoadModule):
|
||||
core.torch = RestrictedLoadModule(core.torch, load_dependency_checkpoint)
|
||||
except Exception as e:
|
||||
for module_name, module in stale_modules:
|
||||
sys.modules.setdefault(module_name, module)
|
||||
|
||||
@@ -9,8 +9,6 @@ from utils.downloads.unified_downloader import unified_downloader
|
||||
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
|
||||
|
||||
|
||||
from utils.security.path_access import validate_model_paths, allowed_path
|
||||
|
||||
class DramaBoxDownloader:
|
||||
"""Resolve DramaBox checkpoints without using Hugging Face cache storage."""
|
||||
|
||||
@@ -67,7 +65,6 @@ class DramaBoxDownloader:
|
||||
models.insert(0, local_name)
|
||||
return models
|
||||
|
||||
@validate_model_paths
|
||||
def resolve_model_path(self, model_identifier: str = MODEL_NAME) -> Dict[str, str]:
|
||||
model_identifier = model_identifier or self.MODEL_NAME
|
||||
if os.path.isabs(model_identifier) and os.path.isdir(model_identifier):
|
||||
|
||||
@@ -8,7 +8,6 @@ also valid for training.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from utils.security.path_access import resolve_input_path, allowed_path, child_path
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
@@ -43,10 +42,17 @@ def get_dramabox_training_root() -> str:
|
||||
|
||||
|
||||
def _resolve_source_path(value: str) -> Path:
|
||||
path = Path(resolve_input_path(value, datasets=True, models=False))
|
||||
if not (path.is_file()):
|
||||
raise FileNotFoundError(f"Dataset source not found: {value}")
|
||||
return path
|
||||
raw = os.path.expanduser(str(value or "").strip())
|
||||
if not raw:
|
||||
raise ValueError("dataset_source is required")
|
||||
|
||||
candidates = [Path(raw)]
|
||||
input_root = Path(folder_paths.get_input_directory())
|
||||
candidates.extend((input_root / raw, input_root / "datasets" / raw))
|
||||
for candidate in candidates:
|
||||
if candidate.is_file():
|
||||
return candidate.resolve()
|
||||
raise FileNotFoundError(f"DramaBox dataset source not found: {value}")
|
||||
|
||||
|
||||
def _resolve_audio_path(raw_path: Any, *, source_path: Path, audio_dir: str) -> Path:
|
||||
@@ -65,7 +71,7 @@ def _resolve_audio_path(raw_path: Any, *, source_path: Path, audio_dir: str) ->
|
||||
|
||||
for candidate in candidates:
|
||||
if candidate.is_file():
|
||||
return Path(allowed_path(candidate, models=False))
|
||||
return candidate.resolve()
|
||||
raise FileNotFoundError(f"DramaBox audio file not found: {raw_path}")
|
||||
|
||||
|
||||
|
||||
Vendored
+2
-4
@@ -163,15 +163,13 @@ class IDLoRADataset(Dataset):
|
||||
|
||||
def _load_sample(self, data_dir, idx):
|
||||
base = Path(data_dir)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
audio = torch.load(base / "audio_latents" / f"sample_{idx:06d}.pt", weights_only=True)
|
||||
audio = torch.load(base / "audio_latents" / f"sample_{idx:06d}.pt", weights_only=False)
|
||||
# Prefer prefix-stripped text embeddings if they exist (re-encoded with
|
||||
# just the quoted dialogue, dropping the "A woman says, " / "A man
|
||||
# speaks with X accent, " scene-description prefix).
|
||||
stripped = base / "conditions_stripped" / f"sample_{idx:06d}.pt"
|
||||
cond_path = stripped if stripped.exists() else base / "conditions" / f"sample_{idx:06d}.pt"
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
cond = torch.load(cond_path, weights_only=True)
|
||||
cond = torch.load(cond_path, weights_only=False)
|
||||
if isinstance(audio, dict):
|
||||
audio = audio.get("audio_latent", audio.get("latent", list(audio.values())[0]))
|
||||
if audio.dim() == 2:
|
||||
|
||||
@@ -184,11 +184,9 @@ class ECAPA_TDNN(nn.Module):
|
||||
torch.hub._validate_not_a_forked_repo = lambda a, b, c: True
|
||||
try:
|
||||
local_s3prl_path = os.path.expanduser("~/.cache/torch/hub/s3prl_s3prl_main")
|
||||
# TTS Audio Suite patch: restrict checkpoint deserialization.
|
||||
self.feature_extract = torch.hub.load(local_s3prl_path, feat_type, source="local", config_path=config_path, weights_only=True)
|
||||
self.feature_extract = torch.hub.load(local_s3prl_path, feat_type, source="local", config_path=config_path)
|
||||
except: # noqa: E722
|
||||
# TTS Audio Suite patch: restrict checkpoint deserialization.
|
||||
self.feature_extract = torch.hub.load("s3prl/s3prl", feat_type, weights_only=True)
|
||||
self.feature_extract = torch.hub.load("s3prl/s3prl", feat_type)
|
||||
|
||||
if len(self.feature_extract.model.encoder.layers) == 24 and hasattr(
|
||||
self.feature_extract.model.encoder.layers[23].self_attn, "fp32_attention"
|
||||
|
||||
@@ -15,8 +15,7 @@ def main():
|
||||
|
||||
device = "cuda" if torch.cuda.is_available() else "xpu" if torch.xpu.is_available() else "cpu"
|
||||
|
||||
# TTS Audio Suite patch: restrict checkpoint deserialization.
|
||||
predictor = torch.hub.load("tarepan/SpeechMOS:v1.2.0", "utmos22_strong", trust_repo=True, weights_only=True)
|
||||
predictor = torch.hub.load("tarepan/SpeechMOS:v1.2.0", "utmos22_strong", trust_repo=True)
|
||||
predictor = predictor.to(device)
|
||||
|
||||
audio_paths = list(Path(args.audio_dir).rglob(f"*.{args.ext}"))
|
||||
|
||||
@@ -203,8 +203,7 @@ def convert_timm_dit(args, mapping, dtype="float32"):
|
||||
torch_dtype = str_dtype_to_torch(dtype)
|
||||
tensor_parallel = mapping.tp_size
|
||||
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
model_params = dict(torch.load(args.timm_ckpt, weights_only=True))
|
||||
model_params = dict(torch.load(args.timm_ckpt))
|
||||
model_params = {
|
||||
k: v for k, v in model_params["ema_model_state_dict"].items() if k.startswith("ema_model.transformer")
|
||||
}
|
||||
|
||||
@@ -968,8 +968,7 @@ def expand_model_embeddings(ckpt_path, new_ckpt_path, num_new_tokens=42):
|
||||
ckpt = load_file(ckpt_path, device="cpu")
|
||||
ckpt = {"ema_model_state_dict": ckpt}
|
||||
elif ckpt_path.endswith(".pt"):
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=True)
|
||||
ckpt = torch.load(ckpt_path, map_location="cpu")
|
||||
|
||||
ema_sd = ckpt.get("ema_model_state_dict", {})
|
||||
embed_key_ema = "ema_model.transformer.text_embed.text_embed.weight"
|
||||
|
||||
@@ -8,8 +8,6 @@ import folder_paths
|
||||
from utils.downloads.unified_downloader import unified_downloader
|
||||
|
||||
|
||||
from utils.security.path_access import validate_model_paths, allowed_path
|
||||
|
||||
class FishAudioS2Downloader:
|
||||
VARIANTS = {
|
||||
"s2-pro": {
|
||||
@@ -93,7 +91,6 @@ class FishAudioS2Downloader:
|
||||
return "s2-pro"
|
||||
|
||||
@classmethod
|
||||
@validate_model_paths
|
||||
def resolve_model_path(cls, selection: str) -> str:
|
||||
"""Resolve local selections or download the selected official variant."""
|
||||
if str(selection).startswith("local:"):
|
||||
|
||||
@@ -20,8 +20,6 @@ from utils.models.extra_paths import get_preferred_download_path
|
||||
import folder_paths
|
||||
|
||||
|
||||
from utils.security.path_access import validate_model_paths, allowed_path
|
||||
|
||||
class GraniteASRDownloader:
|
||||
"""Downloader for Granite ASR models using unified download system."""
|
||||
|
||||
@@ -148,7 +146,6 @@ class GraniteASRDownloader:
|
||||
if verbose:
|
||||
print("✅ Granite model verification passed")
|
||||
|
||||
@validate_model_paths
|
||||
def resolve_model_path(self, model_identifier: str) -> str:
|
||||
if not model_identifier:
|
||||
model_identifier = "granite-speech-4.1-2b"
|
||||
|
||||
+1
-2
@@ -1,4 +1,3 @@
|
||||
from utils.security.data_loading import load_numpy_data # TTS Audio Suite patch: NumPy dictionaries must not execute pickle globals.
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
@@ -46,7 +45,7 @@ class DACFile:
|
||||
|
||||
@classmethod
|
||||
def load(cls, path):
|
||||
artifacts = load_numpy_data(path)[()]
|
||||
artifacts = np.load(path, allow_pickle=True)[()]
|
||||
codes = torch.from_numpy(artifacts["codes"].astype(int))
|
||||
if artifacts["metadata"].get("dac_version", None) not in SUPPORTED_VERSIONS:
|
||||
raise RuntimeError(f"Given file {path} can't be loaded with this version of descript-audio-codec.")
|
||||
|
||||
@@ -357,8 +357,7 @@ def load_higgs_audio_tokenizer(tokenizer_name_or_path, device="cuda"):
|
||||
**config,
|
||||
device=device,
|
||||
)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
parameter_dict = torch.load(model_path, map_location=device, weights_only=True)
|
||||
parameter_dict = torch.load(model_path, map_location=device)
|
||||
model.load_state_dict(parameter_dict, strict=False)
|
||||
model.to(device)
|
||||
model.eval()
|
||||
|
||||
@@ -12,8 +12,6 @@ from utils.hf_download_logging import quiet_hf_download_logs
|
||||
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
|
||||
|
||||
|
||||
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
|
||||
|
||||
class HiggsAudioV3Downloader:
|
||||
"""Resolve and download official Higgs Audio v3 model folders."""
|
||||
|
||||
@@ -47,7 +45,6 @@ class HiggsAudioV3Downloader:
|
||||
self.base_path = base_path
|
||||
os.makedirs(self.base_path, exist_ok=True)
|
||||
|
||||
@validate_model_paths
|
||||
def resolve_model_path(self, model_identifier: str) -> str:
|
||||
"""Resolve local paths, local: names, or known Higgs v3 model identifiers."""
|
||||
model_identifier = model_identifier or self.MODEL_NAME
|
||||
@@ -104,7 +101,7 @@ class HiggsAudioV3Downloader:
|
||||
with quiet_hf_download_logs():
|
||||
snapshot_download(
|
||||
repo_id=repo_id,
|
||||
local_dir=allowed_path(model_dir),
|
||||
local_dir=model_dir,
|
||||
local_dir_use_symlinks=False,
|
||||
resume_download=True,
|
||||
force_download=force,
|
||||
|
||||
@@ -520,8 +520,7 @@ class BigVGAN(
|
||||
local_files_only=local_files_only,
|
||||
)
|
||||
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint_dict = torch.load(model_file, map_location=map_location, weights_only=True)
|
||||
checkpoint_dict = torch.load(model_file, map_location=map_location)
|
||||
|
||||
try:
|
||||
model.load_state_dict(checkpoint_dict["generator"])
|
||||
|
||||
@@ -63,8 +63,7 @@ def get_padding(kernel_size, dilation=1):
|
||||
def load_checkpoint(filepath, device):
|
||||
assert os.path.isfile(filepath)
|
||||
print(f"Loading '{filepath}'")
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint_dict = torch.load(filepath, map_location=device, weights_only=True)
|
||||
checkpoint_dict = torch.load(filepath, map_location=device)
|
||||
print("Complete.")
|
||||
return checkpoint_dict
|
||||
|
||||
|
||||
@@ -234,8 +234,7 @@ class EnhancedCodec(nn.Module):
|
||||
def load_checkpoint(self, checkpoint_path):
|
||||
"""Load model weights from a checkpoint file."""
|
||||
assert os.path.isfile(checkpoint_path), f"Checkpoint not found: {checkpoint_path}"
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint_dict = torch.load(checkpoint_path, map_location='cpu', weights_only=True)
|
||||
checkpoint_dict = torch.load(checkpoint_path, map_location='cpu')
|
||||
saved_state_dict = checkpoint_dict['model']
|
||||
state_dict = self.state_dict()
|
||||
new_state_dict = {}
|
||||
|
||||
@@ -662,8 +662,12 @@ def load_state_dict(
|
||||
):
|
||||
extra_args = {"mmap": True}
|
||||
weights_only_kwarg = {"weights_only": weights_only} if is_torch_greater_or_equal_than_1_13 else {}
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
return torch.load(checkpoint_file, map_location=map_location, **{**weights_only_kwarg, **extra_args, "weights_only": True})
|
||||
return torch.load(
|
||||
checkpoint_file,
|
||||
map_location=map_location,
|
||||
**weights_only_kwarg,
|
||||
**extra_args,
|
||||
)
|
||||
except Exception as e:
|
||||
try:
|
||||
with open(checkpoint_file) as f:
|
||||
|
||||
@@ -111,8 +111,7 @@ class IndexTTS:
|
||||
self.use_cuda_kernel = False
|
||||
self.bigvgan = Generator(self.cfg.bigvgan, use_cuda_kernel=self.use_cuda_kernel)
|
||||
self.bigvgan_path = os.path.join(self.model_dir, self.cfg.bigvgan_checkpoint)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
vocoder_dict = torch.load(self.bigvgan_path, map_location="cpu", weights_only=True)
|
||||
vocoder_dict = torch.load(self.bigvgan_path, map_location="cpu")
|
||||
self.bigvgan.load_state_dict(vocoder_dict["generator"])
|
||||
self.bigvgan = self.bigvgan.to(self.device)
|
||||
# remove weight norm on eval mode
|
||||
|
||||
@@ -192,17 +192,6 @@ class IndexTTS2:
|
||||
self.gpt.eval()
|
||||
print(">> GPT weights restored from:", self.gpt_path)
|
||||
|
||||
if use_deepspeed:
|
||||
try:
|
||||
import transformers
|
||||
transformers_major = int(transformers.__version__.split('.', 1)[0])
|
||||
except (ImportError, ValueError, AttributeError):
|
||||
transformers_major = None
|
||||
if transformers_major is not None and transformers_major >= 5:
|
||||
use_deepspeed = False
|
||||
print("⚠️ DeepSpeed kernel injection is not supported by the bundled IndexTTS GPT2 path with Transformers 5.x")
|
||||
print(">> Falling back to standard IndexTTS-2 inference (DeepSpeed disabled)")
|
||||
|
||||
if use_deepspeed:
|
||||
try:
|
||||
import deepspeed
|
||||
@@ -210,15 +199,7 @@ class IndexTTS2:
|
||||
use_deepspeed = False
|
||||
print(f">> Failed to load DeepSpeed. Falling back to normal inference. Error: {e}")
|
||||
|
||||
try:
|
||||
self.gpt.post_init_gpt2_config(use_deepspeed=use_deepspeed, kv_cache=True, half=self.use_fp16)
|
||||
except Exception as e:
|
||||
if not use_deepspeed:
|
||||
raise
|
||||
use_deepspeed = False
|
||||
print(f"⚠️ DeepSpeed initialization failed: {e}")
|
||||
print(">> Falling back to standard IndexTTS-2 inference (DeepSpeed disabled)")
|
||||
self.gpt.post_init_gpt2_config(use_deepspeed=False, kv_cache=True, half=self.use_fp16)
|
||||
self.gpt.post_init_gpt2_config(use_deepspeed=use_deepspeed, kv_cache=True, half=self.use_fp16)
|
||||
|
||||
if self.use_cuda_kernel:
|
||||
# preload the CUDA kernel for BigVGAN
|
||||
@@ -338,8 +319,7 @@ class IndexTTS2:
|
||||
campplus_ckpt_path = os.path.join(campplus_path, "campplus_cn_common.bin")
|
||||
|
||||
campplus_model = CAMPPlus(feat_dim=80, embedding_size=192)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
campplus_model.load_state_dict(torch.load(campplus_ckpt_path, map_location="cpu", weights_only=True))
|
||||
campplus_model.load_state_dict(torch.load(campplus_ckpt_path, map_location="cpu"))
|
||||
self.campplus_model = campplus_model.to(self.load_device)
|
||||
self.campplus_model.eval()
|
||||
print(">> campplus_model weights restored from:", campplus_ckpt_path)
|
||||
@@ -365,13 +345,11 @@ class IndexTTS2:
|
||||
self.tokenizer = TextTokenizer(self.bpe_path, self.normalizer)
|
||||
print(">> bpe model loaded from:", self.bpe_path)
|
||||
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
emo_matrix = torch.load(os.path.join(self.model_dir, self.cfg.emo_matrix), weights_only=True)
|
||||
emo_matrix = torch.load(os.path.join(self.model_dir, self.cfg.emo_matrix))
|
||||
self.emo_matrix = emo_matrix.to(self.load_device)
|
||||
self.emo_num = list(self.cfg.emo_num)
|
||||
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
spk_matrix = torch.load(os.path.join(self.model_dir, self.cfg.spk_matrix), weights_only=True)
|
||||
spk_matrix = torch.load(os.path.join(self.model_dir, self.cfg.spk_matrix))
|
||||
self.spk_matrix = spk_matrix.to(self.load_device)
|
||||
|
||||
self.emo_matrix = torch.split(self.emo_matrix, self.emo_num)
|
||||
|
||||
@@ -170,17 +170,6 @@ class IndexTTS2:
|
||||
self.gpt.eval()
|
||||
print(">> GPT weights restored from:", self.gpt_path)
|
||||
|
||||
if use_deepspeed:
|
||||
try:
|
||||
import transformers
|
||||
transformers_major = int(transformers.__version__.split('.', 1)[0])
|
||||
except (ImportError, ValueError, AttributeError):
|
||||
transformers_major = None
|
||||
if transformers_major is not None and transformers_major >= 5:
|
||||
use_deepspeed = False
|
||||
print("⚠️ DeepSpeed kernel injection is not supported by the bundled IndexTTS GPT2 path with Transformers 5.x")
|
||||
print(">> Falling back to standard IndexTTS-2.5 inference (DeepSpeed disabled)")
|
||||
|
||||
if use_deepspeed:
|
||||
try:
|
||||
import deepspeed
|
||||
@@ -188,15 +177,7 @@ class IndexTTS2:
|
||||
use_deepspeed = False
|
||||
print(f">> Failed to load DeepSpeed. Falling back to normal inference. Error: {e}")
|
||||
|
||||
try:
|
||||
self.gpt.post_init_gpt2_config(use_deepspeed=use_deepspeed, kv_cache=True, half=self.use_bf16)
|
||||
except Exception as e:
|
||||
if not use_deepspeed:
|
||||
raise
|
||||
use_deepspeed = False
|
||||
print(f"⚠️ DeepSpeed initialization failed: {e}")
|
||||
print(">> Falling back to standard IndexTTS-2.5 inference (DeepSpeed disabled)")
|
||||
self.gpt.post_init_gpt2_config(use_deepspeed=False, kv_cache=True, half=self.use_bf16)
|
||||
self.gpt.post_init_gpt2_config(use_deepspeed=use_deepspeed, kv_cache=True, half=self.use_bf16)
|
||||
|
||||
if self.use_cuda_kernel:
|
||||
# preload the CUDA kernel for BigVGAN
|
||||
@@ -217,8 +198,7 @@ class IndexTTS2:
|
||||
self.semantic_model = Wav2Vec2BertModel.from_pretrained(w2v_bert_dir, local_files_only=True)
|
||||
self.semantic_model = self.semantic_model.to(self.device)
|
||||
self.semantic_model.eval()
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
stat_mean_var = torch.load(os.path.join(self.model_dir, self.cfg.w2v_stat), weights_only=True)
|
||||
stat_mean_var = torch.load(os.path.join(self.model_dir, self.cfg.w2v_stat))
|
||||
self.semantic_mean = stat_mean_var["mean"].to(self.device)
|
||||
self.semantic_std = torch.sqrt(stat_mean_var["var"]).to(self.device)
|
||||
|
||||
@@ -262,8 +242,7 @@ class IndexTTS2:
|
||||
campplus_dir = index_tts_downloader.download_model("campplus")
|
||||
campplus_ckpt_path = os.path.join(campplus_dir, "campplus_cn_common.bin")
|
||||
campplus_model = CAMPPlus(feat_dim=80, embedding_size=192)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
campplus_model.load_state_dict(torch.load(campplus_ckpt_path, map_location="cpu", weights_only=True))
|
||||
campplus_model.load_state_dict(torch.load(campplus_ckpt_path, map_location="cpu"))
|
||||
self.campplus_model = campplus_model.to(self.device)
|
||||
self.campplus_model.eval()
|
||||
print(">> campplus_model weights restored from:", campplus_ckpt_path)
|
||||
@@ -286,13 +265,11 @@ class IndexTTS2:
|
||||
self.text_process.load_glossary_from_yaml(self.glossary_path)
|
||||
print(">> Glossary loaded from:", self.glossary_path)
|
||||
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
emo_matrix = torch.load(os.path.join(self.model_dir, self.cfg.emo_matrix), weights_only=True)
|
||||
emo_matrix = torch.load(os.path.join(self.model_dir, self.cfg.emo_matrix))
|
||||
self.emo_matrix = emo_matrix.to(self.device)
|
||||
self.emo_num = list(self.cfg.emo_num)
|
||||
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
spk_matrix = torch.load(os.path.join(self.model_dir, self.cfg.spk_matrix), weights_only=True)
|
||||
spk_matrix = torch.load(os.path.join(self.model_dir, self.cfg.spk_matrix))
|
||||
self.spk_matrix = spk_matrix.to(self.device)
|
||||
|
||||
self.emo_matrix = torch.split(self.emo_matrix, self.emo_num)
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
from utils.security.data_loading import load_numpy_data # TTS Audio Suite patch: NumPy dictionaries must not execute pickle globals.
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
@@ -46,7 +45,7 @@ class DACFile:
|
||||
|
||||
@classmethod
|
||||
def load(cls, path):
|
||||
artifacts = load_numpy_data(path)[()]
|
||||
artifacts = np.load(path, allow_pickle=True)[()]
|
||||
codes = torch.from_numpy(artifacts["codes"].astype(int))
|
||||
if artifacts["metadata"].get("dac_version", None) not in SUPPORTED_VERSIONS:
|
||||
raise RuntimeError(
|
||||
|
||||
@@ -478,8 +478,7 @@ class BigVGAN(
|
||||
local_files_only=local_files_only,
|
||||
)
|
||||
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint_dict = torch.load(model_file, map_location=map_location, weights_only=True)
|
||||
checkpoint_dict = torch.load(model_file, map_location=map_location)
|
||||
|
||||
try:
|
||||
model.load_state_dict(checkpoint_dict["generator"])
|
||||
|
||||
@@ -61,8 +61,7 @@ def get_padding(kernel_size, dilation=1):
|
||||
def load_checkpoint(filepath, device):
|
||||
assert os.path.isfile(filepath)
|
||||
print(f"Loading '{filepath}'")
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint_dict = torch.load(filepath, map_location=device, weights_only=True)
|
||||
checkpoint_dict = torch.load(filepath, map_location=device)
|
||||
print("Complete.")
|
||||
return checkpoint_dict
|
||||
|
||||
|
||||
@@ -220,8 +220,7 @@ def load_F0_models(path):
|
||||
from .JDC.model import JDCNet
|
||||
|
||||
F0_model = JDCNet(num_class=1, seq_len=192)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
params = torch.load(path, map_location="cpu", weights_only=True)["net"]
|
||||
params = torch.load(path, map_location="cpu")["net"]
|
||||
F0_model.load_state_dict(params)
|
||||
_ = F0_model.train()
|
||||
|
||||
@@ -529,8 +528,7 @@ def load_checkpoint(
|
||||
is_distributed=False,
|
||||
load_ema=False,
|
||||
):
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
state = torch.load(path, map_location="cpu", weights_only=True)
|
||||
state = torch.load(path, map_location="cpu")
|
||||
params = state["net"]
|
||||
if load_ema and "ema" in state:
|
||||
print("Loading EMA")
|
||||
@@ -587,8 +585,7 @@ def load_checkpoint2(
|
||||
is_distributed=False,
|
||||
load_ema=False,
|
||||
):
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
state = torch.load(path, map_location="cpu", weights_only=True)
|
||||
state = torch.load(path, map_location="cpu")
|
||||
params = state["net"]
|
||||
if load_ema and "ema" in state:
|
||||
print("Loading EMA")
|
||||
|
||||
@@ -33,8 +33,7 @@ class OpenVoiceBaseClass(object):
|
||||
self.device = device
|
||||
|
||||
def load_ckpt(self, ckpt_path):
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint_dict = torch.load(ckpt_path, map_location=torch.device(self.device), weights_only=True)
|
||||
checkpoint_dict = torch.load(ckpt_path, map_location=torch.device(self.device))
|
||||
a, b = self.model.load_state_dict(checkpoint_dict['model'], strict=False)
|
||||
print("Loaded checkpoint '{}'".format(ckpt_path))
|
||||
print('missing/unexpected keys:', a, b)
|
||||
|
||||
@@ -27,12 +27,9 @@ tone_color_converter = ToneColorConverter(f'{ckpt_converter}/config.json', devic
|
||||
tone_color_converter.load_ckpt(f'{ckpt_converter}/checkpoint.pth')
|
||||
|
||||
# load speaker embeddings
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
en_source_default_se = torch.load(f'{en_ckpt_base}/en_default_se.pth', weights_only=True).to(device)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
en_source_style_se = torch.load(f'{en_ckpt_base}/en_style_se.pth', weights_only=True).to(device)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
zh_source_se = torch.load(f'{zh_ckpt_base}/zh_default_se.pth', weights_only=True).to(device)
|
||||
en_source_default_se = torch.load(f'{en_ckpt_base}/en_default_se.pth').to(device)
|
||||
en_source_style_se = torch.load(f'{en_ckpt_base}/en_style_se.pth').to(device)
|
||||
zh_source_se = torch.load(f'{zh_ckpt_base}/zh_default_se.pth').to(device)
|
||||
|
||||
# This online demo mainly supports English and Chinese
|
||||
supported_languages = ['zh', 'en']
|
||||
|
||||
@@ -536,8 +536,7 @@ class RMVPE:
|
||||
|
||||
def get_default_model():
|
||||
model = E2E(4, 1, (2, 2))
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
ckpt = torch.load(model_path, map_location="cpu", weights_only=True)
|
||||
ckpt = torch.load(model_path, map_location="cpu")
|
||||
model.load_state_dict(ckpt)
|
||||
model.eval()
|
||||
if is_half:
|
||||
|
||||
@@ -91,8 +91,7 @@ class Extract_wav2vectbert:
|
||||
self.semantic_model = Wav2Vec2BertModel.from_pretrained("./MaskGCT_model/w2v_bert/")
|
||||
self.semantic_model.eval()
|
||||
self.semantic_model.to(device)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
self.stat_mean_var = torch.load("./MaskGCT_model/wav2vec2bert_stats.pt", weights_only=True)
|
||||
self.stat_mean_var = torch.load("./MaskGCT_model/wav2vec2bert_stats.pt")
|
||||
self.semantic_mean = self.stat_mean_var["mean"]
|
||||
self.semantic_std = torch.sqrt(self.stat_mean_var["var"])
|
||||
self.semantic_mean = self.semantic_mean.to(device)
|
||||
|
||||
@@ -23,8 +23,7 @@ import yaml
|
||||
|
||||
|
||||
def load_checkpoint(model: torch.nn.Module, model_pth: str) -> dict:
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint = torch.load(model_pth, map_location='cpu', weights_only=True)
|
||||
checkpoint = torch.load(model_pth, map_location='cpu')
|
||||
checkpoint = checkpoint['model'] if 'model' in checkpoint else checkpoint
|
||||
model.load_state_dict(checkpoint, strict=True)
|
||||
info_path = re.sub('.pth$', '.yaml', model_pth)
|
||||
|
||||
@@ -301,7 +301,6 @@ class VocoderInference(object):
|
||||
"melgan",
|
||||
"nsfhifigan",
|
||||
]:
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
ckpt = torch.load(
|
||||
checkpoint_dir,
|
||||
map_location=(
|
||||
@@ -309,7 +308,7 @@ class VocoderInference(object):
|
||||
if torch.cuda.is_available()
|
||||
else torch.device("cpu")
|
||||
),
|
||||
weights_only=True)
|
||||
)
|
||||
if from_multi_gpu:
|
||||
pretrained_generator_dict = ckpt["generator_state_dict"]
|
||||
generator_dict = self.model.state_dict()
|
||||
@@ -329,8 +328,7 @@ class VocoderInference(object):
|
||||
else:
|
||||
self.model.load_state_dict(ckpt["generator_state_dict"])
|
||||
else:
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
self.model.load_state_dict(torch.load(checkpoint_dir, weights_only=True)["state_dict"])
|
||||
self.model.load_state_dict(torch.load(checkpoint_dir)["state_dict"])
|
||||
return str(checkpoint_dir)
|
||||
|
||||
def inference(self):
|
||||
@@ -414,7 +412,6 @@ def load_nnvocoder(
|
||||
if not os.path.isdir(weights_file):
|
||||
# Load from .pt file
|
||||
if vocoder_name in ["bigvgan", "hifigan", "melgan", "nsfhifigan"]:
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
ckpt = torch.load(
|
||||
weights_file,
|
||||
map_location=(
|
||||
@@ -422,7 +419,7 @@ def load_nnvocoder(
|
||||
if torch.cuda.is_available()
|
||||
else torch.device("cpu")
|
||||
),
|
||||
weights_only=True)
|
||||
)
|
||||
if from_multi_gpu:
|
||||
pretrained_generator_dict = ckpt["generator_state_dict"]
|
||||
generator_dict = model.state_dict()
|
||||
@@ -442,8 +439,7 @@ def load_nnvocoder(
|
||||
else:
|
||||
model.load_state_dict(ckpt["generator_state_dict"])
|
||||
else:
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
model.load_state_dict(torch.load(weights_file, weights_only=True)["state_dict"])
|
||||
model.load_state_dict(torch.load(weights_file)["state_dict"])
|
||||
else:
|
||||
# Load from accelerator state dict
|
||||
weights_file = os.path.join(weights_file, "checkpoint")
|
||||
|
||||
@@ -34,8 +34,7 @@ class FAcodecInference(object):
|
||||
return model
|
||||
|
||||
def _load_checkpoint(self):
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
sd = torch.load(self.args.checkpoint_path, map_location="cpu", weights_only=True)
|
||||
sd = torch.load(self.args.checkpoint_path, map_location="cpu")
|
||||
sd = sd["net"] if "net" in sd else sd
|
||||
new_params = dict()
|
||||
for key, state_dict in sd.items():
|
||||
|
||||
@@ -208,8 +208,7 @@ def load_F0_models(path):
|
||||
F0_model = JDCNet(num_class=1, seq_len=192)
|
||||
if not os.path.exists(path):
|
||||
path = hf_hub_download(repo_id="Plachta/JDCnet", filename="bst.t7")
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
params = torch.load(path, map_location="cpu", weights_only=True)["net"]
|
||||
params = torch.load(path, map_location="cpu")["net"]
|
||||
F0_model.load_state_dict(params)
|
||||
_ = F0_model.train()
|
||||
|
||||
@@ -296,8 +295,7 @@ def load_checkpoint(
|
||||
ignore_modules=[],
|
||||
is_distributed=False,
|
||||
):
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
state = torch.load(path, map_location="cpu", weights_only=True)
|
||||
state = torch.load(path, map_location="cpu")
|
||||
params = state["net"]
|
||||
for key in model:
|
||||
if key in params and key not in ignore_modules:
|
||||
|
||||
@@ -81,8 +81,7 @@ class SpeechTokenizer(nn.Module):
|
||||
with open(config_path) as f:
|
||||
cfg = json.load(f)
|
||||
model = cls(cfg)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
params = torch.load(ckpt_path, map_location="cpu", weights_only=True)
|
||||
params = torch.load(ckpt_path, map_location="cpu")
|
||||
model.load_state_dict(params)
|
||||
return model
|
||||
|
||||
|
||||
@@ -132,8 +132,7 @@ def build_semantic_model(path_='./models/tts/maskgct/ckpt/wav2vec2bert_stats.pt'
|
||||
raise RuntimeError(f"Failed to load w2v-bert-2.0 model: {e}")
|
||||
|
||||
semantic_model.eval()
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
stat_mean_var = torch.load(path_, weights_only=True)
|
||||
stat_mean_var = torch.load(path_)
|
||||
semantic_mean = stat_mean_var["mean"]
|
||||
semantic_std = torch.sqrt(stat_mean_var["var"])
|
||||
return semantic_model, semantic_mean, semantic_std
|
||||
|
||||
@@ -43,8 +43,7 @@ def dvae_wav_to_mel(
|
||||
mel = mel_stft(wav)
|
||||
mel = torch.log(torch.clamp(mel, min=1e-5))
|
||||
if mel_norms is None:
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
mel_norms = torch.load(mel_norms_file, map_location=device, weights_only=True)
|
||||
mel_norms = torch.load(mel_norms_file, map_location=device)
|
||||
mel = mel / mel_norms.unsqueeze(0).unsqueeze(-1)
|
||||
return mel
|
||||
|
||||
|
||||
@@ -9,8 +9,6 @@ from utils.hf_download_logging import quiet_hf_download_logs
|
||||
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
|
||||
|
||||
|
||||
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
|
||||
|
||||
class MossSoundEffectV2Downloader:
|
||||
MODEL_NAME = "MOSS-SoundEffect-v2.0"
|
||||
REPO_ID = "OpenMOSS-Team/MOSS-SoundEffect-v2.0"
|
||||
@@ -47,7 +45,6 @@ class MossSoundEffectV2Downloader:
|
||||
os.path.isfile(os.path.join(model_dir, rel_path)) for rel_path in cls.REQUIRED_FILES
|
||||
)
|
||||
|
||||
@validate_model_paths
|
||||
def resolve_model_path(self, identifier: str = MODEL_NAME) -> str:
|
||||
identifier = identifier or self.MODEL_NAME
|
||||
if os.path.isabs(identifier):
|
||||
@@ -82,7 +79,7 @@ class MossSoundEffectV2Downloader:
|
||||
with quiet_hf_download_logs():
|
||||
snapshot_download(
|
||||
repo_id=self.REPO_ID,
|
||||
local_dir=allowed_path(model_dir),
|
||||
local_dir=model_dir,
|
||||
force_download=force,
|
||||
)
|
||||
except Exception as exc:
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
from utils.security.data_loading import load_numpy_data # TTS Audio Suite patch: NumPy dictionaries must not execute pickle globals.
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
@@ -49,7 +48,7 @@ class DACFile:
|
||||
|
||||
@classmethod
|
||||
def load(cls, path):
|
||||
artifacts = load_numpy_data(path)[()]
|
||||
artifacts = np.load(path, allow_pickle=True)[()]
|
||||
codes = torch.from_numpy(artifacts["codes"].astype(int))
|
||||
if artifacts["metadata"].get("dac_version", None) not in SUPPORTED_VERSIONS:
|
||||
raise RuntimeError(
|
||||
|
||||
@@ -14,8 +14,6 @@ from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_down
|
||||
from engines.moss_tts.model_specs import MOSS_CODEC_SPECS, MOSS_MODEL_SPECS
|
||||
|
||||
|
||||
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
|
||||
|
||||
class MossTTSDownloader:
|
||||
"""Resolve and download official MOSS-TTS model folders."""
|
||||
|
||||
@@ -35,7 +33,6 @@ class MossTTSDownloader:
|
||||
self.base_path = base_path
|
||||
os.makedirs(self.base_path, exist_ok=True)
|
||||
|
||||
@validate_model_paths
|
||||
def resolve_model_path(self, model_identifier: str) -> str:
|
||||
"""Resolve local: names, absolute paths, or known MOSS model identifiers."""
|
||||
if not model_identifier:
|
||||
@@ -95,7 +92,7 @@ class MossTTSDownloader:
|
||||
with quiet_hf_download_logs():
|
||||
snapshot_download(
|
||||
repo_id=repo_id,
|
||||
local_dir=allowed_path(model_dir),
|
||||
local_dir=model_dir,
|
||||
force_download=force,
|
||||
)
|
||||
except Exception as e:
|
||||
|
||||
@@ -4,8 +4,6 @@ Shared helpers for MOSS-TTS training.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from utils.security.path_access import allowed_path
|
||||
from utils.security.path_access import resolve_input_path, allowed_path, child_path
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
@@ -44,22 +42,8 @@ def slugify(value: str) -> str:
|
||||
|
||||
|
||||
def load_jsonl(path: str | Path) -> List[Dict[str, Any]]:
|
||||
path = Path(allowed_path(path, models=False))
|
||||
with open(path, "r", encoding="utf-8") as handle:
|
||||
records = [json.loads(line) for line in handle if line.strip()]
|
||||
for record in records:
|
||||
for field in ("audio", "ref_audio", "reference_audio", "reference"):
|
||||
value = record.get(field)
|
||||
def resolve(value):
|
||||
if value is None:
|
||||
return None
|
||||
raw = Path(value)
|
||||
return allowed_path(raw if raw.is_absolute() else path.parent / raw, models=False)
|
||||
if isinstance(value, str):
|
||||
record[field] = resolve(value)
|
||||
elif isinstance(value, list) and all(item is None or isinstance(item, str) for item in value):
|
||||
record[field] = [resolve(item) for item in value]
|
||||
return records
|
||||
return [json.loads(line) for line in handle if line.strip()]
|
||||
|
||||
|
||||
def dump_jsonl(records: Iterable[Dict[str, Any]], path: str | Path) -> None:
|
||||
@@ -70,18 +54,34 @@ def dump_jsonl(records: Iterable[Dict[str, Any]], path: str | Path) -> None:
|
||||
handle.write(json.dumps(record, ensure_ascii=False) + "\n")
|
||||
|
||||
|
||||
def resolve_manifest_path(dataset_source: str) -> Path:
|
||||
path = Path(resolve_input_path(dataset_source, datasets=True, models=False))
|
||||
if not (path.is_file()):
|
||||
raise FileNotFoundError(f"Dataset source not found: {dataset_source}")
|
||||
return path
|
||||
def resolve_manifest_path(dataset_source: str) -> str:
|
||||
raw = os.path.expanduser(str(dataset_source or "").strip())
|
||||
if not raw:
|
||||
raise ValueError("dataset_source is required")
|
||||
|
||||
candidates = [
|
||||
raw,
|
||||
os.path.join(folder_paths.get_input_directory(), raw),
|
||||
os.path.join(folder_paths.get_input_directory(), "datasets", raw),
|
||||
]
|
||||
for candidate in candidates:
|
||||
if os.path.isfile(candidate):
|
||||
return os.path.abspath(candidate)
|
||||
|
||||
raise FileNotFoundError(f"MOSS training manifest not found: {dataset_source}")
|
||||
|
||||
|
||||
def _resolve_dataset_source_path(dataset_source: str) -> Path:
|
||||
path = Path(resolve_input_path(dataset_source, datasets=True, models=False))
|
||||
if not (path.is_file() or path.is_dir()):
|
||||
raise FileNotFoundError(f"Dataset source not found: {dataset_source}")
|
||||
return path
|
||||
raw = os.path.expanduser(str(dataset_source or "").strip())
|
||||
if not raw:
|
||||
raise ValueError("dataset_source is required")
|
||||
|
||||
input_dir = folder_paths.get_input_directory()
|
||||
candidates = [Path(raw), Path(input_dir, raw), Path(input_dir, "datasets", raw)]
|
||||
for candidate in candidates:
|
||||
if candidate.is_file() or candidate.is_dir():
|
||||
return candidate.resolve()
|
||||
raise FileNotFoundError(f"MOSS dataset source not found: {dataset_source}")
|
||||
|
||||
|
||||
def _build_manifest_from_audio_folder(dataset_dir: Path, recursive: bool) -> str:
|
||||
@@ -102,8 +102,6 @@ def _build_manifest_from_audio_folder(dataset_dir: Path, recursive: bool) -> str
|
||||
if not transcript_path.is_file():
|
||||
missing_transcripts.append(str(audio_path.relative_to(dataset_dir)))
|
||||
continue
|
||||
audio_path = Path(allowed_path(audio_path, models=False))
|
||||
transcript_path = Path(allowed_path(transcript_path, models=False))
|
||||
transcript = transcript_path.read_text(encoding="utf-8-sig").strip()
|
||||
if not transcript:
|
||||
missing_transcripts.append(str(audio_path.relative_to(dataset_dir)))
|
||||
@@ -243,7 +241,7 @@ def resolve_continue_from_adapter_path(continue_from: Any) -> str:
|
||||
if not value:
|
||||
return ""
|
||||
if os.path.isdir(value):
|
||||
return allowed_path(value)
|
||||
return value
|
||||
raise FileNotFoundError(f"MOSS continue_from adapter path not found: {value}")
|
||||
|
||||
if isinstance(continue_from, dict):
|
||||
@@ -253,12 +251,12 @@ def resolve_continue_from_adapter_path(continue_from: Any) -> str:
|
||||
raise ValueError("continue_from TRAINING_ARTIFACTS must come from a MOSS training run")
|
||||
adapter_path = str(continue_from.get("model_path", "") or "").strip()
|
||||
if adapter_path and os.path.isdir(adapter_path):
|
||||
return allowed_path(adapter_path)
|
||||
return adapter_path
|
||||
lora_info = continue_from.get("lora_adapter")
|
||||
if isinstance(lora_info, dict):
|
||||
adapter_path = str(lora_info.get("adapter_path", "") or "").strip()
|
||||
if adapter_path and os.path.isdir(adapter_path):
|
||||
return allowed_path(adapter_path)
|
||||
return adapter_path
|
||||
raise FileNotFoundError("continue_from TRAINING_ARTIFACTS does not contain a valid MOSS adapter path")
|
||||
|
||||
raise ValueError(
|
||||
|
||||
@@ -14,8 +14,6 @@ from utils.hf_download_logging import quiet_hf_download_logs
|
||||
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
|
||||
|
||||
|
||||
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
|
||||
|
||||
class OmniVoiceDownloader:
|
||||
"""Resolve and download official OmniVoice model folders."""
|
||||
|
||||
@@ -46,7 +44,6 @@ class OmniVoiceDownloader:
|
||||
self.base_path = base_path
|
||||
os.makedirs(self.base_path, exist_ok=True)
|
||||
|
||||
@validate_model_paths
|
||||
def resolve_model_path(self, model_identifier: str) -> str:
|
||||
"""Resolve absolute paths, local: names, or known official identifiers."""
|
||||
if not model_identifier:
|
||||
@@ -103,7 +100,7 @@ class OmniVoiceDownloader:
|
||||
with quiet_hf_download_logs():
|
||||
snapshot_download(
|
||||
repo_id=repo_id,
|
||||
local_dir=allowed_path(model_dir),
|
||||
local_dir=model_dir,
|
||||
local_dir_use_symlinks=False,
|
||||
resume_download=True,
|
||||
force_download=force,
|
||||
|
||||
@@ -109,8 +109,7 @@ def log_mel_spectrogram(
|
||||
|
||||
|
||||
def get_T_after_cnn(L_in, dilation=1):
|
||||
# TTS Audio Suite patch: this architecture list is constant data.
|
||||
for (padding, kernel_size, stride) in [(1, 3, 1), (1, 3, 2)]:
|
||||
for (padding, kernel_size, stride) in eval("[(1,3,1)] + [(1,3,2)] "):
|
||||
L_out = L_in + 2 * padding - dilation * (kernel_size - 1) - 1
|
||||
L_out = 1 + L_out // stride
|
||||
L_in = L_out
|
||||
|
||||
@@ -21,8 +21,6 @@ from utils.hf_download_logging import quiet_hf_download_logs
|
||||
import folder_paths
|
||||
|
||||
|
||||
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
|
||||
|
||||
class Qwen3ASRDownloader:
|
||||
"""Downloader for Qwen3-ASR models using Hugging Face snapshot downloads."""
|
||||
|
||||
@@ -87,7 +85,7 @@ class Qwen3ASRDownloader:
|
||||
with quiet_hf_download_logs():
|
||||
snapshot_download(
|
||||
repo_id=repo_id,
|
||||
local_dir=allowed_path(model_dir),
|
||||
local_dir=model_dir,
|
||||
local_dir_use_symlinks=False,
|
||||
resume_download=True
|
||||
)
|
||||
@@ -100,7 +98,6 @@ class Qwen3ASRDownloader:
|
||||
print(f"\n❌ Download failed: {e}")
|
||||
raise
|
||||
|
||||
@validate_model_paths
|
||||
def resolve_model_path(self, model_identifier: str) -> str:
|
||||
if not model_identifier:
|
||||
model_identifier = "Qwen3-ASR-1.7B"
|
||||
|
||||
@@ -22,8 +22,6 @@ from utils.hf_download_logging import quiet_hf_download_logs
|
||||
from utils.models.extra_paths import get_preferred_download_path, get_all_tts_model_paths
|
||||
import folder_paths
|
||||
|
||||
|
||||
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
|
||||
|
||||
class Qwen3TTSDownloader:
|
||||
"""Downloader for Qwen3-TTS models using unified download system."""
|
||||
@@ -225,7 +223,7 @@ class Qwen3TTSDownloader:
|
||||
with quiet_hf_download_logs():
|
||||
snapshot_download(
|
||||
repo_id=repo_id,
|
||||
local_dir=allowed_path(model_dir),
|
||||
local_dir=model_dir,
|
||||
local_dir_use_symlinks=False,
|
||||
resume_download=True
|
||||
)
|
||||
@@ -280,7 +278,6 @@ class Qwen3TTSDownloader:
|
||||
|
||||
print(f"\n✅ Essential models ready!")
|
||||
|
||||
@validate_model_paths
|
||||
def resolve_model_path(self, model_identifier: str) -> str:
|
||||
"""
|
||||
Resolve model path handling "local:" prefix and auto-download.
|
||||
|
||||
@@ -305,7 +305,7 @@ def _convert_bin_to_safetensors(bin_path: str, safetensors_path: str) -> bool:
|
||||
|
||||
# Load the .bin file
|
||||
print(f"📂 Loading {os.path.basename(bin_path)}...")
|
||||
state_dict = torch.load(bin_path, map_location="cpu", weights_only=True)
|
||||
state_dict = torch.load(bin_path, map_location="cpu")
|
||||
|
||||
# Save as .safetensors
|
||||
print(f"💾 Saving as {os.path.basename(safetensors_path)}...")
|
||||
|
||||
@@ -567,8 +567,7 @@ class MusicSeparationModel:
|
||||
if model['Stem'] == "BOTH":
|
||||
device = torch.device(self.GPU_device)
|
||||
mdx23 = tfc_tdf.TFC_TDF_net(self.MDX23_config)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
mdx23.load_state_dict( torch.load(model['PATH'], map_location = device, weights_only=True) )
|
||||
mdx23.load_state_dict( torch.load(model['PATH'], map_location = device) )
|
||||
mdx23 = mdx23.to(device)
|
||||
mdx23.eval()
|
||||
|
||||
|
||||
@@ -309,8 +309,7 @@ class MDX23CSeparator:
|
||||
model = TFC_TDF_net(**model_config)
|
||||
|
||||
# Load weights
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint = torch.load(model_path, map_location=self.device, weights_only=True)
|
||||
checkpoint = torch.load(model_path, map_location=self.device, weights_only=False)
|
||||
|
||||
# Handle different checkpoint formats
|
||||
if 'state_dict' in checkpoint:
|
||||
|
||||
@@ -68,8 +68,7 @@ def load_hubert(model_path: str, config):
|
||||
print(f"🔧 Attempting direct .pt loading: {model_path}")
|
||||
try:
|
||||
import torch
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint = torch.load(model_path, map_location='cpu', weights_only=True)
|
||||
checkpoint = torch.load(model_path, map_location='cpu', weights_only=False)
|
||||
|
||||
# Try to create HuBERT model directly from .pt checkpoint
|
||||
# This will work if the checkpoint contains the full model with config
|
||||
@@ -186,8 +185,7 @@ def load_hubert(model_path: str, config):
|
||||
|
||||
with open(model_path, 'rb') as f:
|
||||
# Load raw data and try to extract model state_dict only
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint = torch.load(f, map_location='cpu', weights_only=True)
|
||||
checkpoint = torch.load(f, map_location='cpu', weights_only=False)
|
||||
|
||||
# Extract state_dict from various possible formats
|
||||
if hasattr(checkpoint, 'state_dict'):
|
||||
|
||||
@@ -641,8 +641,7 @@ class RMVPE:
|
||||
self.model = ort_session
|
||||
else:
|
||||
model = E2E(4, 1, (2, 2))
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
ckpt = torch.load(model_path, map_location="cpu", weights_only=True)
|
||||
ckpt = torch.load(model_path, map_location="cpu")
|
||||
model.load_state_dict(ckpt)
|
||||
model.eval()
|
||||
if is_half == True:
|
||||
|
||||
@@ -562,8 +562,7 @@ class SCNetSeparator:
|
||||
model = SCNet(**model_config)
|
||||
|
||||
# Load weights
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
checkpoint = torch.load(model_path, map_location=self.device, weights_only=True)
|
||||
checkpoint = torch.load(model_path, map_location=self.device, weights_only=False)
|
||||
|
||||
# Handle different checkpoint formats
|
||||
if 'state_dict' in checkpoint:
|
||||
|
||||
@@ -42,8 +42,7 @@ class UVR5Base:
|
||||
}
|
||||
mp = ModelParameters(os.path.join(dir_path,"uvr5_pack","vr_network","modelparams","4band_v2.json"))
|
||||
model = CascadedASPPNet(mp.param["bins"] * 2)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
cpk = torch.load(model_path, map_location=self.device, weights_only=True)
|
||||
cpk = torch.load(model_path, map_location=self.device, weights_only=False)
|
||||
try:
|
||||
model.load_state_dict(cpk)
|
||||
except RuntimeError as e:
|
||||
@@ -300,8 +299,7 @@ class UVR5New(UVR5Base):
|
||||
mp = ModelParameters(os.path.join(dir_path,"uvr5_pack","vr_network","modelparams","4band_v3.json"))
|
||||
nout = 64 if dereverb else 48
|
||||
model = CascadedNet(mp.param["bins"] * 2, nout)
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
cpk = torch.load(model_path, map_location=self.device, weights_only=True)
|
||||
cpk = torch.load(model_path, map_location=self.device, weights_only=False)
|
||||
try:
|
||||
model.load_state_dict(cpk)
|
||||
except RuntimeError as e:
|
||||
|
||||
@@ -115,8 +115,7 @@ def main():
|
||||
optimizer = th.optim.Adam(model.parameters(), lr=args.lr)
|
||||
|
||||
try:
|
||||
# TTS Audio Suite patch: restrict checkpoint deserialization.
|
||||
saved = th.load(checkpoint, map_location='cpu', weights_only=True)
|
||||
saved = th.load(checkpoint, map_location='cpu')
|
||||
except IOError:
|
||||
saved = SavedState()
|
||||
else:
|
||||
|
||||
@@ -18,10 +18,9 @@ from torch.nn import functional as F
|
||||
import tqdm
|
||||
import tkinter as tk
|
||||
|
||||
# TTS Audio Suite patch: resolve trusted checkpoint architectures within this bundled package.
|
||||
from .demucs import Demucs
|
||||
from .hdemucs import HDemucs
|
||||
from .utils import center_trim, DummyPoolExecutor
|
||||
from demucs import Demucs
|
||||
from hdemucs import HDemucs
|
||||
from utils import center_trim, DummyPoolExecutor
|
||||
|
||||
Model = tp.Union[Demucs, HDemucs]
|
||||
|
||||
|
||||
@@ -12,9 +12,8 @@ import torch
|
||||
from torch import nn
|
||||
from torch.nn import functional as F
|
||||
|
||||
# TTS Audio Suite patch: resolve trusted checkpoint architectures within this bundled package.
|
||||
from .states import capture_init
|
||||
from .utils import center_trim, unfold
|
||||
from states import capture_init
|
||||
from utils import center_trim, unfold
|
||||
|
||||
|
||||
class BLSTM(nn.Module):
|
||||
|
||||
@@ -12,11 +12,10 @@ import typing as tp
|
||||
import torch
|
||||
from torch import nn
|
||||
from torch.nn import functional as F
|
||||
# TTS Audio Suite patch: resolve trusted checkpoint architectures within this bundled package.
|
||||
from .filtering import wiener
|
||||
from .demucs import DConv, rescale_module
|
||||
from .states import capture_init
|
||||
from .spec import spectro, ispectro
|
||||
from filtering import wiener
|
||||
from demucs import DConv, rescale_module
|
||||
from states import capture_init
|
||||
from spec import spectro, ispectro
|
||||
|
||||
def pad1d(x: torch.Tensor, paddings: tp.Tuple[int, int], mode: str = 'constant', value: float = 0.):
|
||||
"""Tiny wrapper around F.pad, just to allow for reflect padding on small input.
|
||||
|
||||
@@ -9,20 +9,19 @@ This code contains the spectrogram and Hybrid version of Demucs.
|
||||
"""
|
||||
import math
|
||||
|
||||
# TTS Audio Suite patch: resolve trusted checkpoint architectures within this bundled package.
|
||||
from .filtering import wiener
|
||||
from filtering import wiener
|
||||
import torch
|
||||
from torch import nn
|
||||
from torch.nn import functional as F
|
||||
from fractions import Fraction
|
||||
from einops import rearrange
|
||||
|
||||
from .transformer import CrossTransformerEncoder
|
||||
from transformer import CrossTransformerEncoder
|
||||
|
||||
from .demucs import rescale_module
|
||||
from .states import capture_init
|
||||
from .spec import spectro, ispectro
|
||||
from .hdemucs import pad1d, ScaledEmbedding, HEncLayer, MultiWrap, HDecLayer
|
||||
from demucs import rescale_module
|
||||
from states import capture_init
|
||||
from spec import spectro, ispectro
|
||||
from hdemucs import pad1d, ScaledEmbedding, HEncLayer, MultiWrap, HDecLayer
|
||||
|
||||
|
||||
class HTDemucs(nn.Module):
|
||||
|
||||
@@ -9,8 +9,7 @@ import math
|
||||
import torch as th
|
||||
from torch import nn
|
||||
|
||||
# TTS Audio Suite patch: resolve trusted checkpoint architectures within this bundled package.
|
||||
from .utils import capture_init, center_trim
|
||||
from utils import capture_init, center_trim
|
||||
|
||||
|
||||
class BLSTM(nn.Module):
|
||||
|
||||
@@ -17,13 +17,12 @@ import logging
|
||||
from diffq import DiffQuantizer
|
||||
import torch.hub
|
||||
|
||||
# TTS Audio Suite patch: resolve trusted checkpoint architectures within this bundled package.
|
||||
from .model import Demucs
|
||||
from .tasnet_v2 import ConvTasNet
|
||||
from .utils import set_state
|
||||
from model import Demucs
|
||||
from tasnet_v2 import ConvTasNet
|
||||
from utils import set_state
|
||||
|
||||
from .hdemucs import HDemucs
|
||||
from .repo import RemoteRepo, LocalRepo, ModelOnlyRepo, BagOnlyRepo, AnyModelRepo, ModelLoadingError # noqa
|
||||
from hdemucs import HDemucs
|
||||
from repo import RemoteRepo, LocalRepo, ModelOnlyRepo, BagOnlyRepo, AnyModelRepo, ModelLoadingError # noqa
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
ROOT_URL = "https://dl.fbaipublicfiles.com/demucs/mdx_final/"
|
||||
@@ -134,8 +133,7 @@ def load_pretrained(name):
|
||||
|
||||
def _load_state(name, model, quantizer=None):
|
||||
url = get_url(name)
|
||||
# TTS Audio Suite patch: restrict checkpoint deserialization.
|
||||
state = torch.hub.load_state_dict_from_url(url, map_location='cpu', check_hash=True, weights_only=True)
|
||||
state = torch.hub.load_state_dict_from_url(url, map_location='cpu', check_hash=True)
|
||||
set_state(model, quantizer, state)
|
||||
if quantizer:
|
||||
quantizer.detach()
|
||||
|
||||
@@ -15,9 +15,8 @@ import typing as tp
|
||||
import torch
|
||||
import yaml
|
||||
|
||||
# TTS Audio Suite patch: resolve trusted checkpoint architectures within this bundled package.
|
||||
from .apply import BagOfModels, Model
|
||||
from .states import load_model
|
||||
from apply import BagOfModels, Model
|
||||
from states import load_model
|
||||
|
||||
|
||||
AnyModel = tp.Union[Model, BagOfModels]
|
||||
@@ -62,16 +61,8 @@ class RemoteRepo(ModelOnlyRepo):
|
||||
url = self._models[sig]
|
||||
except KeyError:
|
||||
raise ModelLoadingError(f'Could not find a pre-trained model with signature {sig}.')
|
||||
# TTS Audio Suite patch: restrict checkpoint deserialization.
|
||||
# TTS Audio Suite patch: download normally, then restrict legacy metadata loading.
|
||||
from urllib.parse import urlsplit
|
||||
filename = Path(urlsplit(url).path).name
|
||||
cached = Path(torch.hub.get_dir()) / "checkpoints" / filename
|
||||
if not cached.is_file():
|
||||
cached.parent.mkdir(parents=True, exist_ok=True)
|
||||
checksum = torch.hub.HASH_REGEX.search(filename)
|
||||
torch.hub.download_url_to_file(url, str(cached), hash_prefix=checksum.group(1) if checksum else None)
|
||||
return load_model(cached)
|
||||
pkg = torch.hub.load_state_dict_from_url(url, map_location='cpu', check_hash=True)
|
||||
return load_model(pkg)
|
||||
|
||||
|
||||
class LocalRepo(ModelOnlyRepo):
|
||||
|
||||
@@ -34,16 +34,6 @@ def get_quantizer(model, args, optimizer=None):
|
||||
return quantizer
|
||||
|
||||
|
||||
# TTS Audio Suite patch: preserve legacy architecture metadata without allowing
|
||||
# pickle to construct or import executable model objects.
|
||||
def _load_model_package(path):
|
||||
from utils.security.demucs_loading import load_demucs_package
|
||||
|
||||
return load_demucs_package(path, map_location="cpu",
|
||||
architecture_package=__package__ or "",
|
||||
architecture_directory=Path(__file__).parent)
|
||||
|
||||
|
||||
def load_model(path_or_package, strict=False):
|
||||
"""Load a model from the given serialized model, either given as a dict (already loaded)
|
||||
or a path to a file on disk."""
|
||||
@@ -53,8 +43,7 @@ def load_model(path_or_package, strict=False):
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("ignore")
|
||||
path = path_or_package
|
||||
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
|
||||
package = _load_model_package(path)
|
||||
package = torch.load(path, 'cpu')
|
||||
else:
|
||||
raise ValueError(f"Invalid type for {path_or_package}.")
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user