Compare commits

..
Author SHA1 Message Date
diodiogod 5a28a46010 Add audio.cpp multi-task support 2026-08-13 21:41:16 -03:00
diodiogod 211b192f4a Improve audio.cpp console feedback 2026-08-12 18:04:27 -03:00
diodiogod 4a99f15851 Add audio.cpp TTS engine integration 2026-08-12 11:50:10 -03:00
279 changed files with 17049 additions and 3908 deletions
-14
View File
@@ -1,14 +0,0 @@
# Registry archives contain runtime files and user documentation.
# These exclusions do not remove anything from the Git repository.
/.github/
/tests/
/scripts/
/PROJECT_INDEX.md
/AGENTS.md
/CLAUDE.md
/docs/BUMP_SCRIPT_INSTRUCTIONS.md
# Keep the metadata YAML files: the auxiliary registry reads its YAML at runtime.
/docs/Dev reports/*
!/docs/Dev reports/tts_audio_suite_engines.yaml
!/docs/Dev reports/tts_audio_suite_aux_models.yaml
-3
View File
@@ -14,9 +14,6 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v4
- name: Preserve Unicode filenames in Registry archives
# comfy-cli reads git ls-files as plain paths; Git must not escape emoji.
run: git config core.quotepath false
- name: Publish Custom Node
uses: Comfy-Org/publish-node-action@main
with:
-168
View File
@@ -5,174 +5,6 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [5.10.0] - 2026-10-09
### Added
- Shared Runtime installation controls and suite security improvements
- Add automatic Shared Runtime setup during TTS Audio Suite installation and updates, with a ComfyUI setting and file option to skip it.
### Changed
- Improve runtime behavior so workflows use prepared environments without installing Python packages.
- Improve file access and model loading across TTS, voice conversion, audio tools, and training to reject unsafe paths and executable checkpoint objects while keeping supported weight formats.
### Fixed
- Fix Registry packages missing guides and workflows with Unicode filenames, and exclude development-only files from published packages.
### Removed
- Remove Dedicated Runtime choices from Qwen3-TTS and Step Audio EditX; existing workflows migrate to Shared Runtime without moving other settings.
## [5.9.3] - 2026-10-08
### Fixed
- Fix isolated TTS worker startup failures from conflicting Python modules
- Fix Qwen3-TTS and Voice Design failing on first use or after Step Audio EditX with utils is not a package
- Fix conflicting module paths affecting all isolated TTS and ASR workers
- Fix isolated workers losing ComfyUI imports when their working directory changes
- Fix Step Audio EditX importing bundled modules during model discovery
- Fix Step Audio EditX CPU and MPS device patch not targeting the bundled encoder
## [5.9.2] - 2026-10-04
### Fixed
- Fix Classic ChatterBox crashing during short or consecutive speech segments (PR #368, thanks @melayyoub)
- Fix Classic ChatterBox short segments being replaced with silence by the old workaround
- Fix ChatterBox Classic and Official 23-Lang unclear failures when generation produces no usable speech; report a clear error instead
### Removed
- Remove Classic ChatterBox's old protection template and automatic short-text padding from Text and SRT; existing workflows remain compatible
## [5.9.1] - 2026-10-04
### Fixed
- Fix ChatterBox Classic SRT cache reuse during character and language switching
- Fix unnecessary language model loading when ChatterBox Classic SRT speech is already cached
- Fix ChatterBox Classic pause processing ignoring the disabled audio-cache setting
## [5.9.0] - 2026-10-03
### Added
- Add Step Audio EditX checkpoint selection and expanded sound editing
- Add dated 2026-01-23 and 2025-11-28 Step Audio EditX model choices with separate downloads
- Add expanded Step Audio EditX sound tags, including giggle, chuckle, clears throat, and new vocal reactions
- Add new Step Audio EditX sounds plus fear and humour emotions to the Multiline TTS Tag Editor
### Changed
- Improve Step Audio EditX model documentation and editor help while preserving existing workflow widget order and installed weights
### Fixed
- Fix Step Audio EditX Audio Editor ignoring the connected engine's selected model
- Fix stale Step Audio EditX edits when changing models or generation settings
- Fix ChatterBox v2/v3 native sound tags being treated as Step Audio EditX edits
## [5.8.11] - 2026-09-19
### Added
- Add Multiline TTS Tag Editor globe language picker
- Add an animated globe for choosing and swapping languages
- Add realistic simplified continent outlines and smooth travel between language locations
- Add square language buttons with a resizable layout that remembers its size
### Changed
- Improve language selection from the node's left panel
## [5.8.10] - 2026-09-04
### Fixed
- Fix Step Audio EditX failing when its speech recognition model is missing
- Remove unused dependency constraints that could break installation on newer Python versions
- Improve Echo-TTS and FAISS dependency handling through the extension installer
## [5.8.9] - 2026-09-03
### Fixed
- Fix TTS model discovery when extra_model_paths.yaml defines a shared TTS directory
- Fix local ComfyUI models/TTS installations being hidden by shared storage configuration.
- Fix download path ordering so the configured default directory remains preferred.
## [5.8.8] - 2026-08-31
### Added
- Clarify that parallel workers are available only for ChatterBox Classic
- Prevent unsupported TTS engines from entering experimental parallel processing
### Changed
- Improve legacy ChatterBox parallel-worker guidance
- Explain that the option is not true model batching and may not improve speed
## [5.8.7] - 2026-08-28
### Added
- Add native Transformers 5 support for VibeVoice and KugelAudio
- Run VibeVoice and KugelAudio directly in the main ComfyUI environment
### Fixed
- Fix VibeVoice 1.5B and KugelAudio model loading and generation on current ComfyUI installations
- Remove misleading compatibility warnings during VibeVoice loading and generation
### Removed
- Remove the VibeVoice runtime isolation setting
## [5.8.6] - 2026-08-28
### Changed
- Improve Vocal Removal cache hashing without temporary WAV encoding
### Fixed
- Fix Vocal Removal crashing before audio separation on systems with incompatible SoundFile virtual I/O
## [5.8.5] - 2026-08-26
### Added
- Add support for loading RVC models from nested folders
- Add automatic matching for index files stored beside nested RVC models
- Preserve support for the existing shared RVC index folder
### Changed
- Improve RVC character model organization
## [5.8.4] - 2026-08-21
### Added
- Keep the example workflow reliable by disabling DeepSpeed by default
- Automatically use standard inference when the optional DeepSpeed path is unsupported
### Fixed
- Fix IndexTTS DeepSpeed failures
## [5.8.3] - 2026-08-18
### Changed
- Improve long-running ChatterBox session stability
### Fixed
- Fix ChatterBox generations degrading after extended use
- Fix English ChatterBox returning very short audio after many generations
- Fix Official 23-Lang v1/v2 accumulating background processing hooks
## [5.8.2] - 2026-08-18
### Added
- Keep trained adapters visible and selectable with the matching community model
### Fixed
- Fix MOSS Voice Acting LoRA selection
- Fix LoRA controls appearing unavailable when the LAION Voice Acting 8B model is selected
## [5.8.1] - 2026-08-11
### Added
+6 -21
View File
@@ -17,7 +17,7 @@
**Key architectural rules:**
- Chunking happens in the **processor**, not the adapter (`generate_single()` on adapter = raw single call)
- Runtime routing happens through `ModelLoadConfig.runtime_mode` + `runtime_profile`, not ad-hoc subprocess calls
- Shared runtime workers are currently used for fragile engine families such as Qwen3-TTS / ASR, Granite forced alignment, and Higgs Audio 2. VibeVoice/Kugel and other engines that support the modern stack run natively in the main Transformers 5 environment.
- Shared runtime workers are currently used for fragile engine families such as VibeVoice, Qwen3-TTS / ASR, Granite forced alignment, and Higgs Audio 2. Engines that support the modern stack run natively in the main Transformers 5 environment.
- YAML (`docs/Dev reports/tts_audio_suite_engines.yaml`) is source of truth for engine doc tables → run `python3 scripts/generate_engine_tables.py --readme` to regenerate
- Auxiliary YAML (`docs/Dev reports/tts_audio_suite_aux_models.yaml`) is source of truth for helper/post-process model docs → run `python3 scripts/generate_aux_model_docs.py`
- All models download to `ComfyUI/models/TTS/<model-name>/`
@@ -78,7 +78,6 @@
- `ENGINE_COMPARISON.md`, `LANGUAGE_SUPPORT.md`, `FEATURE_COMPARISON.md` - Auto-generated tables
### Dev Docs (`docs/Dev reports/`)
- `REGISTRY_COMPATIBILITY_PLAN.md` - File access, safe loading, config parsing, runtime installation/preferences, archive fixes, verification and Registry approval evidence
- `tts_audio_suite_engines.yaml` - **Source of truth** for all engine metadata
- `tts_audio_suite_aux_models.yaml` - **Source of truth** for helper/post-process model metadata
- `SRT_IMPLEMENTATION.md` - SRT timing technical details
@@ -120,7 +119,7 @@
- `nodes/subtitles/text_to_srt_builder_node.py` - Build SRT from transcript text plus timing data, or estimate timings from plain text
- `nodes/subtitles/srt_advanced_options_node.py` - Subtitle readability / segmentation policy options
- `nodes/text/tts_tag_editor_node.py` - 🏷️ Multiline TTS Tag Editor: rich text editor with character/language/parameter dropdowns, preset system, syntax highlighting, undo/redo — pairs with `web/string_multiline_tag_editor.js`
- `nodes/step_audio_editx_special/step_audio_editx_audio_editor_node.py` - 🎨 Audio Editor: post-process ANY engine's audio with Step Audio EditX (emotion/style, paralinguistic effects, speed); dated checkpoint choices and tags documented in `docs/STEP_AUDIO_EDITX_CHECKPOINTS.md`
- `nodes/step_audio_editx_special/step_audio_editx_audio_editor_node.py` - 🎨 Audio Editor: post-process ANY engine's audio with Step Audio EditX (14 emotions, 32 styles, paralinguistic effects like `<Laughter>`, speed control) — universal, not just for Step Audio EditX engine
- `nodes/engines/index_tts_emotion_options_node.py` - IndexTTS-2 emotion radar chart
### Audio / Video Nodes
@@ -137,19 +136,19 @@
- `unified_model_interface.py` - Universal factory pattern for all engines
- `engine_registry.py` - Engine capability definitions
- `factory_config.py` - standardized model load config, runtime mode/profile normalization
- `step_audio_editx_checkpoints.py` - Dated Step Audio EditX choices and pinned Hugging Face revisions
- `manager.py` - Model discovery and caching
- `comfyui_model_wrapper/` - ComfyUI native model management integration
- `extra_paths.py` - extra_model_paths.yaml support
### Isolated Runtimes (`utils/runtimes/`)
- `profiles.py` - named runtime profiles (shared/dedicated legacy Transformers 4 environments and engine-specific runtimes)
- `launcher.py` - worker environment construction, suite-first import paths, Windows toolchain env setup
- `profiles.py` - named runtime profiles (`vibevoice_transformers4_shared`, dedicated variants, etc.)
- `launcher.py` - runtime bootstrap, venv creation, Windows toolchain env setup
- `session.py`, `protocol.py` - JSONL worker transport and message protocol
- `bootstrap.py` - shared runtime bootstrap helpers
- `vibevoice_proxy.py`, `qwen3_tts_proxy.py`, `qwen3_asr_proxy.py`, `higgs_audio_proxy.py` - parent-process proxies
- `workers/` - worker subprocess entrypoints for VibeVoice, Qwen3-TTS, Qwen3-ASR/aligner, Higgs Audio, Fish Audio S2, and Step Audio EditX; each prioritizes suite imports before third-party dependencies
- `workers/` - worker subprocess entrypoints for VibeVoice, Qwen3-TTS, Qwen3-ASR/aligner, Higgs Audio
- Current shared legacy T4 runtime profile is reused by:
- VibeVoice / Kugel
- Qwen3-TTS
- Qwen3-ASR and Granite's optional Qwen forced aligner
- Higgs Audio 2
@@ -204,20 +203,6 @@
- `web/asr_srt_preset_widgets.js` - ASR SRT preset locking
## Scripts & Config
- `.comfyignore` - Registry archive exclusions for development tools and reports; keeps runtime metadata YAML, user documentation, and engine/training code
- `scripts/bump_version_enhanced.py` - Version bump with changelog (use `patch`/`minor`/`major`)
- `scripts/generate_engine_tables.py` - Regenerate all docs from YAML (`--readme` flag for README too)
- `requirements.txt`, `pyproject.toml` - Dependencies and project metadata
## Registry compatibility helpers
- `utils/security/path_access.py` - Configured data-folder containment, link checks and safe output paths
- `utils/security/data_loading.py` - Restricted checkpoint/dictionary/NumPy loading and scoped dependency adapters
- `utils/security/demucs_loading.py`, `audio_separator_loading.py` - Preserve legacy separator metadata with restricted dependency loaders
- `utils/downloads/unified_downloader.py` - Model download destinations contained in registered data folders
- `utils/security/config_values.py` - Data-only config expressions with bounded list arithmetic
- `utils/runtimes/bootstrap.py` - Runtime lookup/readiness only; generation never installs packages
- `utils/runtimes/installation.py` - Installation-time preparation invoked by `install.py`
- `utils/runtimes/settings.py`, `settings_api.py` - One persisted installation preference and same-origin UI writes
- `web/runtime_installation_settings.js` - Shared runtime installation setting and status/cleanup details
- `web/runtime_mode_compatibility.js` - Dedicated-to-shared migration for older workflows
+9 -40
View File
@@ -7,7 +7,7 @@
[![Dynamic TOML Badge][version-shield]][version-url]
[![Ko-Fi](https://img.shields.io/badge/Ko--fi-F16061?style=for-the-badge&logo=ko-fi&logoColor=white)](https://ko-fi.com/diogogo)
# TTS Audio Suite v5.10.0
# TTS Audio Suite v5.8.1
[![ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/diogogo)
@@ -246,8 +246,8 @@ For comprehensive technical information, refer to the [SRT_IMPLEMENTATION.md](do
This is the new architectural baseline for the suite.
* **Main environment moved forward**: the primary ComfyUI environment is now meant to run on **Transformers 5**
* **Isolation for fragile engines**: engines that still behave better on the older stack can use **a shared** legacy runtime instead of forcing the whole suite backward
* **Cleaner engine strategy**: modern engines such as **Higgs Audio v3**, **MOSS-TTS**, and other compatible stacks can stay native in the main environment
* **Isolation for fragile engines**: engines that still behave better on the older stack can use **shared** or **dedicated** legacy runtimes instead of forcing the whole suite backward
* **Cleaner engine strategy**: modern engines such as **Higgs Audio v3**, **Step Audio EditX**, **MOSS-TTS**, and other compatible stacks can stay native in the main environment
* **Less dependency deadlock**: adding new engines no longer has to mean globally freezing the entire project to one old Transformers version
This matters because the suite now has a clearer split:
@@ -429,7 +429,7 @@ dataset formats, training workflow, adapter loading, and CPU-safe preflight.
3. Choose between Custom Character Switching (recommended) or Native Multi-Speaker mode
4. Generate long-form content with automatic voice cloning from your voices folder
VibeVoice and KugelAudio run directly in the suite's Transformers 5 main environment.
**Isolation note:** On current ComfyUI stacks, **Kugel/VibeVoice is usually meant to run with `⚠️ Runtime Isolation = Shared Runtime`**. If you force `Main Environment`, you are opting back into the dependency-conflict path.
**Perfect for:**
@@ -811,10 +811,9 @@ Welcome to our show! [Alice:happy_sarah] I'm so excited to be here!
**NEW in v4.15**: Revolutionary LLM-based audio post-processing with emotion, style, and paralinguistic control!
* **🎨 Step Audio EditX - Audio Editor Node**: Post-process ANY TTS audio with advanced editing capabilities
* **📅 Dated Checkpoints**: Choose `2026-01-23` (default for new engine nodes) or `2025-11-28` in the existing model dropdown. Downloads use pinned revisions and separate folders; existing unversioned/local selections keep their files. [Checkpoint guide](docs/STEP_AUDIO_EDITX_CHECKPOINTS.md).
* **🗣️ Paralinguistic Effects**: Original tags plus January sounds such as `<giggle>`, `<chuckle>`, `<inhale>`, `<exhale>`, `<cough>`, and `<clears_throat>`; iteration syntax remains available, e.g. `<giggle:2>`.
* **😊 Emotion Controls**: happy, sad, angry, excited, fear, humour, surprised, disgusted, confusion, empathy, embarrass, depressed, coldness, admiration; older `calm` and `fearful` options remain accepted
* **🎭 Speaking Styles**: whisper, serious, child, older, girl, pure, sister, sweet, exaggerated, ethereal, generous, recite, act_coy, warm, shy, comfort, authority, chat, radio, soulful, gentle, story, vivid, program, news, advertising, roar, murmur, shout, deeply, loudly, arrogant, friendly
* **🗣️ Paralinguistic Effects**: Insert natural sounds - Laughter, Breathing, Sigh, Uhm, Surprise (oh/ah/wa), Confirmation, Question, Dissatisfaction
* **😊 14 Emotion Controls**: happy, sad, angry, excited, calm, fearful, surprised, disgusted, confusion, empathy, embarrass, depressed, coldness, admiration
* **🎭 32 Speaking Styles**: whisper, serious, child, older, girl, pure, sister, sweet, exaggerated, ethereal, generous, recite, act_coy, warm, shy, comfort, authority, chat, radio, soulful, gentle, story, vivid, program, news, advertising, roar, murmur, shout, deeply, loudly, arrogant, friendly
* **⚡ Speed Control**: faster, slower, more_faster, more_slower with multi-iteration support
* **🔊 Voice Restoration**: ChatterBox VC integration to restore original voice resemblance after editing
* **🏷️ Inline Edit Tags**: Apply effects directly in text using `<Laughter:2>`, `<emotion:happy>`, `<style:whisper>` tags
@@ -1218,31 +1217,6 @@ Perfect for:
- 🟢 **OpenSeeFace mouth movement**: ✅ Working (experimental)
- 🔴 **MediaPipe mouth movement**: ❌ Incompatible (use OpenSeeFace)
### Shared Runtime installation
**Most users can keep the defaults.** Manager prepares one Shared Runtime during installation or updates. Qwen3-TTS/ASR, Step Audio EditX, Higgs Audio 2, and forced alignment reuse it. Model weights download only when needed. Running a workflow or restarting ComfyUI does not install Python packages.
If an engine reports that Shared Runtime is missing or needs repair, enable **Install shared runtime automatically** under **Settings > TTS Audio Suite > Runtime installation**, then repair/reinstall TTS Audio Suite through Manager. Restart when Manager requests it.
<details>
<summary>Advanced: skip installation or remove an unused shared runtime</summary>
Turn off **Install shared runtime automatically** in the setting above to skip preparation during future installs and updates. Engines still set to Shared Runtime can use an existing environment; the setting does not switch engines to Main Environment.
Open **TTS Audio Suite > Runtime installation details** from ComfyUI's menu to see the configuration file, runtime folder, and installation status. To reclaim the runtime's space, turn off automatic installation, close ComfyUI, and manually delete the displayed **shared_legacy_t4** folder. Model weights and voice files are separate. Engines set to Shared Runtime need that environment restored before they can run again.
To skip the first installation, create this file before installing the suite:
**ComfyUI/user/__tts_audio_suite/runtime_settings.json**
```json
{"install_shared_runtime": false}
```
The UI and installer read the same file. With a custom ComfyUI user directory, use its **__tts_audio_suite** folder. After the suite has loaded, it remembers that location for Manager updates and reinstalls. Before the first installation with a custom user directory, set **TTS_AUDIO_SUITE_USER_DIRECTORY** to that directory in the installer's environment.
</details>
### Option 2: Manual Installation
**Same intelligent installer, manual setup:**
@@ -1442,12 +1416,7 @@ Core TTS installation will continue; only the listed features may be unavailable
A common problem is installing dependencies in the wrong Python environment. Always ensure you are installing dependencies within your ComfyUI's Python environment.
Where an engine offers **⚠️ Runtime Isolation**, there are two choices:
- **Main Environment** uses ComfyUI's installed Python packages.
- **⚠️ Shared Runtime** uses one support environment shared by compatible engines, helping avoid dependency conflicts. Keep the engine's default unless you need to change it.
Older workflows that selected Dedicated Runtime now use Shared Runtime. Their other settings stay in place. Existing dedicated folders are kept; after closing ComfyUI, you can delete an unused dedicated folder to reclaim its space. Keep the **shared_legacy_t4** folder if you use Shared Runtime.
If the engine comparison table shows **`Shared`** or **`Dedicated`** in the **Isolation** column, that engine has its own secondary-environment path for dependency conflicts. Configure that on the engine node with `⚠️ Runtime Isolation` instead of trying to downgrade your main ComfyUI environment.
* **Verify your Python environment:** After activating your venv or navigating to your portable ComfyUI installation, check the Python executable being used:
@@ -1565,7 +1534,7 @@ For offline/manual setup:
| VibeVoice | `ComfyUI/models/TTS/VibeVoice/` | ✅ | 1.5B and 7B variants |
| RVC | `ComfyUI/models/TTS/RVC/` | ✅* | Base models auto; character `.pth` can be user-provided |
| IndexTTS-2 | `ComfyUI/models/TTS/IndexTTS/` | ✅ | Emotion components included |
| Step Audio EditX | `ComfyUI/models/TTS/step_audio_editx/` | ✅ | Lazy dated checkpoints + tokenizer stack; unversioned installs retained |
| Step Audio EditX | `ComfyUI/models/TTS/step_audio_editx/` | ✅ | Main model + tokenizer stack |
| CosyVoice3 | `ComfyUI/models/TTS/CosyVoice/` | ✅ | Variant-specific lazy downloads |
| Qwen3-TTS / ASR | `ComfyUI/models/TTS/qwen3_tts/` | ✅ | Per-variant download + shared tokenizer |
| MOSS-TTS | `ComfyUI/models/TTS/moss_tts/` | ✅ | Local/Delay/VoiceGenerator/SoundEffect v1/TTSD models plus shared MOSS-Audio-Tokenizer codec |
+6 -10
View File
@@ -344,7 +344,6 @@ def setup_api_routes():
import folder_paths
from server import PromptServer
from aiohttp import web
from utils.security.path_access import resolve_input_path, filename_component, child_path
def _get_ui_data_dir():
base_dir = os.path.join(folder_paths.get_system_user_directory("tts_audio_suite"), "ui")
@@ -356,8 +355,8 @@ def setup_api_routes():
from utils.voice.alias_api import register_character_alias_routes
register_character_alias_routes(PromptServer.instance.routes, web)
from utils.runtimes.settings_api import register_runtime_settings_routes
register_runtime_settings_routes(PromptServer.instance.routes, web)
from utils.audio_cpp.capability_api import register_audio_cpp_capability_routes
register_audio_cpp_capability_routes(PromptServer.instance.routes, web)
@PromptServer.instance.routes.get("/api/tts-audio-suite/index-tts-emotion-presets")
async def get_index_tts_emotion_presets_endpoint(request):
@@ -644,8 +643,7 @@ print(json.dumps({"devices": devices}))
if not audio_file:
return web.json_response({"error": "audio_file is required for preview analysis"}, status=400)
audio_file = resolve_input_path(audio_file)
node_id = filename_component(data.get("node_id") or "preview")
node_id = str(data.get("node_id") or "preview")
analyzer_node_path = os.path.join(os.path.dirname(__file__), "nodes", "audio", "analyzer_node.py")
spec = importlib.util.spec_from_file_location("tts_audio_suite_audio_analyzer_node", analyzer_node_path)
@@ -653,7 +651,7 @@ print(json.dumps({"devices": devices}))
spec.loader.exec_module(analyzer_node_module)
analyzer_node = analyzer_node_module.AudioAnalyzerNode()
analysis_result = analyzer_node.analyze_audio(
analyzer_node.analyze_audio(
audio_file=audio_file,
analysis_method=data.get("analysis_method", "silence"),
precision_level=data.get("precision_level", "milliseconds"),
@@ -667,9 +665,7 @@ print(json.dumps({"devices": devices}))
)
import folder_paths
if analysis_result[1].startswith("Error:"):
return web.json_response({"error": analysis_result[1]}, status=400)
cache_file = child_path(folder_paths.get_output_directory(), f"audio_analyzer_cache_{node_id}.json")
cache_file = os.path.join(folder_paths.get_output_directory(), f"audio_analyzer_cache_{node_id}.json")
with open(cache_file, "r", encoding="utf-8") as f:
payload = json.load(f)
@@ -678,7 +674,7 @@ print(json.dumps({"devices": devices}))
return response
except Exception as e:
print(f"⚠️ Audio analyzer preview failed: {e}")
return web.json_response({"error": str(e)}, status=400 if isinstance(e, (ValueError, FileNotFoundError)) else 500)
return web.json_response({"error": str(e)}, status=500)
@PromptServer.instance.routes.get("/api/tts-audio-suite/training-progress")
async def get_training_progress_endpoint(request):
@@ -1,9 +1,5 @@
# Isolated Runtimes Plan
Historical design. The current implementation uses Main Environment or one Shared Runtime,
prepared by install.py. Dedicated profiles and first-use package installation were removed;
see [Registry compatibility work](REGISTRY_COMPATIBILITY_PLAN.md).
## Goal
Keep ComfyUI in one modern main runtime while routing fragile engines into
@@ -1,248 +0,0 @@
# Registry compatibility work
Scope implemented locally on 2026-10-09: file containment, restricted data loading,
configuration parsing, installation-time Shared Runtime setup, user preferences,
dedicated-runtime migration, and Registry archive cleanup. Local verification is recorded below.
These verification records do not constitute Registry approval.
## 1. File access
- Restrict workflow and HTTP-supplied file paths to the configured ComfyUI
input/output/temp or explicitly registered model/voice folders, as appropriate
for the operation. Resolve links before checking containment.
- Keep the audio analyzer's existing widget and browser upload/drag-and-drop.
Uploaded audio is already placed in input by ComfyUI's upload endpoint.
- Reject paths outside the permitted folders. Copying an arbitrary server-side
path into input would still read that path and expose its contents.
- Audit the analyzer preview route and node execution together, plus the suite's
other file-backed widgets and routes. Protect generated cache/output names
from raw node IDs and other request values.
- Preserve explicitly configured external model and voice directories; do not
turn their normal use into an unrestricted path exception.
- Restrict model download destinations before network access, writes, or replacement;
reject traversal and links escaping the permitted data roots. Custom-node code
folders are excluded from data roots even though ComfyUI registers them.
## 2. Checkpoint and dictionary loading
- Prefer restricted PyTorch loading for state dictionaries; keep existing model
formats where safe loading supports them. Do not require conversion or copies
of upstream weights as the default solution.
- The local format audit covered 52 PyTorch files and one plain-pickle language
dictionary: 50 checkpoints opened directly with weights_only=True, two Dots
latent-statistics files needed a scoped NumPy type allowlist, and the dictionary
loaded with a data-only unpickler that refuses class/global lookups.
- The audit used the configured test interpreter with Torch 2.10.0+cu130. It did
not establish full engine inference, every remote variant, TorchScript safety,
or compatibility with old whole-model HuBERT files and all training resumes.
- Review RVC inference/separation/training loaders, DramaBox training, Dots
integration, optional Russian stress dictionaries, and remaining unsafe
helpers. No automatic fallback to unrestricted loading for an untrusted file.
- Keep dependency patches scoped to suite integration rather than changing
unrelated installed packages or globally weakening deserialization.
## 3. Configuration expressions
- Replace the Qwen tokenizer's constant eval with its literal value.
- Replace bundled FunASR configuration eval calls with restricted data parsing.
Preserve required list/tuple expressions, including existing list addition and
repetition; literal_eval alone does not cover those expressions.
- Keep neural-network model.eval() calls. Add the required adjacent TTS Audio
Suite patch comments for bundled-code changes.
## 4. Optional isolated environments
The maintainer rejected a manual setup command as the normal user experience.
Required behavior: users get automatic setup or a clear install action inside
ComfyUI, without editing files, running terminal commands, or learning about
Python environments. Keep existing workflow widgets and their order. Generation now checks prepared-runtime metadata and never installs packages.
The installer prepares or reuses Shared Runtime independently of its main-dependency fast path.
Implemented: prepare the single shared legacy environment automatically from the existing install.py hook during normal
Manager installation/update. Default Qwen3-TTS, Qwen ASR, Step Audio EditX, and
Higgs Audio 2 already select this environment. Users keep the existing experience
without terminal setup or additional support packages. Model weights still
download only when the corresponding engine is used.
The local shared environment measured 139.1 MB (132.7 MiB), including 6,429
files. It inherits the main environment's heavy packages rather than installing
another PyTorch copy. This measures the current machine, not download size or
a universal upper bound; missing/incompatible host dependencies can change it.
Keep an advanced opt-out for automatic shared-environment preparation. A local
installer configuration can apply before first installation; a ComfyUI setting
can control subsequent installation/update/repair runs. Skipping preparation
must neither delete an existing environment nor silently switch engines to an
incompatible main environment. A missing required environment should report
that support must be enabled and installation repaired through Manager.
### Implemented advanced controls
- Add a default-on "Install shared runtime automatically" setting under
ComfyUI Settings > TTS Audio Suite > Runtime installation.
- Store the preference in one server-side JSON file read by the installer.
The UI edits that same file and shows its actual location; a hand-edited
preference must not be overwritten with stale browser defaults on page load.
Prefer the suite's existing ComfyUI System User storage, outside package files
replaced during updates. Resolve the configured user directory consistently
in the running suite and the separate installer process.
Default location: ComfyUI/user/__tts_audio_suite/runtime_settings.json.
Document the equivalent location under a custom user directory so pre-install
opt-out does not depend on opening a UI that is not installed yet.
- File schema:
```json
{
"install_shared_runtime": false
}
```
- Only accept the defined boolean. Do not
accept executable paths, package names, URLs, or shell commands from this file,
a workflow, or a management request. Protect settings-management writes.
- Turning shared preparation off affects later installer runs. It does not
remove the installed environment, stop its use by existing workflows, or
select Main Environment instead. The file option supports pre-install opt-out;
the UI becomes available after the suite is installed.
- Do not add automatic deletion on restart. Show whether the environment is
installed, its folder location, and instructions to close ComfyUI before
manually deleting runtimes/shared_legacy_t4. Explain that engines still set
to Shared Runtime then need that environment restored or another prepared,
compatible runtime selected. Model weights and voice files are separate.
### Runtime choices implemented locally
- Remove Dedicated Runtime from Qwen3-TTS and Step Audio EditX. Runtime widgets
stay in their existing positions with Main Environment and Shared Runtime.
- Old Dedicated Runtime widget/API values and old dedicated profile names map
to Shared Runtime. Frontend migration also updates saved workflow widgets.
- Remove the separate Qwen, Step, and VibeVoice legacy profile definitions; keep
the shared profile unchanged so existing installations can be reused.
- Leave existing dedicated environment folders on disk. Do not delete user files
as part of this change. Users can remove unused folders after closing ComfyUI.
- No dedicated installation controls or file settings are needed. Reuse valid
shared runtime metadata on later updates, checking profile changes separately
from the main dependency installer's fast path.
Manager can defer its installation jobs until restart, including on Windows.
That is still Manager-requested package installation. A workflow node recording
flags and our prestartup/import code calling pip on every boot is a different
boundary; timing alone does not make it acceptable. No new installer node is
needed for the recommended default. The implementation still requires Registry review; it does not assert approval under the latest policy.
Reuse existing compatible environments. An update must not silently delete a
working environment during generation. Model downloads remain separate from
installing executable Python dependencies. An HTTP route or workflow node that
invokes pip after a click would still be runtime installation; renaming pip or
hiding it in a helper is not a policy fix.
Evidence checked:
- [Official standards](https://docs.comfy.org/registry/standards) prohibit runtime
package installation through subprocess calls.
- [Manager's install queue](https://github.com/Comfy-Org/ComfyUI-Manager/blob/main/glob/manager_server.py)
accepts registered node IDs/versions and delegates to its package installer.
Its [special-purpose files documentation](https://github.com/Comfy-Org/ComfyUI-Manager#custom-node-support)
documents automatic install.py execution during installation.
- [Manager's deferred installer](https://github.com/Comfy-Org/ComfyUI-Manager/blob/main/glob/manager_core.py#L2070)
schedules installation jobs for restart when appropriate, rather than requiring
each node pack to create an independent startup installer.
- [SAM3 Registry versions](https://api.comfy.org/nodes/comfyui-sam3/versions?include_status_reason=true):
0.1.21 is Active, with a manual SAFE decision by drltdata@comfy.org under
policy-v0.1. Its [published archive](https://cdn.comfy.org/pznodes/comfyui-sam3/0.1.21/node.zip)
contains install.py calling comfy_env.install(), plus isolated-environment
dependency configuration. This is an accepted installation-time precedent,
not a blanket exception for our current first-use bootstrap or newer policies.
- [comfy-env installation documentation](https://docs.comfy-forge.org/comfy-env/install/)
describes installation-time creation rather than environment creation during
node execution. Adoption of that library is not required for this plan.
- Registry review requests [226](https://github.com/Comfy-Org/registry-backend/issues/226),
[261](https://github.com/Comfy-Org/registry-backend/issues/261), and
[210](https://github.com/Comfy-Org/registry-backend/issues/210) are unanswered;
they do not prove approval of optional runtime installers.
### User documentation
README.md now describes automatic installation, the UI/file opt-out, custom user
directories, repair through Manager, and manual cleanup. The details command shows
the actual preference file and runtime folder; no UI route or node runs pip.
## 5. Registry archive contents
- Added .comfyignore to exclude root tests/scripts, GitHub automation, the project
index, and development reports from Registry archives. Repository contents
remain available for development.
- Keep the two metadata YAML files in docs/Dev reports. The auxiliary model
registry reads its YAML at runtime. Keep user guides, example workflows,
frontend assets, and all engine/training code.
- The publisher's current Git filename handling skips quoted emoji paths.
Configure core.quotepath=false in publishing so those guides/examples ship.
## Verification for the implementation
- Compare an archive with an independently collected, NUL-delimited Git source
manifest (git ls-files -z), not the packer's own parsed filename list. Require
every missing file to have an intentional exclusion and verify retained bytes.
This catches silent omissions from quoting, decoding, and path parsing.
- Exercise uploads, allowed typed paths, rejected traversal/link escapes, and
malformed node IDs through both node execution and HTTP preview routes.
- Check restricted loaders against the actual local formats, then exercise the
affected engine inference and training paths using the configured test setup.
- Check existing prepared environments and missing/stale profiles. Confirm that
generation cannot install packages or replace an environment.
- Recheck the published Registry status separately from successful upload;
packaging cleanup by itself does not fix the runtime findings.
## Local checks for dedicated runtime removal
- The existing runtime unit test module passed all 27 cases, including legacy
mode/profile migration and rejection of unsupported values.
- JavaScript migration checks covered 20 hook/value combinations and confirmed
other widget positions/values and unrelated nodes remain unchanged.
- The restarted ComfyUI server exposes only Main Environment and Shared Runtime
for Qwen3-TTS and Step Audio EditX, with the original input order. Its prompt
validation accepts old dedicated values and rejects unknown modes. Requests
deliberately omitted required downstream inputs, so no generation was queued.
- ComfyUI serves the new frontend extension. Full-fix checks are recorded below.
- Source rewrites must match AST call identity. Compare preserved attribute-call
identities against the original source, independently of edit selection; builtin
eval removal must not rename model.eval() or other unrelated methods.
## Completed local verification
- 59 targeted existing unit cases passed, covering runtime/profile migration,
installer repairs, audio processing, voice compatibility, and dependency guidance.
- An empty temporary Shared Runtime was created by the real installer using the
configured test Python. It installed Transformers 4.57.3, inherited host PyTorch,
passed import/readiness/reuse checks, and left the host packages unchanged. The
temporary environment was removed. Windows console logging uses ASCII output.
- Installer hook checks covered enabled preparation and file-based opt-out.
Missing/stale runtime checks cannot run subprocess installers during generation;
simulated failed installation restores the previous runtime.
- Live API checks passed for uploaded audio analysis, permitted paths, blocked
traversal/external paths/node IDs, strict settings values, same-origin writes,
preference persistence, and preservation of installed runtime files.
- The real browser showed the default-on setting and the details menu with the
correct file/folder paths and manual cleanup instructions. Backend file edits
override stale browser preferences. Frontend checks also covered rapid toggles
and restoring the checkbox after a failed save.
- Live Step TTS followed by a giggle edit succeeded through the Shared Runtime,
including migration from the old dedicated API value. The final run produced
non-silent 24 kHz audio lasting 3.12 seconds.
- Restricted Dots latent statistics, the 3,060,182-entry Russian dictionary, NumPy
data/DAC formats, and Demucs architecture metadata passed. A small real Demucs
model restored its state through audio-separator's existing load_model entry
point. Malicious pickle/NumPy/Torch payloads and model-constructor REDUCE calls
were rejected. Dependency patches preserve global torch/importlib references.
- Python source compilation and independent AST comparison preserved every
existing neural model.eval() call. Required config list arithmetic remains usable.
- The candidate archive includes intended uncommitted source without staging it:
1,936 packaged files and 104 intentional exclusions. Runtime files, Unicode
paths, user guides, and both metadata YAML files retain exact bytes. The auxiliary
model registry reads its YAML successfully from the packaged layout.
Limits: Manager is absent from the test installation, so its real queue/restart
flow was not exercised; the install.py hook and fresh runtime installation were
tested directly. This does not establish every engine's inference/training resume,
support arbitrary whole-object checkpoints, or constitute Registry approval.
@@ -109,7 +109,7 @@ Decision
Operational Recommendation
- Treat Qwen3-TTS as a legacy-stack engine for now.
- Route it into the shared legacy Transformers 4 runtime used by compatible engines (the same runtime family already proven for Qwen in prior testing).
- Route it into the shared legacy Transformers 4 runtime used by compatible engines (same family already proven for VibeVoice/Qwen in prior testing).
- Keep the main environment on Transformers 5.
- Revisit native Transformers 5 support only if Qwen3-TTS becomes strategically important enough to justify deeper behavior-level compatibility work.
+25 -29
View File
@@ -12,6 +12,10 @@ runtime_isolation_modes:
label: "Shared Runtime"
short_label: "Shared"
description: "Use a secondary shared runtime for engines with dependency conflicts. This runtime reuses heavy base packages like PyTorch from the main environment when configured."
dedicated_runtime:
label: "Dedicated Runtime"
short_label: "Dedicated"
description: "Use a dedicated secondary runtime only for one engine when the shared runtime is not enough."
engines:
- id: f5-tts
@@ -372,6 +376,17 @@ engines:
- "Multilingual (KugelAudio variants)"
- "4-bit quantization"
runtime_isolation:
default_mode: "shared_runtime"
supported_modes:
- "main_environment"
- "shared_runtime"
- "dedicated_runtime"
status: "implemented"
runtime_profile: "vibevoice_transformers4_shared"
dedicated_profile: "vibevoice_transformers4_dedicated"
notes: "Recommended on modern ComfyUI stacks because VibeVoice/Kugel currently needs a secondary dependency layer. The shared runtime reuses heavy base packages like PyTorch from the main environment to avoid full duplication."
model_sources:
- component: "vibevoice-1.5B"
source_name: "microsoft/VibeVoice-1.5B"
@@ -850,8 +865,10 @@ engines:
supported_modes:
- "main_environment"
- "shared_runtime"
- "dedicated_runtime"
status: "implemented"
runtime_profile: "vibevoice_transformers4_shared"
dedicated_profile: "qwen3_tts_transformers4_dedicated"
notes: "Recommended on modern ComfyUI stacks because Qwen3-TTS, Qwen3-ASR, and the reused Qwen forced aligner currently rely on the shared legacy T4 runtime for stable operation."
model_sources:
@@ -1044,35 +1061,19 @@ engines:
asr: false
training: false
runtime_isolation:
default_mode: "shared_runtime"
supported_modes:
- "main_environment"
- "shared_runtime"
status: "implemented"
runtime_profile: "vibevoice_transformers4_shared"
notes: "Shared Transformers 4 is recommended; the main Transformers 5 stack can produce invalid audio tokens."
special_features:
- "Second Pass Speech Editing Node: 14 emotions"
- "32 speaking styles"
- "Dated checkpoints: 2026-01-23 / 2025-11-28"
- "Expanded January paralinguistic effects with legacy tag compatibility"
- "Selectable main or shared Python runtime (shared Transformers 4 runtime recommended)"
- "Paralinguistic effects"
- "Selectable main, shared, or dedicated Python runtime (shared Transformers 4 runtime recommended)"
model_sources:
- component: "Step-Audio-EditX-2026-01-23"
- component: "Step-Audio-EditX"
source_name: "stepfun-ai/Step-Audio-EditX"
source_url: "https://huggingface.co/stepfun-ai/Step-Audio-EditX"
size: "~7GB"
auto_download: true
notes: "Default 3B checkpoint; pinned to 5fe2f8a05c2353301ad47d3c1747b262115da138"
- component: "Step-Audio-EditX-2025-11-28"
source_name: "stepfun-ai/Step-Audio-EditX"
source_url: "https://huggingface.co/stepfun-ai/Step-Audio-EditX/tree/7f3de603ae46c96dff6f06f47b1d5a45aabd34fe"
size: "~7GB"
auto_download: true
notes: "Legacy checkpoint; pinned November weights in a separate folder"
notes: "Main 3B audio editing model"
- component: "Step-Audio-Tokenizer"
source_name: "stepfun-ai/Step-Audio-Tokenizer"
source_url: "https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer"
@@ -1216,6 +1217,7 @@ engines:
default_mode: "main_environment"
main_environment: true
shared_runtime: false
dedicated_runtime: false
special_features:
- "Free-form sub-word emotion/prosody tags"
@@ -1966,7 +1968,7 @@ readme_model_download_table:
- engine: "Step Audio EditX"
primary_model_path: "ComfyUI/models/TTS/step_audio_editx/"
auto_download: "✅"
notes: "Lazy dated checkpoints + tokenizer stack; unversioned installs retained"
notes: "Main model + tokenizer stack"
- engine: "CosyVoice3"
primary_model_path: "ComfyUI/models/TTS/CosyVoice/"
auto_download: "✅"
@@ -2205,20 +2207,14 @@ model_layouts_markdown: |
```text
ComfyUI/models/TTS/step_audio_editx/
├── Step-Audio-EditX-2026-01-23/
│ └── CosyVoice-300M-25Hz/
├── Step-Audio-EditX-2025-11-28/ # downloaded only when selected
│ └── CosyVoice-300M-25Hz/
├── Step-Audio-EditX/ # existing unversioned installations
├── Step-Audio-EditX/
│ └── CosyVoice-300M-25Hz/
└── FunASR-Paraformer/
```
Notes:
- Selected checkpoints and tokenizer assets auto-download at pinned revisions.
- Existing unversioned/local weights are retained and folders are not relabeled; known config compatibility fields may be normalized.
- See [Step Audio EditX checkpoints](STEP_AUDIO_EDITX_CHECKPOINTS.md) for selection and expanded tags.
- Main model, tokenizer assets, and speech stack auto-download.
## Higgs Audio v3
+3 -3
View File
@@ -7,14 +7,14 @@
| **F5-TTS** | Main | Base, v1, E2TTS + 8 lang models | ~1.2GB each | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | CC-BY-NC-4.0 | Targeted Word/Speech Editing, Speed control | 10 |
| **ChatterBox** | Main | EN, DE×3, IT, FR, RU, HY, KA, JA, KO, NO | ~4.3GB | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | MIT | Expressiveness slider | 10 |
| **ChatterBox 23L** | Main | v1, v2, v3, Vietnamese (Viterbox), Egyptian Arabic (oddadmix) | ~4.3GB | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | MIT | V1, V2, and V3 official checkpoints, Emotion tokens (v2; currently ineffective), V3 skips the legacy alignment analyzer and trims the final token artifact | 25 |
| **VibeVoice** | Main | 1.5B, 7B, KugelAudio-0 (7B), kugel-2 (7B), Hindi-1.5B/7B | 5.4GB / 18GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | MIT (research-only per model card) | 90-min long-form, Native 4-speaker (Base models), Multilingual (KugelAudio variants), 4-bit quantization | 27 |
| **VibeVoice** | Shared | 1.5B, 7B, KugelAudio-0 (7B), kugel-2 (7B), Hindi-1.5B/7B | 5.4GB / 18GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | MIT (research-only per model card) | 90-min long-form, Native 4-speaker (Base models), Multilingual (KugelAudio variants), 4-bit quantization | 27 |
| **Higgs Audio 2** | Shared | 3B | ~9GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Boson Higgs Audio 2 Community License | 3 multi-speaker, CUDA graphs (55+ tokens/sec) | 5 |
| **Higgs Audio v3** | Main | 4B | ~8GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Boson Higgs Audio v3 Research and Non-Commercial License | Native inline emotion/style/prosody/SFX tags, Zero-shot voice cloning, 100+ language support | 100+ |
| **IndexTTS 2 / 2.5** | Main | IndexTTS-2, IndexTTS-2.5 | ~4.7GB / ~5.49GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | bilibili Model Use License | Emotion Control: 8 vectors, Text as reference, Audio as reference, IndexTTS-2.5 official internal feature-duration scaling (not prosody planning), IndexTTS-2.5 pronunciation annotations | 5 |
| **CosyVoice3** | Main | 0.5B, 0.5B-RL | ~5.4GB | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | Apache-2.0 | Paralinguistic tags | 4 |
| **Qwen3-TTS** | Shared | 0.6B, 1.7B (CustomVoice/VoiceDesign/Base) | ~3-6GB | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | Apache-2.0 | Voice design, ASR (Automatic Speech Recognition) | 10 |
| **Granite ASR** | Main | granite-4.0-1b-speech, granite-speech-4.1-2b, granite-speech-4.1-2b-plus | ~4.6GB | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | Apache-2.0 | Native speaker attribution / diarization (plus model variant), Native word-level timestamps (plus model variant), ASR (Automatic Speech Recognition), Custom timestamps/SRT via reused Qwen forced aligner, Speech translation (experimental), Optional forced aligner auto-routed through shared legacy T4 runtime | 6 |
| **Step Audio EditX** | Shared | 3B LLM + CosyVoice | ~7GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 (verify before commercial use) | Second Pass Speech Editing Node: 14 emotions, 32 speaking styles, Dated checkpoints: 2026-01-23 / 2025-11-28, Expanded January paralinguistic effects with legacy tag compatibility, Selectable main or shared Python runtime (shared Transformers 4 runtime recommended) | 4 |
| **Step Audio EditX** | Main | 3B LLM + CosyVoice | ~7GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 (verify before commercial use) | Second Pass Speech Editing Node: 14 emotions, 32 speaking styles, Paralinguistic effects, Selectable main, shared, or dedicated Python runtime (shared Transformers 4 runtime recommended) | 4 |
| **Echo-TTS** | Main | echo-tts-base + fish-s1-dac-min | ~5.3GB + ~1.8GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | CC-BY-NC-SA-4.0 | Diffusion-based (~30s best), Force Speaker KV (speaker drift control) | 1 |
| **Fish Audio S2 Pro** | Main | S2 Pro 4B / FP8 | ~10.3GB / ~8.0GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Fish Audio Research License | Free-form sub-word emotion/prosody tags, Native multi-speaker and multi-turn dialogue with dynamic speaker references, Zero-shot voice cloning, Optional per-segment custom character switching, Configurable 4K-32K native context with reduced KV-cache VRAM, Optional community FP8 weight-only checkpoint with BF16 activations, Optional on-the-fly BitsAndBytes INT8/NF4 for the official checkpoint | 80+ languages |
| **Dots TTS** | Main | dots.tts-base, dots.tts-soar, dots.tts-mf | ~6GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 | Official auto language detect / language control, SOAR and MeanFlow distilled variants | 19 |
@@ -24,4 +24,4 @@
| **MOSS-SoundEffect v2** | Main | MOSS-SoundEffect-v2.0 | ~11.2GB | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | Apache-2.0 | Durations up to 30 seconds, Native negative prompting, CFG, flow shift, and diffusion-step controls, Prompt-only text-to-sound generation, 48 kHz mono output, Seeded generation | 2 |
| **RVC** | Main | Community .pth | 100-300MB | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | MIT (framework); community models vary | Real-time VC, Integrated training workflow, Pitch shift (±14), 6 HuBERT models, Language-independent | Any |
*Isolation column: `Main` runs in the main ComfyUI environment. `Shared` uses a shared secondary runtime reused by multiple engines.*
*Isolation column: `Main` runs in the main ComfyUI environment. `Shared` uses a shared secondary runtime reused by multiple engines. `Dedicated` uses an engine-specific secondary runtime.*
+2 -21
View File
@@ -11,8 +11,6 @@ The multiline TTS tag editor now has an engine-aware `Inline Tags` panel with se
Use the Step mode when you want post-process editing tags. Use the Higgs or Cosy modes when you want those engines' native generation tags.
The existing **model_path** dropdown now offers `Step-Audio-EditX-2026-01-23` and `Step-Audio-EditX-2025-11-28`. The January checkpoint is the default for new engine nodes and supports the expanded sound vocabulary below. Existing unversioned and local selections keep their files. See [checkpoint selection and compatibility](STEP_AUDIO_EDITX_CHECKPOINTS.md).
## 💡 Inline vs Manual Node Workflow
**ComfyUI is modular** - you can manually chain **TTS → 🎨 Step Audio EditX - Audio Editor** nodes for full control.
@@ -61,19 +59,6 @@ Insert non-verbal sounds at specific positions in the audio:
| `<Question-ei>` | Question sound | `<Question-ei> Really?` |
| `<Dissatisfaction-hnn>` | Dissatisfied "hnn" | `<Dissatisfaction-hnn> Not good` |
The **2026-01-23** checkpoint additionally supports:
| Tag | Effect |
|---|---|
| `<inhale>`, `<exhale>`, `<breath>` | Breathing sounds |
| `<laugh>`, `<chuckle>`, `<giggle>` | Laughter variants |
| `<clears throat>` or `<clears_throat>` | Throat clearing |
| `<snort>`, `<cough>` | Snorting and coughing |
| `<Surprise-yo>` | Additional surprise vocalization |
| `<Question-ah>`, `<Question-en>`, `<Question-yi>`, `<Question-oh>` | Additional questioning vocalizations |
These support the same iteration and pipe syntax: `<giggle:2>` and `<clears_throat:1|style:serious>`. Tags are case-insensitive. For ChatterBox v2/v3, specify an iteration for Step effects (`<giggle:1>`); bare overlapping tags retain their native meaning.
**Position matters:** The sound is inserted where you place the tag.
**💡 Pro Tip - Stronger Effects:** For more reliable and pronounced effects, include relevant text alongside the tag:
@@ -93,7 +78,7 @@ Apply emotional tone to the entire audio segment:
`<emotion:VALUE>` or `<emotion:VALUE:ITERATIONS>`
**Available emotions:** happy, sad, angry, excited, fear, humour, surprised, disgusted, confusion, empathy, embarrass, depressed, coldness, admiration. Older suite options `calm` and `fearful` remain accepted.
**Available emotions:** happy, sad, angry, excited, calm, fearful, surprised, disgusted, confusion, empathy, embarrass, depressed, coldness, admiration
**Examples:**
```
@@ -638,13 +623,9 @@ Choose appropriate effects for the situation:
### All Paralinguistic Tags
`<Breathing>`, `<Laughter>`, `<Sigh>`, `<Uhm>`, `<Surprise-oh>`, `<Surprise-ah>`, `<Surprise-wa>`, `<Confirmation-en>`, `<Question-ei>`, `<Dissatisfaction-hnn>`
January checkpoint vocabulary: `<inhale>`, `<exhale>`, `<laugh>`, `<chuckle>`, `<clears throat>` (alias `<clears_throat>`), `<snort>`, `<giggle>`, `<cough>`, `<breath>`, `<Surprise-yo>`, `<Question-ah>`, `<Question-en>`, `<Question-yi>`, `<Question-oh>`.
### All Emotions
`<emotion:happy>`, `<emotion:sad>`, `<emotion:angry>`, `<emotion:excited>`, `<emotion:calm>`, `<emotion:fearful>`, `<emotion:surprised>`, `<emotion:disgusted>`, `<emotion:confusion>`, `<emotion:empathy>`, `<emotion:embarrass>`, `<emotion:depressed>`, `<emotion:coldness>`, `<emotion:admiration>`
Also available: `<emotion:fear>` and `<emotion:humour>`.
### All Styles
`<style:whisper>`, `<style:serious>`, `<style:child>`, `<style:older>`, `<style:girl>`, `<style:pure>`, `<style:sister>`, `<style:sweet>`, `<style:exaggerated>`, `<style:ethereal>`, `<style:generous>`, `<style:recite>`, `<style:act_coy>`, `<style:warm>`, `<style:shy>`, `<style:comfort>`, `<style:authority>`, `<style:chat>`, `<style:radio>`, `<style:soulful>`, `<style:gentle>`, `<style:story>`, `<style:vivid>`, `<style:program>`, `<style:news>`, `<style:advertising>`, `<style:roar>`, `<style:murmur>`, `<style:shout>`, `<style:deeply>`, `<style:loudly>`, `<style:arrogant>`, `<style:friendly>`
@@ -668,7 +649,7 @@ A: Slightly. The TTS generates normally, then Step Audio EditX loads once and pr
A: Yes! Batch processing is automatically applied - all subtitles are generated first, then all edits are applied at once.
**Q: What if I make a typo in a tag?**
A: The Step parser leaves unknown tags in the text and does not create an edit for them. Correct the spelling before generation; the TTS engine may interpret or speak the remaining text.
A: Unknown tags are ignored and stripped from the text. Check console for warnings.
**Q: Can I nest tags?**
A: No. Use pipe syntax or separate tags: `<tag1|tag2>` or `<tag1><tag2>`
+1 -2
View File
@@ -105,8 +105,7 @@ Use this as the canonical list of model repositories/links for offline setup.
| Component | Source | Size | Auto-Download | Notes |
|---|---|---|---|---|
| Step-Audio-EditX-2026-01-23 | [stepfun-ai/Step-Audio-EditX](https://huggingface.co/stepfun-ai/Step-Audio-EditX) | ~7GB | ✅ | Default 3B checkpoint; pinned to 5fe2f8a05c2353301ad47d3c1747b262115da138 |
| Step-Audio-EditX-2025-11-28 | [stepfun-ai/Step-Audio-EditX](https://huggingface.co/stepfun-ai/Step-Audio-EditX/tree/7f3de603ae46c96dff6f06f47b1d5a45aabd34fe) | ~7GB | ✅ | Legacy checkpoint; pinned November weights in a separate folder |
| Step-Audio-EditX | [stepfun-ai/Step-Audio-EditX](https://huggingface.co/stepfun-ai/Step-Audio-EditX) | ~7GB | ✅ | Main 3B audio editing model |
| Step-Audio-Tokenizer | [stepfun-ai/Step-Audio-Tokenizer](https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer) | Included | ✅ | Tokenizer bundle used by Step EditX |
## Echo-TTS
+2 -8
View File
@@ -194,20 +194,14 @@ Notes:
```text
ComfyUI/models/TTS/step_audio_editx/
├── Step-Audio-EditX-2026-01-23/
│ └── CosyVoice-300M-25Hz/
├── Step-Audio-EditX-2025-11-28/ # downloaded only when selected
│ └── CosyVoice-300M-25Hz/
├── Step-Audio-EditX/ # existing unversioned installations
├── Step-Audio-EditX/
│ └── CosyVoice-300M-25Hz/
└── FunASR-Paraformer/
```
Notes:
- Selected checkpoints and tokenizer assets auto-download at pinned revisions.
- Existing unversioned/local weights are retained and folders are not relabeled; known config compatibility fields may be normalized.
- See [Step Audio EditX checkpoints](STEP_AUDIO_EDITX_CHECKPOINTS.md) for selection and expanded tags.
- Main model, tokenizer assets, and speech stack auto-download.
## Higgs Audio v3
-9
View File
@@ -63,9 +63,6 @@ Step Audio EditX examples:
```text
<Laughter>
<giggle>
<clears throat:2>
<Question-oh>
<emotion:happy>
<style:whisper:2>
<restore:1@2>
@@ -96,12 +93,6 @@ OmniVoice examples:
Use the dedicated inline tag controls in the sidebar when you do not want to type these by hand.
The Step sound menu includes the original sounds plus the expanded sounds for the `2026-01-23` checkpoint: `inhale`, `exhale`, `laugh`, `chuckle`, `clears throat`, `snort`, `giggle`, `cough`, `breath`, `Surprise-yo`, `Question-ah`, `Question-en`, `Question-yi`, and `Question-oh`. Select `Step-Audio-EditX-2026-01-23` in the engine node to use these sounds. The `2025-11-28` checkpoint supports the original vocabulary; see the [checkpoint guide](STEP_AUDIO_EDITX_CHECKPOINTS.md).
Validation, highlighting, and the tag-swap menu recognize these sounds. Names are case-insensitive, and `<clears_throat>` is accepted as an alias for `<clears throat>`. The insertion controls preserve the existing iteration syntax, such as `<giggle:2>`, and pipe combinations, such as `<giggle:2|emotion:happy>`. The emotion menu also includes `fear` and `humour`.
The editor checks syntax for the chosen tag engine but does not inspect the engine node's selected model checkpoint. A valid new sound still requires the January weights.
Important differences:
- `Step Audio EditX` tags are post-process controls
@@ -93,7 +93,7 @@ Manual tests:
Runtime plan:
- Main Environment only
- or Shared Runtime if dependency conflicts are expected
- or Shared Runtime / Dedicated Runtime if dependency conflicts are expected
```
If the scope is vague, do not code yet.
-51
View File
@@ -1,51 +0,0 @@
# Step Audio EditX checkpoints
Choose the checkpoint in the existing **model_path** dropdown of **⚙️ Step Audio EditX Engine**. No widget was added or reordered, so saved device, precision, quantization, generation, and runtime values keep their positions.
| Model choice | Checkpoint | Download revision |
|---|---|---|
| `Step-Audio-EditX-2026-01-23` | Updated weights and expanded sound tags; default for new engine nodes | `5fe2f8a05c2353301ad47d3c1747b262115da138` |
| `Step-Audio-EditX-2025-11-28` | Legacy weights | `7f3de603ae46c96dff6f06f47b1d5a45aabd34fe` |
| `Step-Audio-EditX` | Compatibility choice for an existing unversioned installation | Existing files are retained; a missing installation downloads the January checkpoint |
| `local:...` | Explicitly selected local model | Existing files are retained |
Dates identify the weight uploads. The January release was announced on January 29; its pinned revision also includes subsequent configuration fixes. These are versions of the same 3B model architecture, not official “v1/v2” names.
Sources: [official model](https://huggingface.co/stepfun-ai/Step-Audio-EditX), [November snapshot](https://huggingface.co/stepfun-ai/Step-Audio-EditX/tree/7f3de603ae46c96dff6f06f47b1d5a45aabd34fe), [upstream release notes](https://github.com/stepfun-ai/Step-Audio-EditX#readme).
## Existing workflows and storage
- Existing `Step-Audio-EditX` and `local:...` selections remain valid. The suite does not replace their existing weights, rename their folders, or infer their checkpoint. The existing compatibility normalization of `config.json` still applies.
- Dated models download lazily into separate folders under `ComfyUI/models/TTS/step_audio_editx/`, respecting `extra_model_paths.yaml`.
- Selecting both checkpoints stores both sets of model files. The second LLM checkpoint requires approximately another 7 GB.
- Model and edit-result caches distinguish the selected checkpoint.
- The shared Transformers 4 runtime remains recommended. Selecting the updated standard checkpoint does not switch to vLLM or to the separate upstream AWQ checkpoint. The suite's `int4` option remains bitsandbytes NF4.
## Sound tags
Use the **2026-01-23** checkpoint for expanded sound tags. The November checkpoint is available to preserve older behavior; selecting it does not add the new learned sounds.
Existing spellings remain supported:
`<Breathing>`, `<Laughter>`, `<Sigh>`, `<Uhm>`, `<Surprise-oh>`, `<Surprise-ah>`, `<Surprise-wa>`, `<Confirmation-en>`, `<Question-ei>`, `<Dissatisfaction-hnn>`.
The January vocabulary also exposes:
`<inhale>`, `<exhale>`, `<laugh>`, `<chuckle>`, `<clears throat>`, `<snort>`, `<giggle>`, `<cough>`, `<breath>`, `<Surprise-yo>`, `<Question-ah>`, `<Question-en>`, `<Question-yi>`, `<Question-oh>`.
Input is case-insensitive. `<clears_throat>` is an alias for `<clears throat>`; both become `[clears throat]` in the model instruction. Expanded tags support the existing iteration and pipe syntax, for example:
```text
[Alice] That was funny <giggle:2>.
[Bob] Let me explain <clears_throat:1|style:serious>.
```
For ChatterBox v2/v3, use an explicit iteration to request Step editing: `<giggle:1>`. Bare `<giggle>`, `<inhale>`, `<exhale>`, and `<cough>` retain their ChatterBox-native meaning. CosyVoice's native sound tags remain native as well.
## Audio Editor and other engines
Connect the dated **Step Audio EditX Engine** output to **🎨 Step Audio EditX - Audio Editor** to choose a checkpoint explicitly. In the Audio Editor's transcript, use bare sound tags such as `<giggle>` or `<clears throat>`; its **n_edit_iterations** input controls the number of passes.
Without a connected engine, the Audio Editor prefers an installed January checkpoint, then an existing unversioned installation. If neither exists, it downloads the dated January checkpoint. This same fallback applies to automatic Step editing of other engines' audio. An existing unversioned installation may contain older weights; use the manual Audio Editor with a dated engine for explicit control.
See [the inline tag guide](INLINE_EDIT_TAGS_USER_GUIDE.md) for position, iteration, restoration, language, and duration behavior.
+2 -2
View File
@@ -56,7 +56,7 @@ python -c "import triton; print(triton.__version__)"
**Main environment**: no special configuration beyond a working compiler toolchain.
**Shared Runtime isolation**:
**Shared / Dedicated Runtime isolation**:
- TTS Audio Suite now tries to detect Visual Studio automatically and inject a proper MSVC build environment into the isolated worker
- This is meant to preserve `torch.compile` parity with old direct Transformers 4 environments
- It still depends on the Build Tools actually being installed
@@ -112,7 +112,7 @@ pip install --upgrade torchvision --index-url https://download.pytorch.org/whl/c
**Solution**:
1. Install or repair Visual Studio Build Tools with the C++ workload
2. Restart ComfyUI so the new environment is visible
3. If using Shared Runtime, let TTS Audio Suite auto-detect the toolchain
3. If using Shared/Dedicated Runtime, let TTS Audio Suite auto-detect the toolchain
If detection still fails, your machine state is broken, not just the Python package stack.
+486
View File
@@ -0,0 +1,486 @@
"""audio.cpp adapter for the Suite's unified ASR pipeline."""
from __future__ import annotations
import json
import os
import re
import time
from typing import Any, Dict, Iterable, Mapping, Optional
import torch
from utils.asr.types import ASRRequest, ASRResult, ASRSegment, ASRWord
from utils.audio.processing import AudioProcessingUtils
_NATIVE_CHUNK_FAMILIES = {
"fun_asr_nano",
"higgs_audio_stt",
"hviske_asr",
"qwen3_asr",
"vibevoice_asr",
"voxtral_realtime",
}
# audio.cpp release-0.5.1 keeps stale offline decoder state for these loaders:
# the first request transcribes normally and later requests return empty text.
# A fresh owned process is currently the only reliable reset contract.
_RESTART_BETWEEN_CHUNKS_FAMILIES = {"nemotron_asr", "voxtral_realtime"}
def _session(config: Mapping[str, Any]):
from utils.audio_cpp.session import get_audio_cpp_session
return get_audio_cpp_session(dict(config))
def _advanced_options(config: Mapping[str, Any]) -> Dict[str, Any]:
value = config.get("advanced_options", config.get("request_options", {}))
if value in (None, ""):
return {}
if isinstance(value, str):
try:
value = json.loads(value)
except json.JSONDecodeError as exc:
raise ValueError(f"Invalid audio.cpp advanced JSON: {exc.msg}") from exc
if not isinstance(value, Mapping):
raise ValueError("audio.cpp advanced options must be a JSON object")
return dict(value)
def _audio_path(audio: Mapping[str, Any]) -> str:
waveform = audio.get("waveform")
sample_rate = audio.get("sample_rate")
if not torch.is_tensor(waveform):
raise TypeError("audio.cpp ASR input must contain a waveform tensor")
if sample_rate is None or int(sample_rate) <= 0:
raise ValueError("audio.cpp ASR input must contain a positive sample_rate")
return os.path.abspath(
AudioProcessingUtils.save_audio_to_temp_file(waveform, int(sample_rate))
)
def _waveform_3d(audio: Mapping[str, Any]) -> tuple[torch.Tensor, int]:
waveform = audio.get("waveform")
sample_rate = int(audio.get("sample_rate") or 0)
if not torch.is_tensor(waveform):
raise TypeError("audio.cpp ASR input must contain a waveform tensor")
if sample_rate <= 0:
raise ValueError("audio.cpp ASR input must contain a positive sample_rate")
if waveform.ndim == 1:
waveform = waveform.unsqueeze(0).unsqueeze(0)
elif waveform.ndim == 2:
waveform = waveform.unsqueeze(0)
elif waveform.ndim != 3:
raise ValueError(
"audio.cpp ASR waveform must have [samples], [channels, samples], or "
"[batch, channels, samples] shape"
)
if waveform.shape[0] != 1:
raise ValueError("audio.cpp ASR accepts one audio item at a time")
if waveform.shape[-1] <= 0:
raise ValueError("audio.cpp ASR input audio is empty")
return waveform.detach().cpu(), sample_rate
def _chunk_ranges(
total_samples: int,
sample_rate: int,
chunk_size: int,
overlap: int,
) -> list[tuple[int, int]]:
if chunk_size <= 0:
return [(0, total_samples)]
if overlap < 0:
raise ValueError("ASR overlap must be zero or greater")
if overlap >= chunk_size:
raise ValueError("ASR overlap must be smaller than chunk_size")
chunk_samples = chunk_size * sample_rate
if total_samples <= chunk_samples:
return [(0, total_samples)]
step_samples = (chunk_size - overlap) * sample_rate
ranges = []
start = 0
while start < total_samples:
end = min(start + chunk_samples, total_samples)
ranges.append((start, end))
if end >= total_samples:
break
start += step_samples
return ranges
def _normalized_token(value: str) -> str:
return re.sub(r"[^\w]+", "", value, flags=re.UNICODE).casefold()
def _merge_transcript(parts: Iterable[str]) -> str:
merged: list[str] = []
for part in parts:
incoming = str(part or "").strip().split()
if not incoming:
continue
if not merged:
merged.extend(incoming)
continue
limit = min(len(merged), len(incoming), 80)
duplicate_count = 0
for size in range(limit, 0, -1):
left = [_normalized_token(token) for token in merged[-size:]]
right = [_normalized_token(token) for token in incoming[:size]]
if all(left) and left == right:
duplicate_count = size
break
merged.extend(incoming[duplicate_count:])
return " ".join(merged).strip()
def _offset_words(
words: Iterable[ASRWord], offset: float, unique_after: Optional[float]
) -> list[ASRWord]:
shifted = []
for word in words:
item = ASRWord(start=word.start + offset, end=word.end + offset, text=word.text)
if unique_after is not None and (item.start + item.end) / 2.0 < unique_after:
continue
shifted.append(item)
return shifted
def _offset_segments(
segments: Iterable[ASRSegment], offset: float, unique_after: Optional[float]
) -> list[ASRSegment]:
shifted = []
for segment in segments:
item = ASRSegment(
start=segment.start + offset,
end=segment.end + offset,
text=segment.text,
speaker=segment.speaker,
)
if unique_after is not None and (item.start + item.end) / 2.0 < unique_after:
continue
shifted.append(item)
return shifted
def _seconds(value: Any, sample_rate: int) -> float:
try:
return max(0.0, float(value) / float(sample_rate))
except (TypeError, ValueError, ZeroDivisionError):
return 0.0
def _words(payload: Mapping[str, Any], sample_rate: int) -> list[ASRWord]:
words = []
for item in payload.get("words") or []:
if not isinstance(item, Mapping):
continue
text = str(item.get("word", item.get("text", ""))).strip()
if not text:
continue
words.append(
ASRWord(
start=_seconds(item.get("start_sample"), sample_rate),
end=_seconds(item.get("end_sample"), sample_rate),
text=text,
)
)
return words
def _plain_segments(payload: Mapping[str, Any], sample_rate: int) -> list[ASRSegment]:
segments = []
for item in payload.get("segments") or []:
if not isinstance(item, Mapping):
continue
text = str(item.get("text", "")).strip()
segments.append(
ASRSegment(
start=_seconds(item.get("start_sample"), sample_rate),
end=_seconds(item.get("end_sample"), sample_rate),
text=text,
)
)
return segments
def _speaker_segments(payload: Mapping[str, Any], sample_rate: int) -> list[ASRSegment]:
segments = []
for item in payload.get("speaker_turns") or []:
if not isinstance(item, Mapping):
continue
speaker = str(item.get("speaker_id", "")).strip()
if speaker and not speaker.lower().startswith("speaker"):
speaker = f"Speaker {speaker}"
segments.append(
ASRSegment(
start=_seconds(item.get("start_sample"), sample_rate),
end=_seconds(item.get("end_sample"), sample_rate),
text=str(item.get("text", "")).strip(),
speaker=speaker or None,
)
)
return segments
def _attach_words(segments: Iterable[ASRSegment], words: Iterable[ASRWord]) -> None:
segment_list = list(segments)
for word in words:
midpoint = (word.start + word.end) / 2.0
target = next(
(segment for segment in segment_list if segment.start <= midpoint <= segment.end),
None,
)
if target is not None:
target.words.append(word)
class AudioCppASREngineAdapter:
"""Normalize audio.cpp transcript/timing output into ``ASRResult``."""
def __init__(self, engine_data: Dict[str, Any]):
self.engine_data = dict(engine_data)
self.config = dict(engine_data.get("config", engine_data))
def _session_config(self) -> Dict[str, Any]:
config = dict(self.config)
if str(config.get("connection_mode", "auto")).lower() != "external_server":
config["requested_task"] = "asr"
config["task"] = "asr"
return config
def transcribe(self, req: ASRRequest) -> ASRResult:
if req.task != "transcribe":
raise ValueError(
"audio.cpp release-0.5.1 ASR loaders support transcription, not the "
"Unified ASR translate mode"
)
config = self._session_config()
family = str(config.get("family", "")).strip()
warnings: list[str] = []
notes: list[str] = []
options = _advanced_options(config)
# VibeVoice-ASR owns diarization across its full recording. Independent
# Suite requests can restart speaker numbering, so preserve its native
# chunking only for this mode. All other ASR uses Suite-side windows.
native_diarization = (
family == "vibevoice_asr" and req.diarization and req.chunk_size > 0
)
if native_diarization:
options.setdefault("audio_chunk_mode", "fixed")
options.setdefault("audio_chunk_seconds", int(req.chunk_size))
if req.overlap > 0:
notes.append(
"VibeVoice-ASR diarization uses native chunking to preserve speaker "
"identity; the Suite overlap setting is not applied."
)
elif family in _NATIVE_CHUNK_FAMILIES:
options.setdefault("audio_chunk_mode", "none")
if req.timestamps == "word" and family == "qwen3_asr":
session_options = config.get("session_options") or {}
aligner = session_options.get("qwen3_asr.forced_aligner_model_path")
if aligner:
options["return_timestamps"] = True
else:
warnings.append(
"Qwen3-ASR word timestamps require the optional Qwen3 Forced Aligner; "
"transcription continued without downloading that auxiliary model."
)
waveform, source_rate = _waveform_3d(req.audio)
ranges = (
[(0, waveform.shape[-1])]
if native_diarization
else _chunk_ranges(
waveform.shape[-1], source_rate, int(req.chunk_size), int(req.overlap)
)
)
session = _session(config)
if str(getattr(session, "task", "asr")) != "asr":
raise ValueError(
f"audio.cpp model '{session.model_id}' is configured for task "
f"'{session.task}', not ASR"
)
restart_between_chunks = (
len(ranges) > 1 and family in _RESTART_BETWEEN_CHUNKS_FAMILIES
)
if restart_between_chunks and not bool(getattr(session, "owned", False)):
raise RuntimeError(
f"audio.cpp release-0.5.1 {family} returns empty text after its first "
"offline request. Suite-side chunking therefore requires a managed "
"audio.cpp server so the Suite can reset it between chunks. Set "
"connection_mode to managed, or set ASR chunk_size to 0 when using "
"an external server."
)
if restart_between_chunks:
notes.append(
f"audio.cpp release-0.5.1 {family} requires a managed server reset "
"between Suite chunks to avoid empty repeated-request results."
)
display_family = family or "external model"
print(f"🎧 audio.cpp ASR: Transcribing with {display_family}...")
if len(ranges) > 1:
notes.append(
f"Suite-side ASR chunking used {len(ranges)} windows of "
f"{int(req.chunk_size)}s with {int(req.overlap)}s overlap."
)
print(
f"🧩 audio.cpp ASR: {len(ranges)} chunks "
f"({int(req.chunk_size)}s, {int(req.overlap)}s overlap)"
)
payloads: list[Mapping[str, Any]] = []
chunk_timings: list[Mapping[str, Any]] = []
chunk_diagnostics: list[Dict[str, Any]] = []
started_at = time.time()
for index, (start, end) in enumerate(ranges, start=1):
if index > 1 and restart_between_chunks:
print(
f"🔄 audio.cpp ASR: Resetting {family} session for chunk "
f"{index}/{len(ranges)}"
)
session.restart_owned_runtime()
chunk_waveform = waveform[..., start:end]
chunk_rms = float(torch.sqrt(torch.mean(chunk_waveform.float().square())).item())
chunk_peak = float(chunk_waveform.float().abs().max().item())
temp_path = _audio_path({
"waveform": chunk_waveform,
"sample_rate": source_rate,
})
try:
request: Dict[str, Any] = {"audio": temp_path, "options": dict(options)}
if req.language:
request["language"] = req.language
result = session.run(request)
payload = result.raw if isinstance(result.raw, Mapping) else {}
payloads.append(payload)
if isinstance(payload.get("timing"), Mapping):
chunk_timings.append(payload["timing"])
chunk_diagnostics.append({
"index": index,
"start": round(start / source_rate, 3),
"end": round(end / source_rate, 3),
"rms": round(chunk_rms, 6),
"peak": round(chunk_peak, 6),
"text": str(payload.get("text", "")).strip(),
"characters": len(str(payload.get("text", "")).strip()),
"upstream_timing": (
dict(payload["timing"])
if isinstance(payload.get("timing"), Mapping)
else None
),
})
finally:
try:
os.remove(temp_path)
except FileNotFoundError:
pass
if len(ranges) > 1:
chunk_chars = len(str(payload.get("text", "")).strip())
print(
f" ASR chunk {index}/{len(ranges)} complete "
f"({chunk_chars} chars, RMS {chunk_rms:.4f}, peak {chunk_peak:.4f})"
)
words: list[ASRWord] = []
speaker_segments: list[ASRSegment] = []
plain_segments: list[ASRSegment] = []
overlap_seconds = float(req.overlap) if len(ranges) > 1 else 0.0
for index, ((start, _end), payload) in enumerate(zip(ranges, payloads)):
offset = start / source_rate
unique_after = offset + overlap_seconds if index > 0 else None
words.extend(_offset_words(_words(payload, source_rate), offset, unique_after))
speaker_segments.extend(
_offset_segments(
_speaker_segments(payload, source_rate), offset, unique_after
)
)
plain_segments.extend(
_offset_segments(_plain_segments(payload, source_rate), offset, unique_after)
)
if req.diarization:
segments = speaker_segments
if segments:
_attach_words(segments, words)
else:
warnings.append(
f"audio.cpp {family or 'ASR model'} returned no speaker-attributed turns."
)
segments = plain_segments
elif req.timestamps == "word" and words:
segments = [
ASRSegment(start=word.start, end=word.end, text=word.text, words=[word])
for word in words
]
elif req.timestamps == "word":
segments = plain_segments
else:
segments = []
text = _merge_transcript(payload.get("text", "") for payload in payloads)
if req.diarization and speaker_segments:
text = " ".join(
f"[{segment.speaker}] {segment.text}" if segment.speaker else segment.text
for segment in speaker_segments
if segment.text
).strip()
if not text and speaker_segments:
text = " ".join(segment.text for segment in speaker_segments if segment.text).strip()
if req.timestamps == "word" and not words:
warnings.append(f"audio.cpp {family or 'ASR model'} returned no word timestamps.")
empty_chunks = sum(
1 for payload in payloads if not str(payload.get("text", "")).strip()
)
if len(payloads) > 1 and empty_chunks:
warnings.append(
f"audio.cpp {family or 'ASR model'} returned no text for "
f"{empty_chunks} of {len(payloads)} Suite chunks."
)
raw: Dict[str, Any] = {}
if warnings:
raw["warnings"] = warnings
if notes:
raw["notes"] = notes
if len(payloads) == 1 and chunk_timings:
raw["timing"] = dict(chunk_timings[0])
elif len(payloads) > 1:
raw["timing"] = {
"wall_ms": round((time.time() - started_at) * 1000.0, 3),
"suite_chunks": len(payloads),
"suite_chunk_size_seconds": int(req.chunk_size),
"suite_overlap_seconds": int(req.overlap),
"upstream_wall_ms": round(
sum(float(item.get("wall_ms", 0.0)) for item in chunk_timings), 3
),
}
raw["chunks"] = chunk_diagnostics
output_language = next(
(
str(payload.get("language", "")).strip()
for payload in payloads
if str(payload.get("language", "")).strip()
),
str(req.language or "").strip(),
) or None
print(
f"✅ audio.cpp ASR: Complete ({len(text)} chars, "
f"{len(segments)} timed/speaker segments)"
)
return ASRResult(
text=text,
language=output_language,
segments=segments,
raw=raw or None,
)
__all__ = ["AudioCppASREngineAdapter"]
+372
View File
@@ -0,0 +1,372 @@
"""Adapter between the suite's TTS processors and an audio.cpp session."""
from __future__ import annotations
import json
import os
import threading
from typing import Any, Dict, Mapping, Optional, Tuple
import torch
from utils.audio.audio_hash import generate_stable_audio_component
from utils.audio.cache import get_audio_cache
from utils.audio.processing import AudioProcessingUtils
from utils.voice.reference import effective_voice_audio
_CACHE_SAMPLE_RATES: Dict[str, int] = {}
_CACHE_SAMPLE_RATES_LOCK = threading.Lock()
def _get_session(config: Mapping[str, Any]):
"""Import lazily so the node can still be discovered before optional setup."""
from utils.audio_cpp.session import get_audio_cpp_session
return get_audio_cpp_session(dict(config))
def _canonical_json(value: Mapping[str, Any]) -> str:
return json.dumps(value, sort_keys=True, separators=(",", ":"), default=str)
class AudioCppEngineAdapter:
"""Build generic ``/v1/tasks/run`` requests and retain their real sample rate."""
_COMMON_REQUEST_FIELDS = (
"temperature",
"top_p",
"top_k",
"repetition_penalty",
"max_tokens",
"max_steps",
"num_inference_steps",
"guidance_scale",
"speaking_rate",
)
def __init__(self, config: Optional[Dict[str, Any]] = None):
self.config = dict(config or {})
self.audio_cache = get_audio_cache()
self._last_sample_rate: Optional[int] = None
self._reference_files: Dict[str, str] = {}
self._reference_lock = threading.RLock()
@property
def sample_rate(self) -> Optional[int]:
return self._last_sample_rate
def update_config(self, new_config: Optional[Dict[str, Any]]) -> None:
self.config = dict(new_config or {})
@staticmethod
def _reference_text(voice_ref: Any) -> str:
if not isinstance(voice_ref, Mapping):
return ""
return str(
voice_ref.get("reference_text")
or voice_ref.get("prompt_text")
or voice_ref.get("text")
or ""
).strip()
def _materialize_reference(self, voice_ref: Any) -> Tuple[Optional[str], str, str, Optional[str]]:
"""Return path, transcript, stable hash, and the path that must be removed."""
reference_text = AudioCppEngineAdapter._reference_text(voice_ref)
if not isinstance(voice_ref, Mapping):
return None, reference_text, "default_voice", None
audio = effective_voice_audio(voice_ref)
if audio is None:
return None, reference_text, "default_voice", None
if isinstance(audio, (str, os.PathLike)):
path = os.path.abspath(os.path.expanduser(os.fspath(audio)))
if not os.path.isfile(path):
raise FileNotFoundError(f"audio.cpp reference audio not found: {path}")
component = generate_stable_audio_component(audio_file_path=path)
return path, reference_text, component, None
if isinstance(audio, Mapping):
waveform = audio.get("waveform")
sample_rate = audio.get("sample_rate")
audio_dict = dict(audio)
elif torch.is_tensor(audio):
waveform = audio
sample_rate = voice_ref.get("sample_rate")
audio_dict = {"waveform": waveform, "sample_rate": sample_rate}
else:
raise TypeError(f"Unsupported audio.cpp voice reference type: {type(audio).__name__}")
if not torch.is_tensor(waveform):
raise TypeError("audio.cpp reference audio must contain a waveform tensor")
if sample_rate is None or int(sample_rate) <= 0:
raise ValueError("audio.cpp reference audio must contain a positive sample_rate")
audio_dict["sample_rate"] = int(sample_rate)
component = generate_stable_audio_component(reference_audio=audio_dict)
if component not in {"ref_audio_error", "ref_audio_error_not_tensor"}:
with self._reference_lock:
cached_path = self._reference_files.get(component)
if cached_path and os.path.isfile(cached_path):
return cached_path, reference_text, component, None
temp_path = os.path.abspath(
AudioProcessingUtils.save_audio_to_temp_file(waveform, int(sample_rate))
)
self._reference_files[component] = temp_path
return temp_path, reference_text, component, None
# Hash failures must not make unrelated references share one file.
temp_path = os.path.abspath(
AudioProcessingUtils.save_audio_to_temp_file(waveform, int(sample_rate))
)
return temp_path, reference_text, component, temp_path
def close(self) -> None:
with self._reference_lock:
paths = list(self._reference_files.values())
self._reference_files.clear()
for path in paths:
try:
os.remove(path)
except FileNotFoundError:
pass
except OSError:
pass
def __del__(self):
try:
self.close()
except Exception:
pass
def _advanced_options(self) -> Dict[str, Any]:
value = self.config.get(
"advanced_options",
self.config.get("request_options", self.config.get("advanced_json", {})),
)
if value in (None, ""):
return {}
if isinstance(value, str):
try:
value = json.loads(value)
except json.JSONDecodeError as exc:
raise ValueError(f"Invalid audio.cpp advanced JSON: {exc.msg}") from exc
if not isinstance(value, Mapping):
raise ValueError("audio.cpp advanced options must be a JSON object")
return dict(value)
def _resolved_task(self, session: Any) -> str:
requested = str(self.config.get("task", self.config.get("requested_task", "auto"))).lower()
for source in (session, getattr(session, "config", None)):
if source is None:
continue
value = source.get("task") if isinstance(source, Mapping) else getattr(source, "task", None)
if str(value).lower() in {"tts", "clon", "vdes"}:
return str(value).lower()
if requested in {"tts", "clon", "vdes"}:
return requested
if str(self.config.get("connection_mode", "auto")).lower() == "external_server":
return "auto"
try:
from utils.audio_cpp.catalog import resolve_task
return str(
resolve_task(
self.config.get("family", ""),
self.config.get("package_id", ""),
requested="auto",
)
).lower()
except (ImportError, KeyError, TypeError, ValueError):
return "tts"
def _build_request(
self,
text: str,
voice_path: Optional[str],
reference_text: str,
seed: int,
advanced: Dict[str, Any],
task: str,
) -> Dict[str, Any]:
request: Dict[str, Any] = {"text": text, "seed": str(int(seed)), "options": advanced}
del task # The persistent session owns its one configured model/task.
language = str(self.config.get("language", "")).strip()
if language and language.lower() not in {"auto", "none"}:
request["language"] = language
voice_id = str(self.config.get("voice_id", self.config.get("voice", ""))).strip()
if voice_id:
request["voice_id"] = voice_id
if voice_path:
request["voice_ref"] = voice_path
if reference_text:
request["reference_text"] = reference_text
instruct = str(self.config.get("instruct", "")).strip()
if instruct:
request["instruct"] = instruct
for key in self._COMMON_REQUEST_FIELDS:
value = self.config.get(key)
if value is not None and value != "":
request[key] = value
return request
def _cache_key(
self,
text: str,
audio_component: str,
reference_text: str,
seed: int,
task: str,
advanced: Dict[str, Any],
character_name: Optional[str],
session: Any,
) -> str:
session_config = getattr(session, "config", {})
if not isinstance(session_config, Mapping):
session_config = {}
session_family = getattr(session, "family", None) or session_config.get(
"family", self.config.get("family", "")
)
session_model_id = getattr(session, "model_id", None) or session_config.get(
"model_id", self.config.get("model_id", "")
)
# Owned servers use a random loopback port on every restart; that port is
# transport state, not model identity. External endpoints are stable and
# must participate in the cache key.
if bool(getattr(session, "owned", False)):
session_endpoint = ""
else:
session_endpoint = getattr(session, "endpoint", None) or self.config.get(
"server_url", self.config.get("external_server_url", "")
)
extra_identity = {
"options": advanced,
"speaking_rate": self.config.get("speaking_rate"),
"connection_mode": self.config.get("connection_mode", "auto"),
"server_url": session_endpoint,
"binary_path": session_config.get("binary_path", self.config.get("binary_path", "")),
"backend": session_config.get("backend", self.config.get("backend", "")),
"device": session_config.get("device", self.config.get("device", "")),
"load_options": session_config.get("load_options", self.config.get("load_options", {})),
"session_options": session_config.get(
"session_options", self.config.get("session_options", {})
),
"default_request_options": session_config.get(
"default_request_options", self.config.get("default_request_options", {})
),
}
return self.audio_cache.generate_cache_key(
"audio_cpp",
text=text,
audio_component=audio_component,
reference_text=reference_text,
family=session_family,
package_id=session_config.get("package_id", self.config.get("package_id", "")),
model_path=session_config.get("model_path", self.config.get("model_path", "")),
model_id=session_model_id,
task=task,
language=self.config.get("language", ""),
voice_id=self.config.get("voice_id", self.config.get("voice", "")),
instruct=self.config.get("instruct", ""),
temperature=self.config.get("temperature"),
top_p=self.config.get("top_p"),
top_k=self.config.get("top_k"),
repetition_penalty=self.config.get("repetition_penalty"),
max_tokens=self.config.get("max_tokens"),
max_steps=self.config.get("max_steps"),
num_inference_steps=self.config.get("num_inference_steps"),
guidance_scale=self.config.get("guidance_scale"),
seed=int(seed),
request_options=_canonical_json(extra_identity),
character=character_name or "narrator",
)
@staticmethod
def _normalize_result(result: Any) -> Tuple[torch.Tensor, int]:
waveform = result.get("waveform") if isinstance(result, Mapping) else getattr(result, "waveform", None)
sample_rate = result.get("sample_rate") if isinstance(result, Mapping) else getattr(result, "sample_rate", None)
if waveform is None:
named = result.get("named_audio", {}) if isinstance(result, Mapping) else getattr(result, "named_audio", {})
values = list(named.values()) if isinstance(named, Mapping) else list(named or [])
if len(values) == 1:
item = values[0]
waveform = item.get("waveform") if isinstance(item, Mapping) else getattr(item, "waveform", None)
sample_rate = sample_rate or (item.get("sample_rate") if isinstance(item, Mapping) else getattr(item, "sample_rate", None))
if waveform is None:
raise RuntimeError("audio.cpp returned no primary audio output")
if not torch.is_tensor(waveform):
waveform = torch.as_tensor(waveform, dtype=torch.float32)
waveform = waveform.detach().to(device="cpu", dtype=torch.float32)
if waveform.dim() == 1:
waveform = waveform.unsqueeze(0)
elif waveform.dim() == 3 and waveform.shape[0] == 1:
waveform = waveform.squeeze(0)
if waveform.dim() != 2:
raise ValueError(f"audio.cpp waveform must be [channels, samples], got {tuple(waveform.shape)}")
if sample_rate is None or int(sample_rate) <= 0:
raise ValueError("audio.cpp returned an invalid sample rate")
return waveform.contiguous(), int(sample_rate)
def generate_single(
self,
text: str,
voice_ref: Optional[Dict[str, Any]] = None,
seed: int = 0,
enable_audio_cache: bool = True,
character_name: Optional[str] = None,
) -> Tuple[torch.Tensor, int]:
stripped = str(text or "").strip()
if not stripped:
if self._last_sample_rate is None:
raise ValueError("audio.cpp cannot determine a sample rate for empty text")
return torch.zeros(1, 0, dtype=torch.float32), self._last_sample_rate
session = _get_session(self.config)
task = self._resolved_task(session)
advanced = self._advanced_options()
cleanup_path: Optional[str] = None
try:
voice_path, reference_text, audio_component, cleanup_path = self._materialize_reference(voice_ref)
cache_key = self._cache_key(
stripped,
audio_component,
reference_text,
seed,
task,
advanced,
character_name,
session,
)
if enable_audio_cache:
cached = self.audio_cache.get_cached_audio(cache_key)
with _CACHE_SAMPLE_RATES_LOCK:
cached_rate = _CACHE_SAMPLE_RATES.get(cache_key)
if cached is not None and cached_rate is not None:
self._last_sample_rate = cached_rate
return cached[0].clone(), cached_rate
request = self._build_request(stripped, voice_path, reference_text, seed, advanced, task)
waveform, sample_rate = self._normalize_result(session.run(request))
self._last_sample_rate = sample_rate
if enable_audio_cache:
duration = waveform.shape[-1] / sample_rate
self.audio_cache.cache_audio(cache_key, waveform, duration)
with _CACHE_SAMPLE_RATES_LOCK:
_CACHE_SAMPLE_RATES[cache_key] = sample_rate
return waveform, sample_rate
finally:
if cleanup_path:
try:
os.remove(cleanup_path)
except FileNotFoundError:
pass
# Short alias for callers that do not use the older ``EngineAdapter`` suffix.
AudioCppAdapter = AudioCppEngineAdapter
+111
View File
@@ -0,0 +1,111 @@
"""audio.cpp adapter for the Suite's unified Voice Changer node."""
from __future__ import annotations
import json
import os
from typing import Any, Dict, Mapping
import torch
from engines.adapters.audio_cpp_adapter import AudioCppEngineAdapter
from utils.audio.processing import AudioProcessingUtils
def _advanced_options(config: Mapping[str, Any]) -> Dict[str, Any]:
value = config.get("advanced_options", config.get("request_options", {}))
if value in (None, ""):
return {}
if isinstance(value, str):
try:
value = json.loads(value)
except json.JSONDecodeError as exc:
raise ValueError(f"Invalid audio.cpp advanced JSON: {exc.msg}") from exc
if not isinstance(value, Mapping):
raise ValueError("audio.cpp advanced options must be a JSON object")
return dict(value)
def _materialize(audio: Mapping[str, Any], label: str) -> str:
waveform = audio.get("waveform")
sample_rate = audio.get("sample_rate")
if not torch.is_tensor(waveform):
raise TypeError(f"audio.cpp {label} must contain a waveform tensor")
if sample_rate is None or int(sample_rate) <= 0:
raise ValueError(f"audio.cpp {label} must contain a positive sample_rate")
return os.path.abspath(
AudioProcessingUtils.save_audio_to_temp_file(waveform, int(sample_rate))
)
class AudioCppVoiceConversionAdapter:
"""Convert source audio toward a target reference using an audio.cpp VC task."""
def __init__(self, config: Dict[str, Any]):
self.config = dict(config)
def _session_config(self) -> Dict[str, Any]:
config = dict(self.config)
if str(config.get("connection_mode", "auto")).lower() != "external_server":
config["requested_task"] = "vc"
config["task"] = "vc"
return config
def convert_voice(
self,
source_audio: Dict[str, Any],
target_audio: Dict[str, Any],
refinement_passes: int = 1,
) -> tuple[Dict[str, Any], str]:
from utils.audio_cpp.session import get_audio_cpp_session
config = self._session_config()
family = str(config.get("family", "")).strip()
passes = max(1, int(refinement_passes))
current = source_audio
output_rate = int(source_audio["sample_rate"])
session = get_audio_cpp_session(config)
if str(getattr(session, "task", "vc")) != "vc":
raise ValueError(
f"audio.cpp model '{session.model_id}' is configured for task "
f"'{session.task}', not voice conversion"
)
for pass_index in range(passes):
source_path = _materialize(current, "source audio")
target_path = _materialize(target_audio, "target reference audio")
try:
request = {
"audio": source_path,
"voice_ref": target_path,
"source_audio": source_path,
"target_voice": target_path,
"options": _advanced_options(config),
}
print(
f"🔄 audio.cpp VC: {family or 'external model'} pass "
f"{pass_index + 1}/{passes}..."
)
result = session.run(request)
waveform, output_rate = AudioCppEngineAdapter._normalize_result(result)
current = {"waveform": waveform.unsqueeze(0), "sample_rate": output_rate}
finally:
for path in (source_path, target_path):
try:
os.remove(path)
except FileNotFoundError:
pass
info = (
f"Model family: {family or getattr(session, 'family', 'external')}\n"
f"Model ID: {session.model_id}\n"
f"Task: voice conversion\n"
f"Refinement passes: {passes}\n"
f"Output sample rate: {output_rate} Hz\n"
"Conversion completed successfully"
)
return current, info
__all__ = ["AudioCppVoiceConversionAdapter"]
+33 -40
View File
@@ -67,43 +67,7 @@ class ChatterBoxEngineAdapter:
"""
self.node.load_tts_model(device, language)
def create_segment_cache(self, character="narrator", *, cache_model_name=None,
cache_probe=False, **params):
"""Share cache identity between generation and pre-load probes."""
if not params.get("enable_audio_cache", True):
return None
from utils.audio.cache import create_cache_function
language = cache_model_name or params.get("current_language", params.get("model", "English"))
device = params.get("device", "auto")
model_source = params.get("model_source")
if not model_source:
# Do not borrow the source of a different currently loaded language.
sources = getattr(self, "_cache_model_sources", {})
if cache_probe:
model_source = sources.get((language, device))
if model_source is None:
return None
else:
if hasattr(self.node, "model_manager"):
model_source = self.node.model_manager.get_model_source("tts")
model_source = model_source or "unknown"
sources[(language, device)] = model_source
self._cache_model_sources = sources
audio_component = params.get("stable_audio_component", "main_reference")
if character != "narrator":
audio_component = f"char_file_{character}"
return create_cache_function(
engine_type="chatterbox", character=character,
exaggeration=params.get("exaggeration", 1.0),
temperature=params.get("temperature", 0.8),
cfg_weight=params.get("cfg_weight", 1.0),
seed=params.get("seed", 0), audio_component=audio_component,
model_source=model_source, device=device, language=language,
)
def generate_segment_audio(self, text: str, char_audio: str,
def generate_segment_audio(self, text: str, char_audio: str,
character: str = "narrator", **params) -> torch.Tensor:
"""
Generate ChatterBox audio for a text segment.
@@ -124,7 +88,34 @@ class ChatterBoxEngineAdapter:
seed = params.get("seed", 0)
enable_cache = params.get("enable_audio_cache", True)
cache_fn = self.create_segment_cache(character=character, **params)
# Create cache function if caching is enabled
cache_fn = None
if enable_cache:
from utils.audio.cache import create_cache_function
# Get current language/model for cache key
current_language = params.get("current_language", params.get("model", "English"))
audio_component = params.get("stable_audio_component", "main_reference")
if character != "narrator":
audio_component = f"char_file_{character}"
# Get model source
model_source = params.get("model_source")
if not model_source and hasattr(self.node, 'model_manager'):
model_source = self.node.model_manager.get_model_source("tts")
cache_fn = create_cache_function(
engine_type="chatterbox",
character=character,
exaggeration=exaggeration,
temperature=temperature,
cfg_weight=cfg_weight,
seed=seed,
audio_component=audio_component,
model_source=model_source or "unknown",
device=params.get("device", "auto"),
language=current_language
)
# Handle caching externally for consistency with F5-TTS
if cache_fn:
@@ -151,7 +142,8 @@ class ChatterBoxEngineAdapter:
enable_pause_tags=True,
character=character,
seed=seed,
enable_cache=enable_cache,
enable_cache=True, # Enable internal caching for pause tag processing
crash_protection_template=params.get("crash_protection_template", "hmm ,, {seg} hmm ,,"),
stable_audio_component=params.get("stable_audio_component", "main_reference")
)
# Don't use external cache for pause tag segments
@@ -168,6 +160,7 @@ class ChatterBoxEngineAdapter:
character=character,
seed=seed,
enable_cache=False, # Disable internal caching since we handle it externally
crash_protection_template=params.get("crash_protection_template", "hmm ,, {seg} hmm ,,"),
stable_audio_component=params.get("stable_audio_component", "main_reference")
)
@@ -407,4 +400,4 @@ class ChatterBoxEngineAdapter:
else:
num_samples = audio_tensor.numel()
return num_samples / 44100 # ChatterBox sample rate
return num_samples / 44100 # ChatterBox sample rate
@@ -110,6 +110,7 @@ class ChatterBoxStreamingAdapter(StreamingEngineAdapter):
"seed": kwargs.get("seed", 42),
"enable_chunking": False, # Don't chunk in streaming - already handled
"enable_audio_cache": kwargs.get("enable_audio_cache", True),
"crash_protection_template": kwargs.get("crash_protection_template", "hmm ,, {seg} hmm ,,"),
"device": kwargs.get("device", "auto"),
"reference_audio": kwargs.get("reference_audio", None)
}
@@ -299,6 +300,7 @@ class ChatterBoxStreamingAdapter(StreamingEngineAdapter):
cfg_weight = kwargs.get('cfg_weight', 0.5)
seed = kwargs.get('seed', 42)
enable_cache = kwargs.get('enable_audio_cache', True)
crash_protection = kwargs.get('crash_protection_template', 'hmm ,, {seg} hmm ,,')
# Check if node has generation method
if hasattr(self.node, '_generate_tts_with_pause_tags'):
@@ -319,6 +321,7 @@ class ChatterBoxStreamingAdapter(StreamingEngineAdapter):
text, voice_path, exaggeration, temperature, cfg_weight,
language, True, character=character, seed=seed,
enable_cache=enable_cache,
crash_protection_template=crash_protection,
stable_audio_component=stable_audio_component
)
elif hasattr(self.node, 'tts_model') and self.node.tts_model:
+30 -26
View File
@@ -73,30 +73,7 @@ class F5TTSEngineAdapter:
self.node.load_f5tts_model(model_name, device)
def create_segment_cache(self, character="narrator", char_text=None, *,
cache_model_name=None, cache_probe=False, **params):
"""Share cache identity between generation and pre-load probes."""
if not params.get("enable_audio_cache", True):
return None
from utils.audio.cache import create_cache_function
audio_component = params.get("stable_audio_component", "main_reference")
if character != "narrator":
audio_component = f"char_file_{character}"
model_name = cache_model_name or getattr(
self.node, "current_model_name", params.get("model", "F5TTS_Base")
)
return create_cache_function(
engine_type="f5tts", character=character, model_name=model_name,
device=params.get("device", "auto"), audio_component=audio_component,
ref_text=char_text, temperature=params.get("temperature", 0.8),
speed=params.get("speed", 1.0), target_rms=params.get("target_rms", 0.1),
cross_fade_duration=params.get("cross_fade_duration", 0.15),
nfe_step=max(1, min(params.get("nfe_step", 32), 71)),
cfg_strength=params.get("cfg_strength", 2.0), seed=params.get("seed", 0),
)
def generate_segment_audio(self, text: str, char_audio: str, char_text: str,
def generate_segment_audio(self, text: str, char_audio: str, char_text: str,
character: str = "narrator", **params) -> torch.Tensor:
"""
Generate F5-TTS audio for a text segment.
@@ -126,7 +103,34 @@ class F5TTSEngineAdapter:
if safe_nfe_step != nfe_step:
print(f"⚠️ F5-TTS: Clamped nfe_step from {nfe_step} to {safe_nfe_step} to prevent ODE solver issues")
cache_fn = self.create_segment_cache(character=character, char_text=char_text, **params)
# Create cache function if caching is enabled
cache_fn = None
if enable_cache:
from utils.audio.cache import create_cache_function
# Get audio component for cache key
audio_component = params.get("stable_audio_component", "main_reference")
if character != "narrator":
audio_component = f"char_file_{character}"
# Get current model name for cache key
current_model = getattr(self.node, 'current_model_name', params.get("model", "F5TTS_Base"))
cache_fn = create_cache_function(
engine_type="f5tts",
character=character,
model_name=current_model,
device=params.get("device", "auto"),
audio_component=audio_component,
ref_text=char_text,
temperature=temperature,
speed=speed,
target_rms=target_rms,
cross_fade_duration=cross_fade_duration,
nfe_step=safe_nfe_step,
cfg_strength=cfg_strength,
seed=seed
)
# Generate audio using F5-TTS with pause tag support
return self.node.generate_f5tts_with_pause_tags(
@@ -185,4 +189,4 @@ class F5TTSEngineAdapter:
else:
num_samples = audio_tensor.numel()
return num_samples / 24000 # F5-TTS sample rate
return num_samples / 24000 # F5-TTS sample rate
+3 -12
View File
@@ -74,10 +74,7 @@ class MaskedDiffWithXvec(torch.nn.Module):
# concat text and prompt_text
mask = (~make_pad_mask(token_len)).float().unsqueeze(-1).to(device)
# TTS Audio Suite patch: bound speech-token indices to prevent CUDA embedding asserts.
token = self.input_embedding(
torch.clamp(token, min=0, max=self.input_embedding.num_embeddings - 1)
) * mask
token = self.input_embedding(torch.clamp(token, min=0)) * mask
# text encode
h, h_lengths = self.encoder(token, token_len)
@@ -127,10 +124,7 @@ class MaskedDiffWithXvec(torch.nn.Module):
token_len1, token_len2 = prompt_token.shape[1], token.shape[1]
token, token_len = torch.concat([prompt_token, token], dim=1), prompt_token_len + token_len
mask = (~make_pad_mask(token_len)).unsqueeze(-1).to(embedding)
# TTS Audio Suite patch: bound speech-token indices to prevent CUDA embedding asserts.
token = self.input_embedding(
torch.clamp(token, min=0, max=self.input_embedding.num_embeddings - 1)
) * mask
token = self.input_embedding(torch.clamp(token, min=0)) * mask
# text encode
h, h_lengths = self.encoder(token, token_len)
@@ -221,10 +215,7 @@ class CausalMaskedDiffWithXvec(torch.nn.Module):
# concat text and prompt_text
token, token_len = torch.concat([prompt_token, token], dim=1), prompt_token_len + token_len
mask = (~make_pad_mask(token_len)).unsqueeze(-1).to(embedding)
# TTS Audio Suite patch: bound speech-token indices to prevent CUDA embedding asserts.
token = self.input_embedding(
torch.clamp(token, min=0, max=self.input_embedding.num_embeddings - 1)
) * mask
token = self.input_embedding(torch.clamp(token, min=0)) * mask
# text encode
h, h_lengths = self.encoder(token, token_len)
@@ -1,5 +1,3 @@
import logging
from .s3tokenizer import (
S3_SR,
S3_HOP,
@@ -29,20 +27,4 @@ def drop_invalid_tokens(x):
e = None
x = x[s: e]
# T3's speech head spans `speech_tokens_dict_size` (8194) while S3Gen's
# input embedding only has SPEECH_VOCAB_SIZE (6561) rows, so a sampled id
# above the vocab that is not exactly SoS/EoS reaches the embedding gather
# and trips a device-side "index out of bounds" assert (which surfaces
# later, at the next CUDA sync, with a misleading traceback).
invalid = x >= SPEECH_VOCAB_SIZE
if invalid.any():
logging.warning(
"Dropping %d out-of-vocabulary speech token(s) (max id %d, vocab %d)",
int(invalid.sum()), int(x.max()), SPEECH_VOCAB_SIZE,
)
x = x[~invalid]
# TTS Audio Suite patch: do not pass an empty speech stream to S3Gen.
if x.numel() == 0:
raise RuntimeError("ChatterBox generated no valid speech tokens. Try another seed or text.")
return x
+8 -3
View File
@@ -16,6 +16,7 @@ from .modules.cond_enc import T3CondEnc, T3Cond
from .modules.t3_config import T3Config
from .llama_configs import LLAMA_CONFIGS
from .inference.t3_hf_backend import T3HuggingfaceBackend
from .inference.alignment_stream_analyzer import AlignmentStreamAnalyzer
logger = logging.getLogger(__name__)
@@ -276,9 +277,13 @@ class T3(nn.Module):
self.tfmr.use_cache = True
self.tfmr.return_dict = True
# TTS Audio Suite patch: alignment analysis is disabled in the backend, so do
# not install its persistent attention hook/forward wrapper on every inference.
alignment_stream_analyzer = None
alignment_stream_analyzer = AlignmentStreamAnalyzer(
self.tfmr,
None,
text_tokens_slice=(len_cond, len_cond + text_tokens.size(-1)),
alignment_layer_idx=9,
eos_idx=self.hp.stop_speech_token,
)
# Create backend with updated settings
patched_model = T3HuggingfaceBackend(
+5 -5
View File
@@ -268,7 +268,7 @@ class ChatterboxTTS:
if custom_path.suffix == ".safetensors":
return load_file(custom_path, device=device)
if custom_path.suffix == ".pt":
return torch.load(custom_path, map_location=device, weights_only=True)
return torch.load(custom_path, map_location=device)
safetensors_path = ckpt_dir / f"{base_name}.safetensors"
pt_path = ckpt_dir / f"{base_name}.pt"
@@ -276,7 +276,7 @@ class ChatterboxTTS:
if safetensors_path.exists():
return load_file(safetensors_path, device=device)
elif pt_path.exists():
return torch.load(pt_path, map_location=device, weights_only=True)
return torch.load(pt_path, map_location=device)
elif not required:
return None
else:
@@ -299,7 +299,7 @@ class ChatterboxTTS:
return load_file(safetensors_path, device=device)
elif pt_path.exists():
print(f"📁 Loading {base_name} from local English model: {pt_path}")
return torch.load(pt_path, map_location=device, weights_only=True)
return torch.load(pt_path, map_location=device)
# Download English model if not available locally
print(f"📦 Downloading English model components for incomplete language model...")
@@ -315,7 +315,7 @@ class ChatterboxTTS:
return load_file(safetensors_path, device=device)
elif pt_path.exists():
print(f"📁 Loading {base_name} from downloaded English model: {pt_path}")
return torch.load(pt_path, map_location=device, weights_only=True)
return torch.load(pt_path, map_location=device)
raise FileNotFoundError(f"Could not load {base_name} from English fallback")
@@ -452,7 +452,7 @@ class ChatterboxTTS:
warnings.simplefilter("ignore")
print(f"📦 Loading checkpoint: {unified_model_path.name}")
checkpoint = torch.load(unified_model_path, map_location=device, weights_only=True)
checkpoint = torch.load(unified_model_path, map_location=device)
# Extract model configuration
model_config = checkpoint.get('model_config', {})
+2 -2
View File
@@ -98,7 +98,7 @@ class ChatterboxVC:
if safetensors_path.exists():
return load_file(safetensors_path, device=device)
elif pt_path.exists():
return torch.load(pt_path, map_location=device, weights_only=True)
return torch.load(pt_path, map_location=device)
else:
raise FileNotFoundError(f"Neither {base_name}.safetensors nor {base_name}.pt found in {ckpt_dir}")
@@ -107,7 +107,7 @@ class ChatterboxVC:
ref_dict = None
if (builtin_voice := ckpt_dir / "conds.pt").exists():
states = torch.load(builtin_voice, map_location=device, weights_only=True)
states = torch.load(builtin_voice, map_location=device)
ref_dict = states['gen']
s3gen = S3Gen()
@@ -1,5 +1,3 @@
import logging
from .s3tokenizer import (
S3_SR,
S3_HOP,
@@ -29,20 +27,4 @@ def drop_invalid_tokens(x):
e = None
x = x[s: e]
# T3's speech head spans `speech_tokens_dict_size` (8194) while S3Gen's
# input embedding only has SPEECH_VOCAB_SIZE (6561) rows, so a sampled id
# above the vocab that is not exactly SoS/EoS reaches the embedding gather
# and trips a device-side "index out of bounds" assert (which surfaces
# later, at the next CUDA sync, with a misleading traceback).
invalid = x >= SPEECH_VOCAB_SIZE
if invalid.any():
logging.warning(
"Dropping %d out-of-vocabulary speech token(s) (max id %d, vocab %d)",
int(invalid.sum()), int(x.max()), SPEECH_VOCAB_SIZE,
)
x = x[~invalid]
# TTS Audio Suite patch: do not pass an empty speech stream to S3Gen.
if x.numel() == 0:
raise RuntimeError("ChatterBox generated no valid speech tokens. Try another seed or text.")
return x
@@ -60,10 +60,6 @@ class AlignmentStreamAnalyzer:
# using it for all layers slows things down too much. We can apply it to just one layer
# by intercepting the kwargs and adding a forward hook (credit: jrm)
self.last_aligned_attns = []
# TTS Audio Suite patch: retain hook handles so each inference can remove its
# analyzer hooks instead of accumulating three more on the shared model.
self._hook_handles = []
self._original_output_attentions = getattr(getattr(tfmr, "config", None), "output_attentions", None)
for i, (layer_idx, head_idx) in enumerate(LLAMA_ALIGNED_HEADS):
self.last_aligned_attns += [None]
self._add_attention_spy(tfmr, i, layer_idx, head_idx)
@@ -85,18 +81,11 @@ class AlignmentStreamAnalyzer:
target_layer = tfmr.layers[layer_idx].self_attn
# Register hook and store the handle
self._hook_handles.append(target_layer.register_forward_hook(attention_forward_hook))
target_layer.register_forward_hook(attention_forward_hook)
if hasattr(tfmr, 'config') and hasattr(tfmr.config, 'output_attentions'):
self.original_output_attentions = tfmr.config.output_attentions
tfmr.config.output_attentions = True
def close(self, tfmr):
"""Remove inference-scoped hooks and restore the transformer setting."""
for hook_handle in self._hook_handles:
hook_handle.remove()
self._hook_handles.clear()
if self._original_output_attentions is not None:
tfmr.config.output_attentions = self._original_output_attentions
def step(self, logits, next_token=None):
"""
Emits an AlignmentAnalysisResult into the output queue, and potentially modifies the logits to force an EOS.
@@ -268,12 +268,6 @@ class T3(nn.Module):
# TODO? synchronize the expensive compile function
# with self.compile_lock:
if not self.compiled:
# TTS Audio Suite patch: clean up hooks left by an interrupted prior inference.
previous_backend = getattr(self, "patched_model", None)
previous_analyzer = getattr(previous_backend, "alignment_stream_analyzer", None)
if previous_analyzer is not None:
previous_analyzer.close(self.tfmr)
# Default to None for English models, only create for multilingual
alignment_stream_analyzer = None
if self.hp.is_multilingual and use_alignment_analyzer:
@@ -408,7 +402,4 @@ class T3(nn.Module):
# Concatenate all predicted tokens along the sequence dimension.
predicted_tokens = torch.cat(predicted, dim=1) # shape: (B, num_tokens)
if self.patched_model.alignment_stream_analyzer is not None:
self.patched_model.alignment_stream_analyzer.close(self.tfmr)
self.patched_model.alignment_stream_analyzer = None
return predicted_tokens
+1 -1
View File
@@ -432,7 +432,7 @@ class ChatterboxOfficial23LangTTS:
warnings.simplefilter("ignore")
print(f"📦 Loading checkpoint: {unified_model_path.name}")
checkpoint = torch.load(unified_model_path, map_location=actual_device, weights_only=True)
checkpoint = torch.load(unified_model_path, map_location=actual_device)
# Extract model configuration
model_config = checkpoint.get('model_config', {})
+2 -2
View File
@@ -114,7 +114,7 @@ class ChatterboxVC:
if safetensors_path.exists():
return load_file(safetensors_path, device=device)
elif pt_path.exists():
return torch.load(pt_path, map_location=device, weights_only=True)
return torch.load(pt_path, map_location=device)
else:
raise FileNotFoundError(f"Neither {base_name}.safetensors nor {base_name}.pt found in {ckpt_dir}")
@@ -123,7 +123,7 @@ class ChatterboxVC:
ref_dict = None
if (builtin_voice := ckpt_dir / "conds.pt").exists():
states = torch.load(builtin_voice, weights_only=True)
states = torch.load(builtin_voice)
ref_dict = states['gen']
s3gen = S3Gen()
@@ -73,8 +73,7 @@ def main():
assert num == len(path_list)
for path in path_list:
print('Processing {}'.format(path))
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
states = torch.load(path, map_location=torch.device('cpu'), weights_only=True)
states = torch.load(path, map_location=torch.device('cpu'))
for k in states.keys():
if k not in ['step', 'epoch']:
if k not in avg.keys():
@@ -134,8 +134,7 @@ def main():
start_step, start_epoch = 0, -1
if args.checkpoint is not None:
if os.path.exists(args.checkpoint):
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
state_dict = torch.load(args.checkpoint, map_location='cpu', weights_only=True)
state_dict = torch.load(args.checkpoint, map_location='cpu')
model.load_state_dict(state_dict, strict=False)
if 'step' in state_dict:
start_step = state_dict['step']
@@ -162,8 +161,7 @@ def main():
# DPO related
if args.dpo is True:
ref_model = deepcopy(configs[args.model])
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
state_dict = torch.load(args.ref_model, map_location='cpu', weights_only=True)
state_dict = torch.load(args.ref_model, map_location='cpu')
ref_model.load_state_dict(state_dict, strict=False)
dpo_loss = DPOLoss(beta=0.01, label_smoothing=0.0, ipo=False)
# NOTE maybe it is not needed to wrap ref_model as ddp because its parameter is not updated
@@ -47,8 +47,7 @@ class CosyVoiceFrontEnd:
providers=["CUDAExecutionProvider" if torch.cuda.is_available() else
"CPUExecutionProvider"])
if os.path.exists(spk2info):
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
self.spk2info = torch.load(spk2info, map_location=self.device, weights_only=True)
self.spk2info = torch.load(spk2info, map_location=self.device)
else:
self.spk2info = {}
self.allowed_special = allowed_special
@@ -64,8 +64,7 @@ class CosyVoiceModel:
self.hift_cache_dict = {}
def load(self, llm_model, flow_model, hift_model):
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
self.llm.load_state_dict(torch.load(llm_model, map_location=self.device, weights_only=True), strict=True)
self.llm.load_state_dict(torch.load(llm_model, map_location=self.device), strict=True)
self.llm.to(self.device).eval()
# FIX: Bundled code fix - Ensure all nested modules are moved to device
# Issue: Some nested layers (embed_tokens, etc.) weren't being moved with .to(device)
@@ -73,16 +72,14 @@ class CosyVoiceModel:
for module in self.llm.modules():
module.to(self.device)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
self.flow.load_state_dict(torch.load(flow_model, map_location=self.device, weights_only=True), strict=True)
self.flow.load_state_dict(torch.load(flow_model, map_location=self.device), strict=True)
self.flow.to(self.device).eval()
# FIX: Bundled code fix - Ensure all nested modules are moved to device
for module in self.flow.modules():
module.to(self.device)
# in case hift_model is a hifigan model
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
hift_state_dict = {k.replace('generator.', ''): v for k, v in torch.load(hift_model, map_location=self.device, weights_only=True).items()}
hift_state_dict = {k.replace('generator.', ''): v for k, v in torch.load(hift_model, map_location=self.device).items()}
self.hift.load_state_dict(hift_state_dict, strict=True)
self.hift.to(self.device).eval()
# FIX: Bundled code fix - Ensure all nested modules are moved to device
@@ -84,8 +84,7 @@ def assert_required_models_available(args):
def load_hifigan(checkpoint_path, device):
h = AttrDict(v1)
hifigan = HiFiGAN(h).to(device)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
hifigan.load_state_dict(torch.load(checkpoint_path, map_location=device, weights_only=True)["generator"])
hifigan.load_state_dict(torch.load(checkpoint_path, map_location=device)["generator"])
_ = hifigan.eval()
hifigan.remove_weight_norm()
return hifigan
@@ -41,8 +41,7 @@ def get_padding(kernel_size, dilation=1):
def load_checkpoint(filepath, device):
assert os.path.isfile(filepath)
print(f"Loading '{filepath}'")
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
checkpoint_dict = torch.load(filepath, map_location=device, weights_only=True)
checkpoint_dict = torch.load(filepath, map_location=device)
print("Complete.")
return checkpoint_dict
+1 -4
View File
@@ -15,8 +15,6 @@ from utils.hf_download_logging import quiet_hf_download_logs
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
class DotsTTSDownloader:
"""Resolve and download official dots.tts model folders."""
@@ -61,7 +59,6 @@ class DotsTTSDownloader:
self.base_path = base_path
os.makedirs(self.base_path, exist_ok=True)
@validate_model_paths
def resolve_model_path(self, model_identifier: str) -> str:
"""Resolve local: names, absolute paths, or known official model identifiers."""
if not model_identifier:
@@ -115,7 +112,7 @@ class DotsTTSDownloader:
with quiet_hf_download_logs():
snapshot_download(
repo_id=repo_id,
local_dir=allowed_path(model_dir),
local_dir=model_dir,
local_dir_use_symlinks=False,
resume_download=True,
force_download=force,
-4
View File
@@ -4,7 +4,6 @@ Dots TTS engine wrapper.
Wraps the official dots.tts runtime with ComfyUI-friendly model lifecycle hooks.
"""
from utils.security.data_loading import RestrictedLoadModule, load_dependency_checkpoint
import os
import sys
import types
@@ -109,9 +108,6 @@ class DotsTTSEngine:
try:
with self._text_normalizer_compat():
from dots_tts.runtime import DotsTtsRuntime
from dots_tts.models.dots_tts import core
if not isinstance(core.torch, RestrictedLoadModule):
core.torch = RestrictedLoadModule(core.torch, load_dependency_checkpoint)
except Exception as e:
for module_name, module in stale_modules:
sys.modules.setdefault(module_name, module)
-3
View File
@@ -9,8 +9,6 @@ from utils.downloads.unified_downloader import unified_downloader
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
from utils.security.path_access import validate_model_paths, allowed_path
class DramaBoxDownloader:
"""Resolve DramaBox checkpoints without using Hugging Face cache storage."""
@@ -67,7 +65,6 @@ class DramaBoxDownloader:
models.insert(0, local_name)
return models
@validate_model_paths
def resolve_model_path(self, model_identifier: str = MODEL_NAME) -> Dict[str, str]:
model_identifier = model_identifier or self.MODEL_NAME
if os.path.isabs(model_identifier) and os.path.isdir(model_identifier):
+12 -6
View File
@@ -8,7 +8,6 @@ also valid for training.
from __future__ import annotations
from utils.security.path_access import resolve_input_path, allowed_path, child_path
import csv
import hashlib
import json
@@ -43,10 +42,17 @@ def get_dramabox_training_root() -> str:
def _resolve_source_path(value: str) -> Path:
path = Path(resolve_input_path(value, datasets=True, models=False))
if not (path.is_file()):
raise FileNotFoundError(f"Dataset source not found: {value}")
return path
raw = os.path.expanduser(str(value or "").strip())
if not raw:
raise ValueError("dataset_source is required")
candidates = [Path(raw)]
input_root = Path(folder_paths.get_input_directory())
candidates.extend((input_root / raw, input_root / "datasets" / raw))
for candidate in candidates:
if candidate.is_file():
return candidate.resolve()
raise FileNotFoundError(f"DramaBox dataset source not found: {value}")
def _resolve_audio_path(raw_path: Any, *, source_path: Path, audio_dir: str) -> Path:
@@ -65,7 +71,7 @@ def _resolve_audio_path(raw_path: Any, *, source_path: Path, audio_dir: str) ->
for candidate in candidates:
if candidate.is_file():
return Path(allowed_path(candidate, models=False))
return candidate.resolve()
raise FileNotFoundError(f"DramaBox audio file not found: {raw_path}")
+2 -4
View File
@@ -163,15 +163,13 @@ class IDLoRADataset(Dataset):
def _load_sample(self, data_dir, idx):
base = Path(data_dir)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
audio = torch.load(base / "audio_latents" / f"sample_{idx:06d}.pt", weights_only=True)
audio = torch.load(base / "audio_latents" / f"sample_{idx:06d}.pt", weights_only=False)
# Prefer prefix-stripped text embeddings if they exist (re-encoded with
# just the quoted dialogue, dropping the "A woman says, " / "A man
# speaks with X accent, " scene-description prefix).
stripped = base / "conditions_stripped" / f"sample_{idx:06d}.pt"
cond_path = stripped if stripped.exists() else base / "conditions" / f"sample_{idx:06d}.pt"
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
cond = torch.load(cond_path, weights_only=True)
cond = torch.load(cond_path, weights_only=False)
if isinstance(audio, dict):
audio = audio.get("audio_latent", audio.get("latent", list(audio.values())[0]))
if audio.dim() == 2:
+2 -4
View File
@@ -184,11 +184,9 @@ class ECAPA_TDNN(nn.Module):
torch.hub._validate_not_a_forked_repo = lambda a, b, c: True
try:
local_s3prl_path = os.path.expanduser("~/.cache/torch/hub/s3prl_s3prl_main")
# TTS Audio Suite patch: restrict checkpoint deserialization.
self.feature_extract = torch.hub.load(local_s3prl_path, feat_type, source="local", config_path=config_path, weights_only=True)
self.feature_extract = torch.hub.load(local_s3prl_path, feat_type, source="local", config_path=config_path)
except: # noqa: E722
# TTS Audio Suite patch: restrict checkpoint deserialization.
self.feature_extract = torch.hub.load("s3prl/s3prl", feat_type, weights_only=True)
self.feature_extract = torch.hub.load("s3prl/s3prl", feat_type)
if len(self.feature_extract.model.encoder.layers) == 24 and hasattr(
self.feature_extract.model.encoder.layers[23].self_attn, "fp32_attention"
+1 -2
View File
@@ -15,8 +15,7 @@ def main():
device = "cuda" if torch.cuda.is_available() else "xpu" if torch.xpu.is_available() else "cpu"
# TTS Audio Suite patch: restrict checkpoint deserialization.
predictor = torch.hub.load("tarepan/SpeechMOS:v1.2.0", "utmos22_strong", trust_repo=True, weights_only=True)
predictor = torch.hub.load("tarepan/SpeechMOS:v1.2.0", "utmos22_strong", trust_repo=True)
predictor = predictor.to(device)
audio_paths = list(Path(args.audio_dir).rglob(f"*.{args.ext}"))
@@ -203,8 +203,7 @@ def convert_timm_dit(args, mapping, dtype="float32"):
torch_dtype = str_dtype_to_torch(dtype)
tensor_parallel = mapping.tp_size
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
model_params = dict(torch.load(args.timm_ckpt, weights_only=True))
model_params = dict(torch.load(args.timm_ckpt))
model_params = {
k: v for k, v in model_params["ema_model_state_dict"].items() if k.startswith("ema_model.transformer")
}
+1 -2
View File
@@ -968,8 +968,7 @@ def expand_model_embeddings(ckpt_path, new_ckpt_path, num_new_tokens=42):
ckpt = load_file(ckpt_path, device="cpu")
ckpt = {"ema_model_state_dict": ckpt}
elif ckpt_path.endswith(".pt"):
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=True)
ckpt = torch.load(ckpt_path, map_location="cpu")
ema_sd = ckpt.get("ema_model_state_dict", {})
embed_key_ema = "ema_model.transformer.text_embed.text_embed.weight"
-3
View File
@@ -8,8 +8,6 @@ import folder_paths
from utils.downloads.unified_downloader import unified_downloader
from utils.security.path_access import validate_model_paths, allowed_path
class FishAudioS2Downloader:
VARIANTS = {
"s2-pro": {
@@ -93,7 +91,6 @@ class FishAudioS2Downloader:
return "s2-pro"
@classmethod
@validate_model_paths
def resolve_model_path(cls, selection: str) -> str:
"""Resolve local selections or download the selected official variant."""
if str(selection).startswith("local:"):
@@ -20,8 +20,6 @@ from utils.models.extra_paths import get_preferred_download_path
import folder_paths
from utils.security.path_access import validate_model_paths, allowed_path
class GraniteASRDownloader:
"""Downloader for Granite ASR models using unified download system."""
@@ -148,7 +146,6 @@ class GraniteASRDownloader:
if verbose:
print("✅ Granite model verification passed")
@validate_model_paths
def resolve_model_path(self, model_identifier: str) -> str:
if not model_identifier:
model_identifier = "granite-speech-4.1-2b"
@@ -1,4 +1,3 @@
from utils.security.data_loading import load_numpy_data # TTS Audio Suite patch: NumPy dictionaries must not execute pickle globals.
import math
from dataclasses import dataclass
from pathlib import Path
@@ -46,7 +45,7 @@ class DACFile:
@classmethod
def load(cls, path):
artifacts = load_numpy_data(path)[()]
artifacts = np.load(path, allow_pickle=True)[()]
codes = torch.from_numpy(artifacts["codes"].astype(int))
if artifacts["metadata"].get("dac_version", None) not in SUPPORTED_VERSIONS:
raise RuntimeError(f"Given file {path} can't be loaded with this version of descript-audio-codec.")
@@ -357,8 +357,7 @@ def load_higgs_audio_tokenizer(tokenizer_name_or_path, device="cuda"):
**config,
device=device,
)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
parameter_dict = torch.load(model_path, map_location=device, weights_only=True)
parameter_dict = torch.load(model_path, map_location=device)
model.load_state_dict(parameter_dict, strict=False)
model.to(device)
model.eval()
@@ -12,8 +12,6 @@ from utils.hf_download_logging import quiet_hf_download_logs
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
class HiggsAudioV3Downloader:
"""Resolve and download official Higgs Audio v3 model folders."""
@@ -47,7 +45,6 @@ class HiggsAudioV3Downloader:
self.base_path = base_path
os.makedirs(self.base_path, exist_ok=True)
@validate_model_paths
def resolve_model_path(self, model_identifier: str) -> str:
"""Resolve local paths, local: names, or known Higgs v3 model identifiers."""
model_identifier = model_identifier or self.MODEL_NAME
@@ -104,7 +101,7 @@ class HiggsAudioV3Downloader:
with quiet_hf_download_logs():
snapshot_download(
repo_id=repo_id,
local_dir=allowed_path(model_dir),
local_dir=model_dir,
local_dir_use_symlinks=False,
resume_download=True,
force_download=force,
@@ -520,8 +520,7 @@ class BigVGAN(
local_files_only=local_files_only,
)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
checkpoint_dict = torch.load(model_file, map_location=map_location, weights_only=True)
checkpoint_dict = torch.load(model_file, map_location=map_location)
try:
model.load_state_dict(checkpoint_dict["generator"])
+1 -2
View File
@@ -63,8 +63,7 @@ def get_padding(kernel_size, dilation=1):
def load_checkpoint(filepath, device):
assert os.path.isfile(filepath)
print(f"Loading '{filepath}'")
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
checkpoint_dict = torch.load(filepath, map_location=device, weights_only=True)
checkpoint_dict = torch.load(filepath, map_location=device)
print("Complete.")
return checkpoint_dict
+1 -2
View File
@@ -234,8 +234,7 @@ class EnhancedCodec(nn.Module):
def load_checkpoint(self, checkpoint_path):
"""Load model weights from a checkpoint file."""
assert os.path.isfile(checkpoint_path), f"Checkpoint not found: {checkpoint_path}"
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
checkpoint_dict = torch.load(checkpoint_path, map_location='cpu', weights_only=True)
checkpoint_dict = torch.load(checkpoint_path, map_location='cpu')
saved_state_dict = checkpoint_dict['model']
state_dict = self.state_dict()
new_state_dict = {}
@@ -662,8 +662,12 @@ def load_state_dict(
):
extra_args = {"mmap": True}
weights_only_kwarg = {"weights_only": weights_only} if is_torch_greater_or_equal_than_1_13 else {}
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
return torch.load(checkpoint_file, map_location=map_location, **{**weights_only_kwarg, **extra_args, "weights_only": True})
return torch.load(
checkpoint_file,
map_location=map_location,
**weights_only_kwarg,
**extra_args,
)
except Exception as e:
try:
with open(checkpoint_file) as f:
+1 -2
View File
@@ -111,8 +111,7 @@ class IndexTTS:
self.use_cuda_kernel = False
self.bigvgan = Generator(self.cfg.bigvgan, use_cuda_kernel=self.use_cuda_kernel)
self.bigvgan_path = os.path.join(self.model_dir, self.cfg.bigvgan_checkpoint)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
vocoder_dict = torch.load(self.bigvgan_path, map_location="cpu", weights_only=True)
vocoder_dict = torch.load(self.bigvgan_path, map_location="cpu")
self.bigvgan.load_state_dict(vocoder_dict["generator"])
self.bigvgan = self.bigvgan.to(self.device)
# remove weight norm on eval mode
+4 -26
View File
@@ -192,17 +192,6 @@ class IndexTTS2:
self.gpt.eval()
print(">> GPT weights restored from:", self.gpt_path)
if use_deepspeed:
try:
import transformers
transformers_major = int(transformers.__version__.split('.', 1)[0])
except (ImportError, ValueError, AttributeError):
transformers_major = None
if transformers_major is not None and transformers_major >= 5:
use_deepspeed = False
print("⚠️ DeepSpeed kernel injection is not supported by the bundled IndexTTS GPT2 path with Transformers 5.x")
print(">> Falling back to standard IndexTTS-2 inference (DeepSpeed disabled)")
if use_deepspeed:
try:
import deepspeed
@@ -210,15 +199,7 @@ class IndexTTS2:
use_deepspeed = False
print(f">> Failed to load DeepSpeed. Falling back to normal inference. Error: {e}")
try:
self.gpt.post_init_gpt2_config(use_deepspeed=use_deepspeed, kv_cache=True, half=self.use_fp16)
except Exception as e:
if not use_deepspeed:
raise
use_deepspeed = False
print(f"⚠️ DeepSpeed initialization failed: {e}")
print(">> Falling back to standard IndexTTS-2 inference (DeepSpeed disabled)")
self.gpt.post_init_gpt2_config(use_deepspeed=False, kv_cache=True, half=self.use_fp16)
self.gpt.post_init_gpt2_config(use_deepspeed=use_deepspeed, kv_cache=True, half=self.use_fp16)
if self.use_cuda_kernel:
# preload the CUDA kernel for BigVGAN
@@ -338,8 +319,7 @@ class IndexTTS2:
campplus_ckpt_path = os.path.join(campplus_path, "campplus_cn_common.bin")
campplus_model = CAMPPlus(feat_dim=80, embedding_size=192)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
campplus_model.load_state_dict(torch.load(campplus_ckpt_path, map_location="cpu", weights_only=True))
campplus_model.load_state_dict(torch.load(campplus_ckpt_path, map_location="cpu"))
self.campplus_model = campplus_model.to(self.load_device)
self.campplus_model.eval()
print(">> campplus_model weights restored from:", campplus_ckpt_path)
@@ -365,13 +345,11 @@ class IndexTTS2:
self.tokenizer = TextTokenizer(self.bpe_path, self.normalizer)
print(">> bpe model loaded from:", self.bpe_path)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
emo_matrix = torch.load(os.path.join(self.model_dir, self.cfg.emo_matrix), weights_only=True)
emo_matrix = torch.load(os.path.join(self.model_dir, self.cfg.emo_matrix))
self.emo_matrix = emo_matrix.to(self.load_device)
self.emo_num = list(self.cfg.emo_num)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
spk_matrix = torch.load(os.path.join(self.model_dir, self.cfg.spk_matrix), weights_only=True)
spk_matrix = torch.load(os.path.join(self.model_dir, self.cfg.spk_matrix))
self.spk_matrix = spk_matrix.to(self.load_device)
self.emo_matrix = torch.split(self.emo_matrix, self.emo_num)
+5 -28
View File
@@ -170,17 +170,6 @@ class IndexTTS2:
self.gpt.eval()
print(">> GPT weights restored from:", self.gpt_path)
if use_deepspeed:
try:
import transformers
transformers_major = int(transformers.__version__.split('.', 1)[0])
except (ImportError, ValueError, AttributeError):
transformers_major = None
if transformers_major is not None and transformers_major >= 5:
use_deepspeed = False
print("⚠️ DeepSpeed kernel injection is not supported by the bundled IndexTTS GPT2 path with Transformers 5.x")
print(">> Falling back to standard IndexTTS-2.5 inference (DeepSpeed disabled)")
if use_deepspeed:
try:
import deepspeed
@@ -188,15 +177,7 @@ class IndexTTS2:
use_deepspeed = False
print(f">> Failed to load DeepSpeed. Falling back to normal inference. Error: {e}")
try:
self.gpt.post_init_gpt2_config(use_deepspeed=use_deepspeed, kv_cache=True, half=self.use_bf16)
except Exception as e:
if not use_deepspeed:
raise
use_deepspeed = False
print(f"⚠️ DeepSpeed initialization failed: {e}")
print(">> Falling back to standard IndexTTS-2.5 inference (DeepSpeed disabled)")
self.gpt.post_init_gpt2_config(use_deepspeed=False, kv_cache=True, half=self.use_bf16)
self.gpt.post_init_gpt2_config(use_deepspeed=use_deepspeed, kv_cache=True, half=self.use_bf16)
if self.use_cuda_kernel:
# preload the CUDA kernel for BigVGAN
@@ -217,8 +198,7 @@ class IndexTTS2:
self.semantic_model = Wav2Vec2BertModel.from_pretrained(w2v_bert_dir, local_files_only=True)
self.semantic_model = self.semantic_model.to(self.device)
self.semantic_model.eval()
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
stat_mean_var = torch.load(os.path.join(self.model_dir, self.cfg.w2v_stat), weights_only=True)
stat_mean_var = torch.load(os.path.join(self.model_dir, self.cfg.w2v_stat))
self.semantic_mean = stat_mean_var["mean"].to(self.device)
self.semantic_std = torch.sqrt(stat_mean_var["var"]).to(self.device)
@@ -262,8 +242,7 @@ class IndexTTS2:
campplus_dir = index_tts_downloader.download_model("campplus")
campplus_ckpt_path = os.path.join(campplus_dir, "campplus_cn_common.bin")
campplus_model = CAMPPlus(feat_dim=80, embedding_size=192)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
campplus_model.load_state_dict(torch.load(campplus_ckpt_path, map_location="cpu", weights_only=True))
campplus_model.load_state_dict(torch.load(campplus_ckpt_path, map_location="cpu"))
self.campplus_model = campplus_model.to(self.device)
self.campplus_model.eval()
print(">> campplus_model weights restored from:", campplus_ckpt_path)
@@ -286,13 +265,11 @@ class IndexTTS2:
self.text_process.load_glossary_from_yaml(self.glossary_path)
print(">> Glossary loaded from:", self.glossary_path)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
emo_matrix = torch.load(os.path.join(self.model_dir, self.cfg.emo_matrix), weights_only=True)
emo_matrix = torch.load(os.path.join(self.model_dir, self.cfg.emo_matrix))
self.emo_matrix = emo_matrix.to(self.device)
self.emo_num = list(self.cfg.emo_num)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
spk_matrix = torch.load(os.path.join(self.model_dir, self.cfg.spk_matrix), weights_only=True)
spk_matrix = torch.load(os.path.join(self.model_dir, self.cfg.spk_matrix))
self.spk_matrix = spk_matrix.to(self.device)
self.emo_matrix = torch.split(self.emo_matrix, self.emo_num)
@@ -1,4 +1,3 @@
from utils.security.data_loading import load_numpy_data # TTS Audio Suite patch: NumPy dictionaries must not execute pickle globals.
import math
from dataclasses import dataclass
from pathlib import Path
@@ -46,7 +45,7 @@ class DACFile:
@classmethod
def load(cls, path):
artifacts = load_numpy_data(path)[()]
artifacts = np.load(path, allow_pickle=True)[()]
codes = torch.from_numpy(artifacts["codes"].astype(int))
if artifacts["metadata"].get("dac_version", None) not in SUPPORTED_VERSIONS:
raise RuntimeError(
@@ -478,8 +478,7 @@ class BigVGAN(
local_files_only=local_files_only,
)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
checkpoint_dict = torch.load(model_file, map_location=map_location, weights_only=True)
checkpoint_dict = torch.load(model_file, map_location=map_location)
try:
model.load_state_dict(checkpoint_dict["generator"])
@@ -61,8 +61,7 @@ def get_padding(kernel_size, dilation=1):
def load_checkpoint(filepath, device):
assert os.path.isfile(filepath)
print(f"Loading '{filepath}'")
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
checkpoint_dict = torch.load(filepath, map_location=device, weights_only=True)
checkpoint_dict = torch.load(filepath, map_location=device)
print("Complete.")
return checkpoint_dict
@@ -220,8 +220,7 @@ def load_F0_models(path):
from .JDC.model import JDCNet
F0_model = JDCNet(num_class=1, seq_len=192)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
params = torch.load(path, map_location="cpu", weights_only=True)["net"]
params = torch.load(path, map_location="cpu")["net"]
F0_model.load_state_dict(params)
_ = F0_model.train()
@@ -529,8 +528,7 @@ def load_checkpoint(
is_distributed=False,
load_ema=False,
):
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
state = torch.load(path, map_location="cpu", weights_only=True)
state = torch.load(path, map_location="cpu")
params = state["net"]
if load_ema and "ema" in state:
print("Loading EMA")
@@ -587,8 +585,7 @@ def load_checkpoint2(
is_distributed=False,
load_ema=False,
):
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
state = torch.load(path, map_location="cpu", weights_only=True)
state = torch.load(path, map_location="cpu")
params = state["net"]
if load_ema and "ema" in state:
print("Loading EMA")
@@ -33,8 +33,7 @@ class OpenVoiceBaseClass(object):
self.device = device
def load_ckpt(self, ckpt_path):
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
checkpoint_dict = torch.load(ckpt_path, map_location=torch.device(self.device), weights_only=True)
checkpoint_dict = torch.load(ckpt_path, map_location=torch.device(self.device))
a, b = self.model.load_state_dict(checkpoint_dict['model'], strict=False)
print("Loaded checkpoint '{}'".format(ckpt_path))
print('missing/unexpected keys:', a, b)
@@ -27,12 +27,9 @@ tone_color_converter = ToneColorConverter(f'{ckpt_converter}/config.json', devic
tone_color_converter.load_ckpt(f'{ckpt_converter}/checkpoint.pth')
# load speaker embeddings
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
en_source_default_se = torch.load(f'{en_ckpt_base}/en_default_se.pth', weights_only=True).to(device)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
en_source_style_se = torch.load(f'{en_ckpt_base}/en_style_se.pth', weights_only=True).to(device)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
zh_source_se = torch.load(f'{zh_ckpt_base}/zh_default_se.pth', weights_only=True).to(device)
en_source_default_se = torch.load(f'{en_ckpt_base}/en_default_se.pth').to(device)
en_source_style_se = torch.load(f'{en_ckpt_base}/en_style_se.pth').to(device)
zh_source_se = torch.load(f'{zh_ckpt_base}/zh_default_se.pth').to(device)
# This online demo mainly supports English and Chinese
supported_languages = ['zh', 'en']
@@ -536,8 +536,7 @@ class RMVPE:
def get_default_model():
model = E2E(4, 1, (2, 2))
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
ckpt = torch.load(model_path, map_location="cpu", weights_only=True)
ckpt = torch.load(model_path, map_location="cpu")
model.load_state_dict(ckpt)
model.eval()
if is_half:
@@ -91,8 +91,7 @@ class Extract_wav2vectbert:
self.semantic_model = Wav2Vec2BertModel.from_pretrained("./MaskGCT_model/w2v_bert/")
self.semantic_model.eval()
self.semantic_model.to(device)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
self.stat_mean_var = torch.load("./MaskGCT_model/wav2vec2bert_stats.pt", weights_only=True)
self.stat_mean_var = torch.load("./MaskGCT_model/wav2vec2bert_stats.pt")
self.semantic_mean = self.stat_mean_var["mean"]
self.semantic_std = torch.sqrt(self.stat_mean_var["var"])
self.semantic_mean = self.semantic_mean.to(device)
@@ -23,8 +23,7 @@ import yaml
def load_checkpoint(model: torch.nn.Module, model_pth: str) -> dict:
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
checkpoint = torch.load(model_pth, map_location='cpu', weights_only=True)
checkpoint = torch.load(model_pth, map_location='cpu')
checkpoint = checkpoint['model'] if 'model' in checkpoint else checkpoint
model.load_state_dict(checkpoint, strict=True)
info_path = re.sub('.pth$', '.yaml', model_pth)
@@ -301,7 +301,6 @@ class VocoderInference(object):
"melgan",
"nsfhifigan",
]:
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
ckpt = torch.load(
checkpoint_dir,
map_location=(
@@ -309,7 +308,7 @@ class VocoderInference(object):
if torch.cuda.is_available()
else torch.device("cpu")
),
weights_only=True)
)
if from_multi_gpu:
pretrained_generator_dict = ckpt["generator_state_dict"]
generator_dict = self.model.state_dict()
@@ -329,8 +328,7 @@ class VocoderInference(object):
else:
self.model.load_state_dict(ckpt["generator_state_dict"])
else:
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
self.model.load_state_dict(torch.load(checkpoint_dir, weights_only=True)["state_dict"])
self.model.load_state_dict(torch.load(checkpoint_dir)["state_dict"])
return str(checkpoint_dir)
def inference(self):
@@ -414,7 +412,6 @@ def load_nnvocoder(
if not os.path.isdir(weights_file):
# Load from .pt file
if vocoder_name in ["bigvgan", "hifigan", "melgan", "nsfhifigan"]:
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
ckpt = torch.load(
weights_file,
map_location=(
@@ -422,7 +419,7 @@ def load_nnvocoder(
if torch.cuda.is_available()
else torch.device("cpu")
),
weights_only=True)
)
if from_multi_gpu:
pretrained_generator_dict = ckpt["generator_state_dict"]
generator_dict = model.state_dict()
@@ -442,8 +439,7 @@ def load_nnvocoder(
else:
model.load_state_dict(ckpt["generator_state_dict"])
else:
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
model.load_state_dict(torch.load(weights_file, weights_only=True)["state_dict"])
model.load_state_dict(torch.load(weights_file)["state_dict"])
else:
# Load from accelerator state dict
weights_file = os.path.join(weights_file, "checkpoint")
@@ -34,8 +34,7 @@ class FAcodecInference(object):
return model
def _load_checkpoint(self):
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
sd = torch.load(self.args.checkpoint_path, map_location="cpu", weights_only=True)
sd = torch.load(self.args.checkpoint_path, map_location="cpu")
sd = sd["net"] if "net" in sd else sd
new_params = dict()
for key, state_dict in sd.items():
@@ -208,8 +208,7 @@ def load_F0_models(path):
F0_model = JDCNet(num_class=1, seq_len=192)
if not os.path.exists(path):
path = hf_hub_download(repo_id="Plachta/JDCnet", filename="bst.t7")
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
params = torch.load(path, map_location="cpu", weights_only=True)["net"]
params = torch.load(path, map_location="cpu")["net"]
F0_model.load_state_dict(params)
_ = F0_model.train()
@@ -296,8 +295,7 @@ def load_checkpoint(
ignore_modules=[],
is_distributed=False,
):
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
state = torch.load(path, map_location="cpu", weights_only=True)
state = torch.load(path, map_location="cpu")
params = state["net"]
for key in model:
if key in params and key not in ignore_modules:
@@ -81,8 +81,7 @@ class SpeechTokenizer(nn.Module):
with open(config_path) as f:
cfg = json.load(f)
model = cls(cfg)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
params = torch.load(ckpt_path, map_location="cpu", weights_only=True)
params = torch.load(ckpt_path, map_location="cpu")
model.load_state_dict(params)
return model
@@ -132,8 +132,7 @@ def build_semantic_model(path_='./models/tts/maskgct/ckpt/wav2vec2bert_stats.pt'
raise RuntimeError(f"Failed to load w2v-bert-2.0 model: {e}")
semantic_model.eval()
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
stat_mean_var = torch.load(path_, weights_only=True)
stat_mean_var = torch.load(path_)
semantic_mean = stat_mean_var["mean"]
semantic_std = torch.sqrt(stat_mean_var["var"])
return semantic_model, semantic_mean, semantic_std
@@ -43,8 +43,7 @@ def dvae_wav_to_mel(
mel = mel_stft(wav)
mel = torch.log(torch.clamp(mel, min=1e-5))
if mel_norms is None:
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
mel_norms = torch.load(mel_norms_file, map_location=device, weights_only=True)
mel_norms = torch.load(mel_norms_file, map_location=device)
mel = mel / mel_norms.unsqueeze(0).unsqueeze(-1)
return mel
+1 -4
View File
@@ -9,8 +9,6 @@ from utils.hf_download_logging import quiet_hf_download_logs
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
class MossSoundEffectV2Downloader:
MODEL_NAME = "MOSS-SoundEffect-v2.0"
REPO_ID = "OpenMOSS-Team/MOSS-SoundEffect-v2.0"
@@ -47,7 +45,6 @@ class MossSoundEffectV2Downloader:
os.path.isfile(os.path.join(model_dir, rel_path)) for rel_path in cls.REQUIRED_FILES
)
@validate_model_paths
def resolve_model_path(self, identifier: str = MODEL_NAME) -> str:
identifier = identifier or self.MODEL_NAME
if os.path.isabs(identifier):
@@ -82,7 +79,7 @@ class MossSoundEffectV2Downloader:
with quiet_hf_download_logs():
snapshot_download(
repo_id=self.REPO_ID,
local_dir=allowed_path(model_dir),
local_dir=model_dir,
force_download=force,
)
except Exception as exc:
@@ -1,4 +1,3 @@
from utils.security.data_loading import load_numpy_data # TTS Audio Suite patch: NumPy dictionaries must not execute pickle globals.
import math
from dataclasses import dataclass
from pathlib import Path
@@ -49,7 +48,7 @@ class DACFile:
@classmethod
def load(cls, path):
artifacts = load_numpy_data(path)[()]
artifacts = np.load(path, allow_pickle=True)[()]
codes = torch.from_numpy(artifacts["codes"].astype(int))
if artifacts["metadata"].get("dac_version", None) not in SUPPORTED_VERSIONS:
raise RuntimeError(
+1 -4
View File
@@ -14,8 +14,6 @@ from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_down
from engines.moss_tts.model_specs import MOSS_CODEC_SPECS, MOSS_MODEL_SPECS
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
class MossTTSDownloader:
"""Resolve and download official MOSS-TTS model folders."""
@@ -35,7 +33,6 @@ class MossTTSDownloader:
self.base_path = base_path
os.makedirs(self.base_path, exist_ok=True)
@validate_model_paths
def resolve_model_path(self, model_identifier: str) -> str:
"""Resolve local: names, absolute paths, or known MOSS model identifiers."""
if not model_identifier:
@@ -95,7 +92,7 @@ class MossTTSDownloader:
with quiet_hf_download_logs():
snapshot_download(
repo_id=repo_id,
local_dir=allowed_path(model_dir),
local_dir=model_dir,
force_download=force,
)
except Exception as e:
+29 -31
View File
@@ -4,8 +4,6 @@ Shared helpers for MOSS-TTS training.
from __future__ import annotations
from utils.security.path_access import allowed_path
from utils.security.path_access import resolve_input_path, allowed_path, child_path
import hashlib
import json
import os
@@ -44,22 +42,8 @@ def slugify(value: str) -> str:
def load_jsonl(path: str | Path) -> List[Dict[str, Any]]:
path = Path(allowed_path(path, models=False))
with open(path, "r", encoding="utf-8") as handle:
records = [json.loads(line) for line in handle if line.strip()]
for record in records:
for field in ("audio", "ref_audio", "reference_audio", "reference"):
value = record.get(field)
def resolve(value):
if value is None:
return None
raw = Path(value)
return allowed_path(raw if raw.is_absolute() else path.parent / raw, models=False)
if isinstance(value, str):
record[field] = resolve(value)
elif isinstance(value, list) and all(item is None or isinstance(item, str) for item in value):
record[field] = [resolve(item) for item in value]
return records
return [json.loads(line) for line in handle if line.strip()]
def dump_jsonl(records: Iterable[Dict[str, Any]], path: str | Path) -> None:
@@ -70,18 +54,34 @@ def dump_jsonl(records: Iterable[Dict[str, Any]], path: str | Path) -> None:
handle.write(json.dumps(record, ensure_ascii=False) + "\n")
def resolve_manifest_path(dataset_source: str) -> Path:
path = Path(resolve_input_path(dataset_source, datasets=True, models=False))
if not (path.is_file()):
raise FileNotFoundError(f"Dataset source not found: {dataset_source}")
return path
def resolve_manifest_path(dataset_source: str) -> str:
raw = os.path.expanduser(str(dataset_source or "").strip())
if not raw:
raise ValueError("dataset_source is required")
candidates = [
raw,
os.path.join(folder_paths.get_input_directory(), raw),
os.path.join(folder_paths.get_input_directory(), "datasets", raw),
]
for candidate in candidates:
if os.path.isfile(candidate):
return os.path.abspath(candidate)
raise FileNotFoundError(f"MOSS training manifest not found: {dataset_source}")
def _resolve_dataset_source_path(dataset_source: str) -> Path:
path = Path(resolve_input_path(dataset_source, datasets=True, models=False))
if not (path.is_file() or path.is_dir()):
raise FileNotFoundError(f"Dataset source not found: {dataset_source}")
return path
raw = os.path.expanduser(str(dataset_source or "").strip())
if not raw:
raise ValueError("dataset_source is required")
input_dir = folder_paths.get_input_directory()
candidates = [Path(raw), Path(input_dir, raw), Path(input_dir, "datasets", raw)]
for candidate in candidates:
if candidate.is_file() or candidate.is_dir():
return candidate.resolve()
raise FileNotFoundError(f"MOSS dataset source not found: {dataset_source}")
def _build_manifest_from_audio_folder(dataset_dir: Path, recursive: bool) -> str:
@@ -102,8 +102,6 @@ def _build_manifest_from_audio_folder(dataset_dir: Path, recursive: bool) -> str
if not transcript_path.is_file():
missing_transcripts.append(str(audio_path.relative_to(dataset_dir)))
continue
audio_path = Path(allowed_path(audio_path, models=False))
transcript_path = Path(allowed_path(transcript_path, models=False))
transcript = transcript_path.read_text(encoding="utf-8-sig").strip()
if not transcript:
missing_transcripts.append(str(audio_path.relative_to(dataset_dir)))
@@ -243,7 +241,7 @@ def resolve_continue_from_adapter_path(continue_from: Any) -> str:
if not value:
return ""
if os.path.isdir(value):
return allowed_path(value)
return value
raise FileNotFoundError(f"MOSS continue_from adapter path not found: {value}")
if isinstance(continue_from, dict):
@@ -253,12 +251,12 @@ def resolve_continue_from_adapter_path(continue_from: Any) -> str:
raise ValueError("continue_from TRAINING_ARTIFACTS must come from a MOSS training run")
adapter_path = str(continue_from.get("model_path", "") or "").strip()
if adapter_path and os.path.isdir(adapter_path):
return allowed_path(adapter_path)
return adapter_path
lora_info = continue_from.get("lora_adapter")
if isinstance(lora_info, dict):
adapter_path = str(lora_info.get("adapter_path", "") or "").strip()
if adapter_path and os.path.isdir(adapter_path):
return allowed_path(adapter_path)
return adapter_path
raise FileNotFoundError("continue_from TRAINING_ARTIFACTS does not contain a valid MOSS adapter path")
raise ValueError(
+1 -4
View File
@@ -14,8 +14,6 @@ from utils.hf_download_logging import quiet_hf_download_logs
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
class OmniVoiceDownloader:
"""Resolve and download official OmniVoice model folders."""
@@ -46,7 +44,6 @@ class OmniVoiceDownloader:
self.base_path = base_path
os.makedirs(self.base_path, exist_ok=True)
@validate_model_paths
def resolve_model_path(self, model_identifier: str) -> str:
"""Resolve absolute paths, local: names, or known official identifiers."""
if not model_identifier:
@@ -103,7 +100,7 @@ class OmniVoiceDownloader:
with quiet_hf_download_logs():
snapshot_download(
repo_id=repo_id,
local_dir=allowed_path(model_dir),
local_dir=model_dir,
local_dir_use_symlinks=False,
resume_download=True,
force_download=force,
@@ -109,8 +109,7 @@ def log_mel_spectrogram(
def get_T_after_cnn(L_in, dilation=1):
# TTS Audio Suite patch: this architecture list is constant data.
for (padding, kernel_size, stride) in [(1, 3, 1), (1, 3, 2)]:
for (padding, kernel_size, stride) in eval("[(1,3,1)] + [(1,3,2)] "):
L_out = L_in + 2 * padding - dilation * (kernel_size - 1) - 1
L_out = 1 + L_out // stride
L_in = L_out
+1 -4
View File
@@ -21,8 +21,6 @@ from utils.hf_download_logging import quiet_hf_download_logs
import folder_paths
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
class Qwen3ASRDownloader:
"""Downloader for Qwen3-ASR models using Hugging Face snapshot downloads."""
@@ -87,7 +85,7 @@ class Qwen3ASRDownloader:
with quiet_hf_download_logs():
snapshot_download(
repo_id=repo_id,
local_dir=allowed_path(model_dir),
local_dir=model_dir,
local_dir_use_symlinks=False,
resume_download=True
)
@@ -100,7 +98,6 @@ class Qwen3ASRDownloader:
print(f"\n❌ Download failed: {e}")
raise
@validate_model_paths
def resolve_model_path(self, model_identifier: str) -> str:
if not model_identifier:
model_identifier = "Qwen3-ASR-1.7B"
+1 -4
View File
@@ -22,8 +22,6 @@ from utils.hf_download_logging import quiet_hf_download_logs
from utils.models.extra_paths import get_preferred_download_path, get_all_tts_model_paths
import folder_paths
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
class Qwen3TTSDownloader:
"""Downloader for Qwen3-TTS models using unified download system."""
@@ -225,7 +223,7 @@ class Qwen3TTSDownloader:
with quiet_hf_download_logs():
snapshot_download(
repo_id=repo_id,
local_dir=allowed_path(model_dir),
local_dir=model_dir,
local_dir_use_symlinks=False,
resume_download=True
)
@@ -280,7 +278,6 @@ class Qwen3TTSDownloader:
print(f"\n✅ Essential models ready!")
@validate_model_paths
def resolve_model_path(self, model_identifier: str) -> str:
"""
Resolve model path handling "local:" prefix and auto-download.
+1 -1
View File
@@ -305,7 +305,7 @@ def _convert_bin_to_safetensors(bin_path: str, safetensors_path: str) -> bool:
# Load the .bin file
print(f"📂 Loading {os.path.basename(bin_path)}...")
state_dict = torch.load(bin_path, map_location="cpu", weights_only=True)
state_dict = torch.load(bin_path, map_location="cpu")
# Save as .safetensors
print(f"💾 Saving as {os.path.basename(safetensors_path)}...")
+1 -2
View File
@@ -567,8 +567,7 @@ class MusicSeparationModel:
if model['Stem'] == "BOTH":
device = torch.device(self.GPU_device)
mdx23 = tfc_tdf.TFC_TDF_net(self.MDX23_config)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
mdx23.load_state_dict( torch.load(model['PATH'], map_location = device, weights_only=True) )
mdx23.load_state_dict( torch.load(model['PATH'], map_location = device) )
mdx23 = mdx23.to(device)
mdx23.eval()
+1 -2
View File
@@ -309,8 +309,7 @@ class MDX23CSeparator:
model = TFC_TDF_net(**model_config)
# Load weights
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
checkpoint = torch.load(model_path, map_location=self.device, weights_only=True)
checkpoint = torch.load(model_path, map_location=self.device, weights_only=False)
# Handle different checkpoint formats
if 'state_dict' in checkpoint:
+2 -4
View File
@@ -68,8 +68,7 @@ def load_hubert(model_path: str, config):
print(f"🔧 Attempting direct .pt loading: {model_path}")
try:
import torch
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
checkpoint = torch.load(model_path, map_location='cpu', weights_only=True)
checkpoint = torch.load(model_path, map_location='cpu', weights_only=False)
# Try to create HuBERT model directly from .pt checkpoint
# This will work if the checkpoint contains the full model with config
@@ -186,8 +185,7 @@ def load_hubert(model_path: str, config):
with open(model_path, 'rb') as f:
# Load raw data and try to extract model state_dict only
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
checkpoint = torch.load(f, map_location='cpu', weights_only=True)
checkpoint = torch.load(f, map_location='cpu', weights_only=False)
# Extract state_dict from various possible formats
if hasattr(checkpoint, 'state_dict'):
+1 -2
View File
@@ -641,8 +641,7 @@ class RMVPE:
self.model = ort_session
else:
model = E2E(4, 1, (2, 2))
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
ckpt = torch.load(model_path, map_location="cpu", weights_only=True)
ckpt = torch.load(model_path, map_location="cpu")
model.load_state_dict(ckpt)
model.eval()
if is_half == True:
+1 -2
View File
@@ -562,8 +562,7 @@ class SCNetSeparator:
model = SCNet(**model_config)
# Load weights
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
checkpoint = torch.load(model_path, map_location=self.device, weights_only=True)
checkpoint = torch.load(model_path, map_location=self.device, weights_only=False)
# Handle different checkpoint formats
if 'state_dict' in checkpoint:
+2 -4
View File
@@ -42,8 +42,7 @@ class UVR5Base:
}
mp = ModelParameters(os.path.join(dir_path,"uvr5_pack","vr_network","modelparams","4band_v2.json"))
model = CascadedASPPNet(mp.param["bins"] * 2)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
cpk = torch.load(model_path, map_location=self.device, weights_only=True)
cpk = torch.load(model_path, map_location=self.device, weights_only=False)
try:
model.load_state_dict(cpk)
except RuntimeError as e:
@@ -300,8 +299,7 @@ class UVR5New(UVR5Base):
mp = ModelParameters(os.path.join(dir_path,"uvr5_pack","vr_network","modelparams","4band_v3.json"))
nout = 64 if dereverb else 48
model = CascadedNet(mp.param["bins"] * 2, nout)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
cpk = torch.load(model_path, map_location=self.device, weights_only=True)
cpk = torch.load(model_path, map_location=self.device, weights_only=False)
try:
model.load_state_dict(cpk)
except RuntimeError as e:
@@ -115,8 +115,7 @@ def main():
optimizer = th.optim.Adam(model.parameters(), lr=args.lr)
try:
# TTS Audio Suite patch: restrict checkpoint deserialization.
saved = th.load(checkpoint, map_location='cpu', weights_only=True)
saved = th.load(checkpoint, map_location='cpu')
except IOError:
saved = SavedState()
else:
@@ -18,10 +18,9 @@ from torch.nn import functional as F
import tqdm
import tkinter as tk
# TTS Audio Suite patch: resolve trusted checkpoint architectures within this bundled package.
from .demucs import Demucs
from .hdemucs import HDemucs
from .utils import center_trim, DummyPoolExecutor
from demucs import Demucs
from hdemucs import HDemucs
from utils import center_trim, DummyPoolExecutor
Model = tp.Union[Demucs, HDemucs]
@@ -12,9 +12,8 @@ import torch
from torch import nn
from torch.nn import functional as F
# TTS Audio Suite patch: resolve trusted checkpoint architectures within this bundled package.
from .states import capture_init
from .utils import center_trim, unfold
from states import capture_init
from utils import center_trim, unfold
class BLSTM(nn.Module):
@@ -12,11 +12,10 @@ import typing as tp
import torch
from torch import nn
from torch.nn import functional as F
# TTS Audio Suite patch: resolve trusted checkpoint architectures within this bundled package.
from .filtering import wiener
from .demucs import DConv, rescale_module
from .states import capture_init
from .spec import spectro, ispectro
from filtering import wiener
from demucs import DConv, rescale_module
from states import capture_init
from spec import spectro, ispectro
def pad1d(x: torch.Tensor, paddings: tp.Tuple[int, int], mode: str = 'constant', value: float = 0.):
"""Tiny wrapper around F.pad, just to allow for reflect padding on small input.
@@ -9,20 +9,19 @@ This code contains the spectrogram and Hybrid version of Demucs.
"""
import math
# TTS Audio Suite patch: resolve trusted checkpoint architectures within this bundled package.
from .filtering import wiener
from filtering import wiener
import torch
from torch import nn
from torch.nn import functional as F
from fractions import Fraction
from einops import rearrange
from .transformer import CrossTransformerEncoder
from transformer import CrossTransformerEncoder
from .demucs import rescale_module
from .states import capture_init
from .spec import spectro, ispectro
from .hdemucs import pad1d, ScaledEmbedding, HEncLayer, MultiWrap, HDecLayer
from demucs import rescale_module
from states import capture_init
from spec import spectro, ispectro
from hdemucs import pad1d, ScaledEmbedding, HEncLayer, MultiWrap, HDecLayer
class HTDemucs(nn.Module):
@@ -9,8 +9,7 @@ import math
import torch as th
from torch import nn
# TTS Audio Suite patch: resolve trusted checkpoint architectures within this bundled package.
from .utils import capture_init, center_trim
from utils import capture_init, center_trim
class BLSTM(nn.Module):
@@ -17,13 +17,12 @@ import logging
from diffq import DiffQuantizer
import torch.hub
# TTS Audio Suite patch: resolve trusted checkpoint architectures within this bundled package.
from .model import Demucs
from .tasnet_v2 import ConvTasNet
from .utils import set_state
from model import Demucs
from tasnet_v2 import ConvTasNet
from utils import set_state
from .hdemucs import HDemucs
from .repo import RemoteRepo, LocalRepo, ModelOnlyRepo, BagOnlyRepo, AnyModelRepo, ModelLoadingError # noqa
from hdemucs import HDemucs
from repo import RemoteRepo, LocalRepo, ModelOnlyRepo, BagOnlyRepo, AnyModelRepo, ModelLoadingError # noqa
logger = logging.getLogger(__name__)
ROOT_URL = "https://dl.fbaipublicfiles.com/demucs/mdx_final/"
@@ -134,8 +133,7 @@ def load_pretrained(name):
def _load_state(name, model, quantizer=None):
url = get_url(name)
# TTS Audio Suite patch: restrict checkpoint deserialization.
state = torch.hub.load_state_dict_from_url(url, map_location='cpu', check_hash=True, weights_only=True)
state = torch.hub.load_state_dict_from_url(url, map_location='cpu', check_hash=True)
set_state(model, quantizer, state)
if quantizer:
quantizer.detach()
+4 -13
View File
@@ -15,9 +15,8 @@ import typing as tp
import torch
import yaml
# TTS Audio Suite patch: resolve trusted checkpoint architectures within this bundled package.
from .apply import BagOfModels, Model
from .states import load_model
from apply import BagOfModels, Model
from states import load_model
AnyModel = tp.Union[Model, BagOfModels]
@@ -62,16 +61,8 @@ class RemoteRepo(ModelOnlyRepo):
url = self._models[sig]
except KeyError:
raise ModelLoadingError(f'Could not find a pre-trained model with signature {sig}.')
# TTS Audio Suite patch: restrict checkpoint deserialization.
# TTS Audio Suite patch: download normally, then restrict legacy metadata loading.
from urllib.parse import urlsplit
filename = Path(urlsplit(url).path).name
cached = Path(torch.hub.get_dir()) / "checkpoints" / filename
if not cached.is_file():
cached.parent.mkdir(parents=True, exist_ok=True)
checksum = torch.hub.HASH_REGEX.search(filename)
torch.hub.download_url_to_file(url, str(cached), hash_prefix=checksum.group(1) if checksum else None)
return load_model(cached)
pkg = torch.hub.load_state_dict_from_url(url, map_location='cpu', check_hash=True)
return load_model(pkg)
class LocalRepo(ModelOnlyRepo):
@@ -34,16 +34,6 @@ def get_quantizer(model, args, optimizer=None):
return quantizer
# TTS Audio Suite patch: preserve legacy architecture metadata without allowing
# pickle to construct or import executable model objects.
def _load_model_package(path):
from utils.security.demucs_loading import load_demucs_package
return load_demucs_package(path, map_location="cpu",
architecture_package=__package__ or "",
architecture_directory=Path(__file__).parent)
def load_model(path_or_package, strict=False):
"""Load a model from the given serialized model, either given as a dict (already loaded)
or a path to a file on disk."""
@@ -53,8 +43,7 @@ def load_model(path_or_package, strict=False):
with warnings.catch_warnings():
warnings.simplefilter("ignore")
path = path_or_package
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
package = _load_model_package(path)
package = torch.load(path, 'cpu')
else:
raise ValueError(f"Invalid type for {path_or_package}.")

Some files were not shown because too many files have changed in this diff Show More