Compare commits

..
Author SHA1 Message Date
diodiogod f7693bc196 Version 5.10.4
Publish IndexTTS memory management fixes

- Include IndexTTS-2 and IndexTTS-2.5 unload improvements from PR #373.
- Preserve IndexTTS-2 low-VRAM placement after model reloads.
- Release and rebuild optional GPT acceleration state across device changes.
- Include passing Fish Audio test mocks and CUDA lifecycle regression coverage.
2026-10-17 09:44:42 -03:00
diodiogod c97ea371db Fix IndexTTS model reload and acceleration cleanup
- Preserve backend device placement through the IndexTTS wrapper handler.
- Release accelerated GPT weights, KV caches, and CUDA graphs before device
  or dtype changes, then rebuild acceleration when generation resumes.
- Reuse a capture stream on each GPU to avoid accumulating CUDA workspaces.
- Restore mocked ComfyUI user-directory APIs for Fish Audio unit tests.

Validation:
- 168 unit tests passed.
- Live IndexTTS-2 low-VRAM generation, unload, and generation passed.
- CUDA capture, replay, cleanup, and rebuild checks passed with lightweight
  models on both GPUs. Full FlashAttention speech was not exercised.
2026-10-17 09:44:21 -03:00
diodiogod 2a916e3ffc Merge pull request #373 from wandermyz/fix/indextts2-to-device
Fix IndexTTS-2 leaving ~2 GB on the GPU after unload
2026-10-17 08:58:29 -03:00
wandermyzandClaude Opus 5.5 828eb754ac Apply the unload fix to IndexTTS-2 and load QwenEmotion on one device
Review feedback on the IndexTTS-2.5 fix: IndexTTS-2 needs the same
complete unload, and QwenEmotion's device_map="auto" placement (Accelerate
hooks, CPU offload when VRAM is short) breaks moving it with .to().

- Share the module + plain-tensor move as indextts.utils.module_device.
- Add IndexTTS2.to() to the IndexTTS-2 backend. In low VRAM mode the
  models keep resting on CPU; only QwenEmotion follows a GPU target.
- Load IndexTTS-2.5's QwenEmotion on a single device, as IndexTTS-2 does,
  and give both QwenEmotion classes a to() that re-selects the GPU.
- Only set a torch.device on the engine in the adapter's fallback path;
  the backends' own to() keeps self.device a string.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 00:27:48 -07:00
wandermyzandClaude Opus 5.5 3539778b2b Fix IndexTTS-2 leaving ~2 GB on the GPU after unload
ComfyUI unloads the wrapped engine with model.to("cpu"). IndexTTS2 had no
to(), so only its nn.Module attributes moved. QwenEmotion, plain tensor
attributes inside modules (s2mel causal_mask/freqs_cis, old-style
weight_norm weights), the emotion/speaker matrices and the cached
reference-voice conditioning stayed on the GPU.

Add IndexTTS2.to(), like the chatterbox engines have.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 21:21:17 -07:00
diodiogod 3f3832a088 Refresh character switching guide and separate F5 reference advice 2026-10-16 09:43:43 -03:00
diodiogod 2feb675d56 Version 5.10.3
Release MOSS-TTS live quantization

- Publish optional BitsAndBytes INT8 and NF4 loading from existing checkpoints.
- Preserve existing workflow widget positions and floating-point defaults.
- Include quantized model lifecycle handling and cache separation across MOSS consumers.

Related to #371; keep the issue open for reporter testing.
2026-10-15 15:00:07 -03:00
diodiogod 55f1464c99 Add MOSS-TTS INT8 and NF4 load-time quantization
- Quantize the language backbone from existing checkpoints with BitsAndBytes.
- Propagate the setting through speech, dialogue, voice design, sound effects,
  model identity, and generated-audio caching.
- Release and reload quantized weights through ComfyUI model management while
  keeping the audio tokenizer on CPU.
- Append the optional widget without shifting saved workflow values.

Validation: nine targeted tests, CUDA smoke checks for all three bundled MOSS
architectures, quantized unload/reload checks, and maintainer audio testing.
Address #371; reporter validation is still requested.
2026-10-15 14:59:50 -03:00
diodiogod 49538a481c Version 5.10.2
Improve Voice Capture recording and preview workflow

- Record and save microphone clips without queueing the workflow.
- Share Character Voices playback and trimming controls with Voice Capture.
- Use saved clips and selected ranges only when the user runs the workflow.
- Remove unused preview space and allow the node to shrink to its content.

Validation: 31 targeted tests, live ComfyUI checks, and user confirmation.
2026-10-10 13:16:52 -03:00
diodiogod a866f02ef9 Decouple Voice Capture recording from workflow execution
- Save microphone clips independently and use the selected clip when queued.
- Reuse Character Voices playback, waveform, and trimming controls.
- Size the recorder preview to its content so the node can shrink vertically.
- Preserve legacy capture settings and invalidate cached audio when files change.

Validation: 31 targeted tests and live recording, stopping, preview, trimming,
caching, and resizing checks. User confirmed the updated UI works.
2026-10-10 13:16:40 -03:00
diodiogod f0f7cf849a Version 5.10.1
Fix Voice Capture recording cache

- Release the merged PR #369 change to re-record live input on every execution.
- Preserve the existing voice_trigger input for saved workflow compatibility.
2026-10-10 11:26:15 -03:00
diodiogod 67759a19c3 Merge pull request #369 from zhaoruizhe81/fix/voicecapture-is-changed
Voice Capture: implement IS_CHANGED to always re-record
2026-10-10 11:09:30 -03:00
zhaoruizhe 0e0dce92cb Voice Capture: implement IS_CHANGED to always re-record
ChatterBoxVoiceCapture records live audio from an input device, so its
output can never be reproduced from its inputs. Without IS_CHANGED,
ComfyUI's cache hits whenever all widget inputs are unchanged and the
node silently replays the previous recording.

Return float("nan") from IS_CHANGED: NaN != NaN makes the cache always
miss and forces a fresh recording on every queue. voice_trigger is kept
for existing-workflow compatibility.

Add a unit test following the repo's direct-load pattern (same as
test_srt_parser.py) asserting the hook exists and never compares equal.

Fixes #343
2026-10-10 17:04:23 +08:00
diodiogod be3c19e9f1 Version 5.10.0
Release Shared Runtime installation controls and suite security improvements

- Prepare Shared Runtime during installation instead of workflow execution.
- Add persistent installation preferences and migrate legacy Dedicated Runtime selections.
- Restrict file paths, checkpoint deserialization, and configuration expressions.
- Correct Registry archive contents and document runtime installation and repair.
2026-10-09 14:16:22 -03:00
diodiogod 58678e2497 Prepare TTS Audio Suite for Registry security review
- Move Shared Runtime preparation to the installation hook and add persistent UI/file opt-out.
- Restrict file and model download paths, checkpoint loading, and configuration parsing across the suite.
- Remove Dedicated Runtime choices and migrate saved workflows without shifting widgets.
- Exclude development files from Registry archives while retaining runtime metadata and Unicode paths.
- Document installation, repair, and manual runtime cleanup.

Validation:
- 59 targeted existing tests, actual fresh runtime installation, and live Step speech/editing.
- File containment, restricted loader compatibility, settings persistence, and archive contents checked.
- Manager queue/restart flow remains untested on the local installation.

Address Registry availability reported in #366.
2026-10-09 14:15:58 -03:00
diodiogod 12a0004daf Version 5.9.3
Release isolated worker import precedence fixes for issue #366

- Prioritize suite imports in the launcher and all six worker entrypoints.
- Preserve host dependency paths across isolated worker directory changes.
- Defer Step implementation imports and correct its encoder device patch.
- Validate first-use conflicts, worker recreation, and live Step editing.
2026-10-08 14:56:20 -03:00
diodiogod 10c1f520bf Fix isolated worker import precedence across TTS engines
Address issue #366:
- Put suite source paths first in worker environments and every entrypoint.
- Resolve inherited host paths before changing the worker working directory.
- Preserve suite precedence when runtime profiles override PYTHONPATH.
- Defer Step device patch loading until its synthesis implementation is imported.
- Correct the Step patch's bundled encoder class and parameter device lookup.
- Add subprocess regressions for six workers and relative host dependencies.

Validation: 22 focused tests, original failure reproduced against HEAD,
Qwen first-use startup and three recreation cycles after Step imports,
and live Step generation/editing before and after a memory-unload request.
2026-10-08 14:55:56 -03:00
diodiogod 85da51e0e2 Version 5.9.2
Fix Classic ChatterBox short-segment crashes

Include Sam Ayoub's S3Gen speech-token bounds fix from PR #368.
Retire padding and segment-skipping workarounds in Classic Text, SRT,
and streaming paths while preserving legacy workflow inputs.
Reject empty speech-token streams before decoding in both ChatterBox engines.

Validation: 42 existing tests, 39 focused checks, five real CUDA workflows,
and successful user testing.
2026-10-04 09:24:14 -03:00
diodiogod 1f040d129d Retire Classic ChatterBox crash workarounds after PR #368
Remove short-text padding, the protection widget, heuristic segment skipping,
and the CUDA-to-silence fallback from Classic ChatterBox Text and SRT paths.
Keep obsolete protection arguments accepted and ignored for legacy workflows.
Reject empty speech-token streams before decoding in both ChatterBox engines.

Validation: 42 existing tests, 39 focused checks, and five real CUDA workflows
passed, including the original token-7134 crash reproduction. User confirmed
that the implementation works perfectly.
2026-10-04 09:23:45 -03:00
diodiogod e3babbd01c Merge pull request #368 from melayyoub/fix/s3gen-speech-token-bounds
fix(chatterbox): bound S3Gen speech-token ids to stop a CUDA device-side abort
2026-10-04 06:58:36 -03:00
diodiogod 6311c33aef Version 5.9.1
Fix ChatterBox Classic multilingual cache lookup

- Share cache construction between adapter generation and early cache checks
- Reuse cached tensors without loading or consulting a different language model
- Load only uncached language groups and preserve narrator alias handling
- Share the cache builder with the F5-TTS adapter and honor disabled pause caching

Address PR #364 with a complete cache lookup fix.
Validation: 26 model-free cache and orchestration regression checks passed.
2026-10-04 06:20:05 -03:00
diodiogod f986a3253a Version 5.9.0
Add Step Audio EditX checkpoint selection and expanded sound editing

- Release pinned 2026-01-23 and 2025-11-28 checkpoint choices without changing
  engine widget order or replacing existing unversioned weights.
- Expand shared sound parsing, editor menus, emotion choices, and help.
- Use the selected checkpoint in Audio Editor loading and result caches.
- Preserve native ChatterBox and CosyVoice tag handling.
2026-10-03 14:58:29 -03:00
diodiogod ecbc149415 Add dated Step Audio EditX checkpoints and expanded tags
- Pin January and November model revisions in separate download folders.
- Preserve existing model selections and the engine widget order.
- Expand sound tags, emotion controls, and Multiline TTS Tag Editor support.
- Respect checkpoint selection in Audio Editor loading and edit-result caching.
- Preserve native ChatterBox and CosyVoice sound tags during Step parsing.
- Update model documentation, editor help, and generated engine tables.

Validation: 42 pause/SRT unit tests, checkpoint and frontend audits, and live
TTS, Audio Editor, and SRT generation in the configured ComfyUI environment.
2026-10-03 14:57:28 -03:00
diodiogod d12a4c286f Version 5.8.11
Release Multiline TTS Tag Editor globe language picker

Release details:
- Bump patch version after user-confirmed UI testing
- Document the reusable animated language picker and proportional resizing
- Preserve existing language tag insertion and swap behavior
2026-09-19 19:10:31 -03:00
diodiogod 1bf42ded63 Add globe language picker to Multiline TTS Tag Editor
Implementation details:
- Add a lightweight animated globe backed by simplified Natural Earth coastline data
- Integrate the picker with inline language swaps and the node sidebar language control
- Support dynamic language lists, landmark collision avoidance, and selected-language restoration
- Add square language tiles and persistent pointer-captured proportional resizing
- Include a standalone reusable prototype for visual refinement
2026-09-19 19:10:17 -03:00
Sam Ayoub d9459ff14d fix(chatterbox): bound S3Gen speech-token ids to stop a CUDA abort
ChatterBox TTS can kill the whole ComfyUI process mid-generation with

    IndexKernelUtils.cu:16: block: [...] Assertion `ind >=0 && ind <
    ind_dim_size && "vectorized gather kernel index out of bounds"` failed.
    Fatal Python error: Aborted

The traceback points at s3gen/transformer/embedding.py in extend_pe, which
is a red herring: CUDA reports a device-side assert at the next
synchronising call, and extend_pe just happens to copy its cached `pe`
host-to-device on the first forward.

The actual gather is flow.input_embedding, an nn.Embedding(6561, 512):

- T3's speech_head spans speech_tokens_dict_size (8194) logits, so
  torch.multinomial can sample any id in [0, 8193].
- drop_invalid_tokens only slices between the SoS/EoS markers (6561/6562)
  and never bounds-checks what is left.
- The call site clamps with min=0 but no max.

So a sampled id in 6563..8193 that lands before EoS is passed straight to
the embedding and indexes past the table. It depends on the sampled token,
which is why it shows up as a rare, seed-dependent crash rather than a
consistent failure.

Two changes:

- drop_invalid_tokens drops out-of-vocabulary ids (with a warning) in both
  the chatterbox and chatterbox_official_23lang engines. Dropping rather
  than clamping is deliberate: an id >= vocab carries no meaning for S3Gen,
  and clamping would splice in an arbitrary phone instead of losing a 40 ms
  frame.
- flow.py clamps the embedding index at both ends as a backstop. This
  mirrors what engines/chatterbox_official_23lang/models/s3gen/flow.py
  already does on all three call sites; the older chatterbox engine had
  been left behind.

Verified against the flow's own vocab (6561): SoS/EoS handling is
unchanged, clean token streams pass through untouched, and out-of-range ids
are removed. Both engines' modules still import cleanly.

Unrelated and left alone: the 2-D branch that the assert allows
(shape [1, T]) slices dim 0 rather than dim 1, so it returns an empty
tensor. That predates this change and the ChatterBox call sites all pass
1-D input.
2026-09-05 22:38:09 -04:00
diodiogod 41be776d9a Version 5.8.10
Fix Step Audio EditX companion model setup and installer requirements

Technical details:
- Download the bundled FunASR companion model for isolated Step Audio EditX runtimes
- Reuse companion model validation across native and isolated loading paths
- Remove installer-managed and unused packages from root requirements
- Prevent external FunASR from forcing incompatible NumPy constraints

Fixes #335
2026-09-04 23:40:10 -03:00
diodiogod b6087f2a33 Version 5.8.9
Fix TTS model discovery with extra model paths

Technical details:
- Append the local ComfyUI models/TTS directory when YAML already registered TTS
- Preserve ComfyUI's configured default path ordering for downloads
- Normalize paths to prevent duplicate registrations
- Address issue #367
2026-09-03 08:57:17 -03:00
diodiogod da220f232a Version 5.8.8
Restrict legacy parallel workers to ChatterBox Classic

Technical details:
- Centralize batch-size policy in the streaming coordinator
- Force unsupported TTS engines to sequential processing
- Update Unified TTS Text and SRT tooltips with accurate scope and performance guidance
- Remove the redundant F5-TTS-specific fallback
2026-08-31 10:25:21 -03:00
diodiogod fab00263fb Version 5.8.7
Run VibeVoice and KugelAudio natively on Transformers 5

Technical details:
- Add compatibility shims for Transformers 5 tokenizer, registration, generation, cache, dtype, and weight-tying APIs
- Materialize VibeVoice and KugelAudio diffusion scheduler state on CPU during meta-device model construction
- Remove the VibeVoice runtime selector and force legacy workflow values onto the main environment
- Update engine metadata and documentation to reflect native Transformers 5 support
- Suppress expected VibeVoice tied-weight and compatibility warnings
2026-08-28 15:06:49 -03:00
diodiogod d7758c968c Version 5.8.6
Fix Vocal Removal cache hashing crash

Technical details:
- Hash NumPy audio samples directly instead of encoding an in-memory WAV
- Include sample rate, dtype, and shape in the cache identity
- Avoid SoundFile virtual I/O and its libffi callback path

Fixes #354
2026-08-28 10:31:57 -03:00
diodiogod 4dec421d62 Version 5.8.5
Release nested RVC model folder support

Technical details:
- Bump the project patch version after recursive RVC discovery was verified
- Document support for relative model and index paths
- Address issue #129
2026-08-26 11:52:34 -03:00
diodiogod 8264a3ad17 Support nested RVC model folders
- Discover RVC model and index files recursively
- Preserve relative paths in model and index selectors
- Auto-pair indexes stored beside nested models

Address issue #129
2026-08-26 11:52:23 -03:00
diodiogod f8faab1215 Version 5.8.4
Fix IndexTTS DeepSpeed compatibility and workflow defaults

- Disable DeepSpeed in the shipped IndexTTS example workflow
- Skip the unsupported DeepSpeed kernel path with Transformers 5.x
- Fall back to standard IndexTTS inference when DeepSpeed initialization fails
2026-08-21 15:30:59 -03:00
diodiogod b466b12ce4 Fix IndexTTS DeepSpeed workflow defaults
- Disable DeepSpeed in the shipped IndexTTS example workflow\n- Skip the unsupported DeepSpeed kernel path with Transformers 5.x\n- Fall back to standard IndexTTS inference when DeepSpeed initialization fails
2026-08-21 15:30:48 -03:00
diodiogod e70ea77eb8 Version 5.8.3
Fix ChatterBox alignment analyzer lifecycle

Technical details:
- Stop installing the unused English ChatterBox attention wrapper on every generation
- Remove Official 23-Lang v1/v2 attention hooks after each inference
- Restore transformer attention configuration during analyzer cleanup
- Address issue #349
2026-08-18 22:32:13 -03:00
diodiogod dedd982ab9 Version 5.8.2
Fix MOSS Voice Acting LoRA controls

Technical details:
- Enable MOSS LoRA widgets for the LAION Voice Acting 8B model
- Preserve existing backend adapter loading behavior
- Address the UI inconsistency reported in issue #340
2026-08-18 19:03:37 -03:00
diodiogod 78a9eff4ec Merge MOSS Voice Acting LoRA UI fix 2026-08-18 19:03:27 -03:00
diodiogod 103d34c209 Fix MOSS Voice Acting LoRA controls
Address issue #340 by enabling the existing MOSS LoRA widgets when the LAION Voice Acting 8B community checkpoint is selected.
2026-08-18 19:03:26 -03:00
diodiogod 08e8e8c884 Version 5.8.1
Add MOSS community full-checkpoint support

Implementation details:
- Add the LAION MOSS-TTS v1.5 Voice Acting 8B community model to the engine dropdown and downloader
- Discover compatible local MOSS full checkpoints and resolve architecture from config.json
- Enable the LAION Delay checkpoint in the existing LoRA training pipeline
- Reject unsupported local MOSS layouts instead of assuming Local Transformer
- Address issue #340
2026-08-11 23:42:44 -03:00
diodiogod 46c324c1ce Version 5.8.0
Release IndexTTS 2.5 model-version support

Implementation details:
- Add the official multilingual 2.5 backend and pinned model snapshot
- Add explicit language, duration-factor, and text-normalization controls
- Preserve IndexTTS 2.0 compatibility and emotion-control features
- Fix Text and SRT processor invalidation when switching model versions
- Separate generated-audio cache entries by model and 2.5 parameters
2026-08-11 23:08:51 -03:00
diodiogod 2bc4c2a18d Add IndexTTS 2.5 support 2026-08-11 23:08:35 -03:00
diodiogod 923f7b3c32 Version 5.7.0
Release DramaBox LoRA training support

Implementation details:
- Add DramaBox dataset preparation, training configuration, and integrated trainer nodes
- Add official audio-branch preprocessing and IC-LoRA training support
- Add DramaBox LoRA adapter loading with runtime reuse and cache-aware strength changes
- Generalize shared training clip staging and dataset-row handling
- Add DramaBox training documentation and example workflow
2026-08-10 18:08:59 -03:00
diodiogod 95223f4800 Merge DramaBox LoRA training support 2026-08-10 18:08:47 -03:00
diodiogod 9c40f4542b Add DramaBox LoRA training workflow
- Add the ready-to-use DramaBox LoRA training workflow and cover
- List the workflow in the README examples
- Keep DramaBox fine-tuning documentation general and capability-focused
2026-08-10 18:08:19 -03:00
diodiogod bf7d83f26e Improve DramaBox LoRA runtime reuse and documentation 2026-08-10 11:10:23 -03:00
diodiogod b09e5023b5 Add DramaBox LoRA training and adapter loading 2026-08-09 23:50:59 -03:00
diodiogod 586bd96e51 Version 5.6.5
Fix MOSS Dataset Prep workflow compatibility

Technical details:
- Restore the original serialized order of all existing optional widgets
- Append recursive_folder_scan after legacy widget values
- Prevent saved workflows from shifting validation and codec settings
- Address issue #340
2026-08-03 17:47:14 -03:00
diodiogod 127bfe32fc Version 5.6.4
Add MOSS training folder dataset import

Implementation details:
- Accept paired audio and transcript folders in MOSS Dataset Prep
- Generate deterministic cached JSONL manifests without changing existing manifest behavior
- Add optional recursive scanning and validation-folder support
- Address issue #340
2026-08-03 08:10:33 -03:00
diodiogod 2f587b22b3 Version 5.6.3
Prevent installer validation from importing engine runtimes

Technical details:
- Walk dotted module specs without importing parent packages
- Keep presence-only validation isolated from third-party startup checks
- Add regression coverage for import side effects
- Address issue #337
2026-08-01 17:46:46 -03:00
diodiogod 48608775e0 Version 5.6.2
Handle incompatible FlashAttention installs in F5-TTS

Technical details:
- Validate optional FlashAttention submodules before enabling the backend
- Keep F5-TTS on PyTorch attention when FlashAttention is partial or incompatible
- Improve the explicit FlashAttention backend error
- Address issue #334
2026-07-30 19:50:54 -03:00
diodiogod eaaacef869 Version 5.6.1
Allow installer to continue without optional audio libraries

Technical details:
- Treat PortAudio and libsamplerate checks as non-fatal warnings on Linux and macOS
- Allow Fish Audio S2 runtime installation to run in headless environments
- Correct Fedora dependency guidance
- Address issues #330 and #332
2026-07-30 17:44:51 -03:00
diodiogod 871c97fd99 Remove DramaBox cover link from workflow table 2026-07-25 12:31:14 -03:00
diodiogod fa30dc6768 Add DramaBox workflow and quotation highlighting
- Add the DramaBox integration workflow and matching cover art
- Link the workflow from the README engine workflow table
- Highlight complete straight and curly quoted spans in the Multiline TTS Tag Editor
- Preserve existing tag colors and ignore unmatched or multiline quote pairs
2026-07-25 12:30:32 -03:00
diodiogod c28d903d02 Version 5.6.0
Release DramaBox and ChatterBox V3 integration

Implementation details:
- Add DramaBox unified TTS and SRT processors with native duration targeting
- Add DramaBox prompt templates, negative prompting, memory strategies, FP8, compile, cache invalidation, and silence diagnostics
- Add ChatterBox 23-Lang V3 checkpoint loading and output artifact handling
- Update engine metadata, generated documentation, parameter switching, installer dependencies, and model downloads
2026-07-25 11:39:24 -03:00
diodiogod 397982556c Merge DramaBox and ChatterBox V3 support 2026-07-25 11:39:08 -03:00
diodiogod 517d11ed4c Add DramaBox and ChatterBox V3 support
- Integrate DramaBox with unified TTS and SRT generation, duration targeting, prompt templates, memory strategies, FP8, compile, and silence diagnostics
- Add ChatterBox 23-Lang V3 checkpoint support and artifact handling
- Extend parameter switching, generated-audio caching, tag editor controls, model downloads, and engine registration
- Add user documentation, engine metadata, generated comparison tables, and FL-MCP launcher reliability updates
2026-07-25 11:38:52 -03:00
diodiogod 625d22d6f6 Version 5.5.3
Release Character Alias Manager improvements

Technical details:
- Add per-user grouped alias persistence and management API
- Add character preview and missing-transcript metadata
- Integrate alias management with Character Voices and Multiline TTS Tag Editor
- Improve drag feedback, waveform rendering, zoom behavior, discovery refresh, and logging
2026-07-24 23:37:50 -03:00
diodiogod 2ecd43a5e6 Extend project evolution timeline 2026-07-24 23:37:34 -03:00
diodiogod 986093a207 Add character voice previews and improve alias manager drag feedback 2026-07-24 21:57:53 -03:00
diodiogod 237765af9e Add character alias management and refresh support 2026-07-23 14:03:34 -03:00
diodiogod d1bfe54ffd Version 5.5.2
Release IndexTTS emotion import modal fix

Technical details:
- Update the emotion configuration overlay stacking order
- Keep feedback notifications visible above the vector editor and import dialog
2026-07-21 14:05:20 -03:00
diodiogod df34b9818c Fix IndexTTS emotion import modal stacking
Technical details:
- Raise the shared emotion import/export overlay above the vector editor
- Keep import feedback notifications above both modal layers
2026-07-21 14:05:11 -03:00
diodiogod 3d7e8dfaa1 Version 5.5.1
Fix Step Audio EditX generation and runtime compatibility

Technical details:
- Restore the correct Step prompt, token layout, untied audio head, and memory-safe attention behavior
- Add shared and dedicated Transformers 4 runtime routing with worker progress reporting
- Preserve runtime selection across TTS, SRT, caching, and inline audio editing
- Fix character voice handling, empty audio validation, sample-rate interface, and deprecated dependency calls
2026-07-18 20:38:56 -03:00
diodiogod 1d52a51206 Version 5.5.0
Release unified voice design and MOSS sound effects

Implementation details:
- Add MOSS-TTS v1.5 model selection and LoRA training support
- Add MOSS-SoundEffect v1 through the MOSS engine
- Add dedicated MOSS-SoundEffect v2 diffusion engine
- Standardize Voice Designer and Sound Effects provider routing
- Add reusable Save Character Voice persistence
- Refactor Character Voices discovery and compact preview layout
- Remove the legacy Qwen-specific voice designer node
- Add example workflows, guides, model metadata, and cleaner download logging
2026-07-17 17:52:48 -03:00
diodiogod 4f22f145d2 Document voice design capabilities 2026-07-17 17:51:44 -03:00
diodiogod a8e08b5508 Link Voice Designer example workflow 2026-07-17 17:47:14 -03:00
diodiogod 9543a99bee Add unified Voice Designer example workflow 2026-07-17 17:46:22 -03:00
diodiogod a0dde24066 Shorten voice design reference text 2026-07-17 17:29:44 -03:00
diodiogod 58a754ac2c Remove legacy Qwen voice designer node 2026-07-17 17:26:53 -03:00
diodiogod 9fcedabf1c Update Qwen3 integration example workflow 2026-07-17 16:03:58 -03:00
diodiogod c1b9e088e3 Improve compact Character Voices layout 2026-07-17 13:37:14 -03:00
diodiogod edb7010184 Add Sound Effects workflow and guide 2026-07-17 11:13:48 -03:00
diodiogod 0ea124a1c0 Add neg alias for negative prompts 2026-07-17 01:34:40 -03:00
diodiogod ccbc721c2d Improve sound effect timeline handling and compile caching 2026-07-17 01:33:18 -03:00
diodiogod d8bf3dc865 Fix moss soundeffect v2 dtype handling and suppress harmless Inductor Ro 2026-07-16 22:46:14 -03:00
diodiogod 0c5201ebf3 Improve MOSS model-specific controls 2026-07-16 22:30:07 -03:00
diodiogod cce0131889 Clarify dependency repair instructions
- Replace technical missing-import output with user-facing engine warnings
- Add explicit close, terminal, install, and restart steps
- Test the complete startup repair message
2026-07-16 15:05:47 -03:00
diodiogod 3cf49110ae Direct dependency warnings to suite installer
- Print the active ComfyUI Python executable with install.py
- Remove misleading requirements.txt and package-specific repair advice
- Add regression coverage for installation guidance
2026-07-16 15:03:40 -03:00
diodiogod 0c3073e491 Standardize engine model dropdown choices 2026-07-16 15:02:12 -03:00
diodiogod 4a1bfd7b75 Merge branch 'main' into codex/unified-voice-design-and-saving 2026-07-16 14:03:31 -03:00
diodiogod a944650fde Version 5.4.16
Release Qwen runtime and legacy voice compatibility fixes

Technical details:
- Pin legacy Transformers 4 runtimes to supported kernels versions
- Correct Qwen generation block progress planning
- Migrate old Character Voices workflow state to canonical transcripts
2026-07-16 13:51:59 -03:00
diodiogod b55b6f03ad Repair legacy Qwen runtime and voice workflows
Implementation details:
- Pin Transformers 4 runtimes to a compatible kernels release for issue #324
- Align Qwen progress planning with actual pause and chunk generation calls
- Migrate legacy Character Voices workflows to canonical reference transcripts
- Add regression coverage for runtime dependencies, block planning, and legacy voices
2026-07-16 13:51:51 -03:00
diodiogod 97f4a0c365 Add unified MOSS audio generation workflows 2026-07-16 12:35:01 -03:00
diodiogod 4ccc0aa5ee Unify Hugging Face download output 2026-07-15 22:52:14 -03:00
diodiogod 198662124c Support MOSS-TTS v1.5 LoRA training 2026-07-15 22:27:19 -03:00
diodiogod a68bafb73f Ignore automatic Fish language prompt tags 2026-07-15 21:41:29 -03:00
diodiogod 08916ea598 Fix MOSS download output and model lifecycle 2026-07-15 21:39:58 -03:00
diodiogod 5329d8767d Fix custom widget rendering across canvas zoom 2026-07-15 21:08:04 -03:00
diodiogod f1f52c9d41 Version 5.4.15
Release Fish and VibeVoice compatibility repairs

Technical details:
- Restore the removed TorchAudio backend probe for Fish Audio S2
- Resolve local VibeVoice paths in the parent ComfyUI process
- Pass absolute model paths to isolated VibeVoice workers
- Add focused runtime regression tests
2026-07-15 15:13:34 -03:00
diodiogod 08b50473f5 Repair Fish and VibeVoice runtime paths for issue 322
- Restore the removed TorchAudio backend probe before Fish initializes
- Resolve local VibeVoice models in the parent ComfyUI process
- Pass absolute model paths into isolated VibeVoice workers
- Add regression coverage for both compatibility paths
2026-07-15 15:13:27 -03:00
diodiogod 798f3f4192 Version 5.4.14
Release engine runtime installation repairs

Technical details:
- Correct Fish Audio S2 namespace package restoration
- Add safe Dots TTS normalization fallback handling
- Remove conflicting VibeVoice WebRTC dependencies
- Invalidate isolated runtimes when dependency profiles change
2026-07-15 06:53:59 -03:00
diodiogod 4937ff9a5b Repair engine runtime installation for issue 322
- Restore Fish S2 implicit namespace packages correctly
- Use the Dots TTS normalizer fallback for incomplete tn installations
- Remove conflicting WebRTC dependencies from VibeVoice runtimes
- Rebuild isolated runtimes when profile definitions change
- Add focused installer regression coverage
2026-07-15 06:53:52 -03:00
diodiogod da88ced2a5 Improve voice design saving and OmniVoice language handling 2026-07-15 06:25:17 -03:00
diodiogod f344099e3b Add unified voice design and character saving 2026-07-15 00:22:32 -03:00
diodiogod 11b7a3c7fc Version 5.4.13
Release RVC HuBERT dimension handling

Technical details:
- Address issue #319 by selecting the HuBERT encoder after reading checkpoint dimensions
- Preserve explicit HuBERT selections
- Add strong experimental warnings for unsupported HuBERT Large training
2026-07-14 18:59:05 -03:00
diodiogod 12ce9be0c0 Fix RVC HuBERT auto-selection
Address issue #319 by deferring automatic HuBERT selection until the RVC checkpoint dimension is known.

- Match 768-dimensional voices with ContentVec automatically
- Match verified 1024-dimensional voices with HuBERT Large automatically
- Warn that HuBERT Large training lacks a compatible pretrained generator
- Recommend ContentVec 768 for reliable RVC training and inference
2026-07-14 18:58:58 -03:00
diodiogod 6c2eb70f8a Simplify training dashboard resizing 2026-07-14 18:17:13 -03:00
diodiogod 122c439c41 Fix training dashboard resize lag 2026-07-14 18:01:28 -03:00
diodiogod 764e28a1aa Add HuBERT Large RVC training support 2026-07-14 17:38:30 -03:00
diodiogod 8b28214d77 Make installer validation fast and accurate 2026-07-14 17:23:56 -03:00
diodiogod 561267d10b Version 5.4.12
Fix Fish Audio S2 and Dots installer compatibility

Technical details:
- Restore Fish source subpackages omitted by the upstream wheel metadata
- Allow the tested Dots source install on Python 3.13 without changing dependencies
2026-07-14 16:42:07 -03:00
diodiogod d3ab465983 Guard RVC training against HuBERT dimension mismatch 2026-07-14 16:41:03 -03:00
diodiogod 1ce69b2d1a Fix Fish and Dots installer compatibility 2026-07-14 16:39:33 -03:00
diodiogod 516ab1595f Version 5.4.11
Address issue #319 RVC Dataset Prep cache rebuild

Technical details:
- Rebuild RVC prepared datasets when the filelist or required feature artifacts are missing
- Preserve filelist-only refresh for complete stale caches
- Prevent first-run dataset preparation from skipping HuBERT feature extraction
2026-07-13 22:15:09 -03:00
diodiogod f81ebf1f9d Compact reference transcript documentation
Keep the Feature Comparison matrix concise by moving transcript mode explanations into a single footnote.
2026-07-13 22:05:04 -03:00
diodiogod 0c527ef541 Version 5.4.10
Document engine reference transcript requirements

Implementation details:
- Add typed reference_transcript metadata for every engine
- Extend the feature table generator for requirement-valued rows
- Regenerate the feature matrix with conditional mode notes
2026-07-13 21:35:59 -03:00
diodiogod 55065cc2bc Version 5.4.9
Refine Character Voices waveform and discovery

Implementation details:
- Add normalized cached waveform rendering to the trim timeline
- Add bounded playback progress with a smooth playhead
- Anchor the trim transcription warning outside layout flow
- Stop node schema queries from forcing voice discovery rescans
- Deduplicate background discovery refreshes across instances

Follow-up to #318
2026-07-13 16:44:04 -03:00
diodiogod d0430d846d Version 5.4.8
Address CosyVoice3 ROCm precision handling (issue #323)

Technical details:
- Run the bundled Qwen LLM with BF16 autocast on ROCm
- Keep the CosyVoice3 flow model and HiFT vocoder in FP32
- Disable unsupported TensorRT loading on ROCm
- Forward LLM worker failures instead of decoding empty tokens
2026-07-13 11:59:53 -03:00
diodiogod 7ff08f6070 Version 5.4.7
Enhance Character Voices reference editing

Implementation details:
- Add automatic transcription loading and workflow-local overrides
- Add modular trim-range controls and reference_audio_only output
- Preserve source provenance while preventing stale path fallbacks
- Unify tensor-first voice reference resolution across affected engines

Fixes #318
2026-07-13 11:54:19 -03:00
diodiogod 0bab378191 Version 5.4.6
Fix IndexTTS-2 emotion vector import interface

Implementation details:
- Reuse the shared emotion configuration modal for import and export
- Allow JSON to be pasted and edited before an explicit Apply action
- Validate every imported value before changing any emotion control
- Remove clipboard-first automatic import behavior
2026-07-13 08:42:22 -03:00
diodiogod af2cf2a4d1 Clarify successful librosa fallback logging 2026-07-13 08:18:51 -03:00
diodiogod 0dbc69d27f Version 5.4.5
Fix Fish Audio S2 installer runtime validation

- Validate fish_speech.inference_engine instead of distribution metadata alone
- Repair incompatible Fish installations automatically
- Pin the tested Fish S2 v2.0.0-beta source revision
- Verify the runtime import after installation
2026-07-12 12:02:00 -03:00
diodiogod 7c64fedede Version 5.4.4
Release Multiline TTS Tag Editor inline emotion controls

Implementation details:
- Add engine-aware one-click and hold-drag tag swapping
- Support combined IndexTTS audio references and vector or text emotion controls
- Improve quoted emotion routing and long inline tag wrapping
- Fix tag parsing, character switching, and newline insertion behavior
2026-07-11 16:25:30 -03:00
diodiogod 6da635cf74 Sync IndexTTS updates and document editor features 2026-07-11 16:24:08 -03:00
diodiogod 8bfd6512de Route quoted emotion tags to editor 2026-07-11 12:00:23 -03:00
diodiogod ffa3f1eded Add fast inline tag swapping 2026-07-11 11:09:01 -03:00
diodiogod 87def26983 Update version bump instructions 2026-07-11 08:09:50 -03:00
diodiogod 057a9ef638 Document and compose inline emotion controls 2026-07-11 08:09:08 -03:00
diodiogod df3a4cb2cf Blend IndexTTS audio and vector emotions 2026-07-11 01:39:43 -03:00
diodiogod 06003e0b3a Fix IndexTTS emotion and character routing 2026-07-11 01:24:38 -03:00
diodiogod 068c3f1f1e Add IndexTTS-2 inline emotion controls 2026-07-10 23:21:53 -03:00
diodiogod 46c051477d Version 5.4.3
Fix IndexTTS-2 emotion vector controls

Implementation details:
- Keep the fixed radar node size so button hit areas remain aligned
- Add clipped canvas rendering to prevent visual overflow
- Replace clipboard-dependent export fallback with a selectable JSON dialog
- Require explicit download and show import success feedback
2026-07-10 15:30:05 -03:00
diodiogod 52b22c0b8d Document FL-MCP Windows bridge fallback 2026-07-10 14:29:22 -03:00
diodiogod b31b31b89f Version 5.4.2
Address issue #312 with MPS-safe RVC tensor typing

Technical details:
- Keep RVC pitch indices integer on Apple MPS devices.
- Normalize phone features to the linear layer weight dtype.
- Normalize pitch values to torch.long at the encoder boundary.
- Preserve CUDA half-precision compatibility.
2026-07-10 14:25:59 -03:00
diodiogod 9757092aae Version 5.4.1
Address issue #311 with resilient MelBand resampling

Technical details:
- Route MelBand input conversion through the shared safe resampler.
- Fall back to torchaudio when librosa/Numba resampling is incompatible.
- Correct the PROJECT_INDEX path for bump instructions.
- Make version tooling use UTF-8 on Windows.
2026-07-10 14:06:12 -03:00
diodiogod 47a8c7c691 Version 5.4.0
Add Fish Audio S2 Pro engine and integration workflow

Implementation details:
- Integrate the official Fish S2 runtime with isolated loading and model selection
- Add native multi-speaker and custom character-switching generation paths
- Add inline instruction translation, language prompting, caching, compilation, and optional quantization
- Add unified Text and SRT workflow coverage with documentation and a workflow cover
2026-07-10 13:37:44 -03:00
diodiogod 1033eb78b9 Merge Fish Audio S2 integration 2026-07-10 13:37:15 -03:00
diodiogod 1f2703a3a3 Add Fish Audio S2 integration workflow
Add the Fish Audio S2 Pro engine workflow with native and custom speaker modes, inline instruction handling, language prompting, compilation, quantization, and SRT support.

Document the workflow in the README and include a square cover image for the example workflow.
2026-07-10 13:37:01 -03:00
diodiogod af77ff8e0b Match Fish workflow documentation layout 2026-07-10 11:56:44 -03:00
diodiogod 8b199980dc Improve Fish Audio S2 integration workflow 2026-07-10 09:39:37 -03:00
diodiogod 26fa2ddf92 Add Fish Audio S2 integration workflow 2026-07-10 09:16:33 -03:00
diodiogod 282616ebe1 Finalize Fish Audio S2 integration 2026-07-10 09:09:17 -03:00
diodiogod 5735616794 Prevent invalid hash seed from breaking isolated runtimes 2026-07-10 00:28:12 -03:00
diodiogod 6ce60f31a2 Document new engine validation rules 2026-07-10 00:21:17 -03:00
diodiogod 102f114ae6 Standardize resolved voice prompt logging 2026-07-10 00:20:00 -03:00
diodiogod 4494381176 Finalize Fish Audio S2 engine integration 2026-07-09 23:13:24 -03:00
diodiogod 84e82b7d31 Refine Fish Audio S2 compile and override logging 2026-07-08 23:21:16 -03:00
diodiogod e54b6d9aa5 Refine Fish speaker mapping and UI slots 2026-07-08 22:21:38 -03:00
diodiogod 984f0e9fa5 Fix VibeVoice native speaker mapping and logging 2026-07-07 00:56:57 -03:00
diodiogod 1218e01683 Fix OmniVoice tag handling to follow suite syntax 2026-07-07 00:13:19 -03:00
diodiogod 9081d3c22a Add Fish Audio S2 engine and fix VibeVoice speaker remap 2026-07-06 23:35:13 -03:00
diodiogod 09df168119 Version 5.3.0
Add Granite ASR 4.1 diarization and timestamp refinements

Implementation details:
- normalize Granite ASR plus diarization output to suite-native [Speaker] tags
- fix Granite ASR plus native timestamp truncation by auto-raising low generation token budgets
- move Unified ASR Transcribe diarization input to preserve old workflow compatibility
- update Granite ASR YAML and README documentation and regenerate engine tables
- improve cross-platform test runner path handling and Granite regression coverage
2026-06-23 18:33:56 -03:00
diodiogod b00b6098c9 Merge pull request #309 from tazztone/feature/combined-granite-echo-testing
Feature/combined granite echo testing
2026-06-23 18:32:36 -03:00
diodiogod a41ac5d89e Merge origin/main into feature/combined-granite-echo-testing 2026-06-23 18:27:49 -03:00
diodiogod 99ea0bc6f9 Refine Granite ASR diarization docs and timestamp handling 2026-06-23 18:21:19 -03:00
diodiogod e6c607b582 Fix Granite diarization label formatting and SRT punctuation regressions 2026-06-23 18:04:44 -03:00
diodiogod ed5e55a5fc Update README.md 2026-06-23 16:21:09 -03:00
diodiogod a0b58e42f7 Normalize large-language engine summaries in generated tables
- add full and compact language summary overrides to the engine table generator
- show Higgs Audio v3 with globe-style 100+ language summary in the README table
- show OmniVoice as 600+ languages in both compact and full generated tables
2026-06-23 12:10:00 -03:00
diodiogod 44d164c2b7 Fix OmniVoice language summary in generated tables
- add compact language summary override support to the table generator
- show OmniVoice as 600+ languages in the README summary table
- rename OmniVoice voice design feature text to instruction-based voice design
2026-06-23 12:06:07 -03:00
diodiogod 14fcba3bed Version 5.2.0
Add OmniVoice engine release polish

Implementation details:
- Finalize OmniVoice engine integration across text and SRT workflows
- Add native duration targeting support for precise subtitle timing
- Add Visual Tag Builder workflow and documentation polish
- Propagate interruption handling across unified generation nodes
- Fix Qwen3 voice designer metadata URL and test path mocking
2026-06-23 11:51:40 -03:00
diodiogod 36ed7d694a Add OmniVoice workflow example and docs polish 2026-06-23 11:46:37 -03:00
tazztone 383518854c refactor: relocate loose diagnostic utility scripts to tests/diagnostics/ 2026-06-23 16:07:45 +02:00
tazztone f2724520f4 docs: document Granite 4.1 2B models and regenerate engine tables 2026-06-23 16:02:13 +02:00
tazztone e2c691fa98 merge branch 'fix/platform-aware-testing' into feature/combined-granite-echo-testing 2026-06-23 15:38:11 +02:00
tazztone e7084f1bbc fix(granite-asr): resolve diarization multi-speaker turns and improve no-timestamp sequential timing 2026-06-23 11:29:39 +02:00
tazztone 8674797e7f feat: prefix diarized text with speaker IDs and update Granite adapter logic and ignore list 2026-06-23 10:54:28 +02:00
tazztone 0a8081f9c3 feat(granite): add speaker diarization for granite-speech-4.1-2b-plus and fix test runner environment detection 2026-06-23 10:30:04 +02:00
tazztone d4461b1bf8 Refactor test runner path configuration and enhance cross-platform reliability 2026-06-23 08:58:53 +02:00
diodiogod 24e6575fd4 Fix Qwen3 voice designer metadata URL 2026-06-22 22:46:01 -03:00
diodiogod fcff4ccf45 Propagate interruption across unified nodes 2026-06-22 22:27:31 -03:00
diodiogod f7abd5a891 Handle OmniVoice interruption without silence fallback 2026-06-22 22:23:04 -03:00
diodiogod 1291346dda Polish Visual Tag Builder output and README 2026-06-22 17:44:12 -03:00
diodiogod 552a67d209 Rename Visual Tag Builder and clean tooltips 2026-06-22 17:05:54 -03:00
tazztone 3436dbdab2 feat(granite): integrate Granite 4.1 2B plus native timestamps, bypass Qwen aligner, fix is_plus undefined NameError and false warnings 2026-06-20 11:05:39 +02:00
tazztone ec659ebc1f chore: add node_modules to .gitignore 2026-06-20 09:48:18 +02:00
tazztone a217786539 chore: ignore .agents directory 2026-06-20 09:46:24 +02:00
tazztone 6fb1bf5e49 chore: ignore .agents directory 2026-06-20 09:46:18 +02:00
tazztone 9ad18b6f3d chore: add node_modules/ to .gitignore 2026-06-20 09:28:05 +02:00
diodiogod 35e0b27913 Persist OmniVoice builder presets and layout 2026-06-19 14:38:44 -03:00
diodiogod 1e92a9f4b1 Polish OmniVoice instruction builder layout 2026-06-19 11:51:29 -03:00
tazztone 05a728313b fix(test): resolve test runner paths and conftest venv resolution dynamically across platforms 2026-06-19 16:21:04 +02:00
tazztone 29515feb35 feat(granite,echo_tts): support granite-speech 4.1 ASR and implement echo-tts interrupt callbacks 2026-06-19 16:10:23 +02:00
diodiogod 5c346693e3 Add OmniVoice instruction builder UI 2026-06-19 01:40:21 -03:00
diodiogod 3717fc0cbe Refine OmniVoice loading and progress logs 2026-06-18 22:57:12 -03:00
diodiogod af13209b47 Add OmniVoice engine with native SRT duration targeting 2026-06-18 22:23:32 -03:00
diodiogod d96fe777e8 Version 5.1.1
Fix isolated runtime bootstrap fallback for Unified ASR on portable ComfyUI

Technical details:
- Fall back from python -m venv to python -m virtualenv when embedded Windows builds cannot create a venv
- Preserve base site-packages inheritance so isolated runtimes still reuse ComfyUI heavy dependencies
- Surface primary and fallback bootstrap stderr/stdout for easier runtime setup debugging
- Address issue #304
2026-06-17 09:27:46 -03:00
diodiogod 41ee406634 Update README.md 2026-06-15 10:46:44 -03:00
diodiogod c3d2ddcc83 Merge branch 'feature/dots-tts' 2026-06-15 10:45:20 -03:00
diodiogod 3a817a2526 Version 5.1.0
Add Dots TTS engine and refine language handling

Implementation details:
- integrate official Dots TTS text and SRT workflows
- improve Dots runtime lifecycle, caching, and progress reporting
- fix Dots language override behavior and refresh generated metadata
2026-06-15 10:39:50 -03:00
diodiogod 78d3e867ef Refine Dots README metadata
Documentation details:
- remove redundant zero-shot voice cloning note from Dots summaries
- replace vague language tags wording with language control
- normalize Dots size wording to ~6GB in generated docs
2026-06-15 10:39:29 -03:00
diodiogod db3a888b77 Refine Dots language handling and metadata 2026-06-15 10:00:13 -03:00
diodiogod 1f5882ba44 Add Dots instruction TTS mode 2026-06-15 08:51:00 -03:00
diodiogod e471cdbaa6 Refine Dots TTS runtime and logging 2026-06-14 16:32:26 -03:00
diodiogod cb6dfe3a11 Add Dots TTS engine integration 2026-06-14 15:27:58 -03:00
diodiogod ae842ea39b Update README.md 2026-06-14 10:41:30 -03:00
diodiogod bc95f20ad3 Merge branch 'resume/higgs-audio-v3-t5' 2026-06-14 10:39:49 -03:00
diodiogod 3c835b10b7 Polish v5.0.0 changelog entry 2026-06-14 10:39:42 -03:00
diodiogod 4344bcaf1a Version 5.0.0
Prepare v5.0.0 release

Major changes:
- Resume Higgs Audio v3 on the main Transformers 5 environment
- Add isolated runtime support for fragile legacy engines
- Improve Multiline TTS Tag Editor inline validation, conversion, and persistence
- Refresh README, workflows, screenshots, and engine documentation
2026-06-14 10:35:44 -03:00
diodiogod 3373c83d84 Remove stale ChatterBox timeline section 2026-06-14 10:35:24 -03:00
diodiogod 03401c0b40 Update README.md 2026-06-14 10:29:36 -03:00
diodiogod 665518d218 Trim stale README lower sections 2026-06-14 10:24:02 -03:00
diodiogod ed0932d078 Trim README summary table isolation details 2026-06-14 10:17:30 -03:00
diodiogod 4ebffc9128 Polish Higgs v3 workflow and cleanup 2026-06-14 10:13:43 -03:00
diodiogod face03b42b Remove redundant MOSS manifest builder node 2026-06-12 15:09:51 -03:00
diodiogod 8760c548d1 Refine tag editor validation and README cleanup 2026-06-12 01:39:57 -03:00
diodiogod 18bfa834e0 Tighten README features and what's new 2026-06-11 17:57:06 -03:00
diodiogod afb35f5469 Summarize language counts in engine comparison 2026-06-11 08:00:28 -03:00
diodiogod b85067cc1c Treat RVC as language-agnostic in docs 2026-06-11 07:16:53 -03:00
diodiogod 4c0a0a934c Expand Higgs Audio v3 language coverage docs 2026-06-11 01:48:16 -03:00
diodiogod 3437d78772 Update README for Higgs v3 and T5 era 2026-06-11 01:28:02 -03:00
diodiogod 9cb78c3157 Finalize Higgs Audio v3 transformers 5 integration 2026-06-11 01:16:55 -03:00
diodiogod 2bf402d9a5 Fix Higgs v3 character mapping and ChatterBox SRT cache key 2026-06-10 00:40:38 -03:00
diodiogod cdb14c4956 Refine Higgs v3 inline tags and editor UI 2026-06-09 23:49:55 -03:00
diodiogod a42f4d3083 Resume Higgs Audio v3 on transformers 5 2026-06-09 19:04:57 -03:00
diodiogod 8367bc12bf Sync isolation runtime documentation 2026-06-09 18:11:35 -03:00
diodiogod 667d12c37b Clean up MOSS transformers warnings 2026-06-09 17:57:59 -03:00
diodiogod 45980761ee Fix RVC empty chunk handling and MelBand import 2026-06-09 00:58:48 -03:00
diodiogod 4807f9c654 Fix RVC training mute asset staging 2026-06-08 23:44:29 -03:00
diodiogod f75837834e Add Higgs Audio shared isolation runtime 2026-06-08 22:20:18 -03:00
diodiogod 287978b6a9 Document Higgs Audio transformers 5 investigation 2026-06-08 17:40:15 -03:00
diodiogod 70b04135ec Route Granite aligner through shared Qwen runtime 2026-06-07 21:55:14 -03:00
diodiogod 53df881108 Polish Qwen shared isolation runtime 2026-06-07 11:16:23 -03:00
diodiogod 14c32a96d1 Add Qwen3-TTS shared isolation runtime 2026-06-06 22:26:45 -03:00
diodiogod aff2baad27 Fix IndexTTS transformers 5 compatibility 2026-06-06 17:04:51 -03:00
diodiogod 010e532b38 Add isolated runtimes implementation plan 2026-06-06 15:12:07 -03:00
diodiogod 2b61bafd73 Add shared VibeVoice isolation runtime 2026-06-06 15:09:27 -03:00
diodiogod 63ff510768 Fix IndexTTS inline edit tag processing 2026-06-06 14:47:28 -03:00
diodiogod 8621d3b64d Fix Step Audio EditX transformers 5 warnings 2026-06-05 15:53:41 -03:00
diodiogod 5c1ab55abc Version 4.27.3
Change RVC custom chunking to opt-in

Technical details:
- Add an RVC engine toggle for extra outer chunking on top of native segmentation
- Default RVC to native segmentation instead of the unified outer chunk layer
- Report active RVC chunking mode in Voice Changer conversion info
2026-05-26 17:14:47 -03:00
diodiogod df938d0ac2 Version 4.27.2
Fix RVC chunked conversion timing drift

Technical details:
- Patch bundled RVC segment cropping to preserve frame-aligned duration
- Remove cumulative sample loss across long chunked voice conversions
- Keep the fix inside the bundled RVC pipeline instead of post-processing output
2026-05-26 11:29:49 -03:00
diodiogod 666e747c3e Fix Unified Voice Changer chunk timing drift
Technical details:
- Replace chunk crossfade joins with timing-preserving concatenation
- Apply consistent chunk recombination across all VC engines
- Clarify chunking tooltips for cleaner smart-boundary usage
2026-05-26 09:46:40 -03:00
diodiogod 09f3654a13 Update MOSS metadata for initial LoRA training release 2026-05-26 01:26:15 -03:00
diodiogod ec56310a89 Version 4.27.0
Add initial MOSS-TTS LoRA training support

Implementation details:
- add unified MOSS Delay 8B LoRA training flow and dataset preparation nodes
- add MOSS training docs and example workflow for clip-based dataset building
- add 4-bit MOSS training option for lower-VRAM LoRA training
- allow MOSS direct TTS inference without forced narrator fallback
2026-05-26 01:13:22 -03:00
diodiogod 86f29abfa5 Add audio analyzer file preview refresh 2026-05-25 13:28:42 -03:00
diodiogod 18dab7b4ca Persist audio analyzer visualization cache 2026-05-25 11:58:59 -03:00
diodiogod 41bfffeefe Fix audio analyzer DOM layout and caching 2026-05-25 11:04:02 -03:00
diodiogod 5fd7ecbea2 Add unified MOSS training pipeline and dataset tooling 2026-05-24 16:45:10 -03:00
diodiogod c056da620d Version 4.26.5
Fix Text to SRT Builder Chinese punctuation spacing

Technical details:
- Add CJK-aware SRT text joining for subtitle rendering
- Tokenize CJK plain-text timing estimates by character instead of whitespace
- Preserve Chinese punctuation and quote spacing without standalone punctuation leakage
- Add regression tests for Chinese punctuation, quote spacing, and English spacing
2026-05-23 18:24:07 -03:00
diodiogod f70654dc46 Version 4.26.4
Add MOSS LoRA loading support and compatibility validation

Implementation details:
- Add managed MOSS LoRA discovery and Hugging Face install flow under models/TTS/moss_tts/loras
- Wire PEFT adapter loading through the MOSS engine, adapter, cache, generation info, and unified model interface
- Validate adapter base-model compatibility before loading to prevent huge PEFT mismatch errors
- Raise real ComfyUI popup errors for MOSS LoRA/base model mismatches instead of generic engine creation failures
- Patch bundled MOSS Delay and TTSD classes for PEFT generation compatibility
- Upgrade installer requirement handling so outdated packages like peft are upgraded automatically
- Document the MOSS LoRA folder contract and update MOSS metadata
2026-05-23 17:42:01 -03:00
diodiogod e3e3685226 Fix RVC librosa fallback decorators shim
Technical details:
- Make the bundled librosa.util fallback package-like
- Register a librosa.util.decorators fallback module
- Provide a no-op deprecated decorator for later librosa.filters imports

Address issue #293
2026-05-21 08:11:58 -03:00
diodiogod 969939317c Version 4.26.3
Fix MOSS model selector stability across local/friendly variants

Technical details:
- Keep local:* model labels visible when local folders exist
- Accept both friendly and local:* values for backward compatibility
- Ensure native mode resolves to TTSD model variant for execution
2026-05-19 18:05:54 -03:00
diodiogod f8c5c2b965 Version 4.26.2
Fix Qwen3-TTS auto attention fallback for broken flash-attn installs

Technical details:
- Validate flash_attn package metadata before selecting flash_attention_2 in auto mode
- Fall back to sdpa when flash-attn is missing or partially installed on Windows
- Add runtime retry to sdpa when auto-selected flash_attention_2 fails during model load

Address issue #294
2026-05-18 21:46:53 -03:00
diodiogod 6d7da1a4af Version 4.26.1
Fix RVC RMVPE mel fallback matrix orientation

Technical details:
- Transpose the torchaudio mel filterbank fallback to match librosa.filters.mel output shape
- Return [n_mels, n_stft] instead of [n_stft, n_mels] for RMVPE pitch extraction
- Prevent matrix multiplication shape errors in RVC RMVPE inference after librosa fallback activation

Address issue #293
2026-05-18 17:15:58 -03:00
diodiogod 3f97a8f709 Add dedicated MOSS section to README 2026-05-18 16:53:17 -03:00
diodiogod dee8cfd0d1 Update README.md 2026-05-18 16:49:15 -03:00
diodiogod 11951d35dd Version 4.26.0
Add MOSS-TTS engine family and finalize native TTSD behavior

Release details:
- Integrate Local/Delay/TTSD MOSS models into unified nodes
- Finalize strict native TTSD incompatibility handling
- Improve native SRT behavior and user-facing error propagation
2026-05-18 16:46:50 -03:00
diodiogod 2b69130a42 Add MOSS-TTS engine family with native TTSD support (#295)
* docs: add new engine integration guide flow

* docs: link new engine guide navigation

* docs: clarify SRT is required for TTS engines

* docs: document SRT reuse of TTS processor

* docs: move new engine guide link below timeline

* Add MOSS-TTS engine integration

* docs: add reference input mode to engine research

* Add MOSS-TTS engine checkpoint

* Refine MOSS prompt fields and parameter switching

* Refine MOSS prompt field warnings and caching

* Improve MOSS inline edit handling and VRAM handoff

* Improve MOSS native dialogue mapping and UI

* docs: add long-form model analysis and VibeVoice reference

* Improve MOSS TTSD chunking, interrupts, and SRT mapping

* Refine MOSS mapping, device checks, and metadata

* Make MOSS native incompatibilities hard-fail

* Document MOSS native hard-fail compatibility rules
2026-05-18 16:42:53 -03:00
diodiogod c8e6317c8d Version 4.25.21
Fix RVC RMVPE mel filter fallback kwargs

Technical details:
- Allow safe_mel_filters to accept librosa.filters.mel keyword arguments
- Pass htk, norm, dtype, and other mel options through to working librosa installs
- Map supported options to the torchaudio fallback when librosa cannot build mel filters
- Prevent RMVPE pitch extraction from failing on htk=True after librosa.util fallback activation

Address issue #293
2026-05-09 13:45:43 -03:00
diodiogod 4e52c14a04 Version 4.25.20
Fix RVC voice conversion librosa fallback

Technical details:
- Replace the fragile RVC inference-time librosa.util import patch with a guarded compatibility shim
- Preserve the real librosa.util module when it imports successfully
- Install bundled fallback utilities only when librosa.util crashes or lacks required helpers
- Cover normalize, pad_center, tiny, fill_off_diagonal, is_positive_int, expand_to, fix_length, and stack for RVC reference code

Address issue #293
2026-05-07 22:43:31 -03:00
diodiogod 4a21a59fb9 Version 4.25.19
Fix RVC RMVPE librosa import fallback for Python 3.13

Technical details:
- Catch runtime import-time errors from librosa.util (not only ImportError)
- Fall back to bundled pad_center/tiny/normalize compatibility helpers
- Prevent RVC dataset prep from crashing when numba/librosa typing errors occur

Address issue #293
2026-05-06 22:09:03 -03:00
diodiogod 309f00a707 Version 4.25.18
Fix Echo-TTS character voice file-path fallback and compressed audio loading

Technical details:
- Preserve Character Voices and dropdown audio_path fallback in Echo-TTS voice mapping
- Allow Echo-TTS narrator and character fallback to use file paths when decoded tensors are unavailable
- Expand safe audio loading fallback order to support OGG and other compressed formats when torchaudio fails
2026-04-17 00:32:32 -03:00
diodiogod 9054b197b0 Version 4.25.17
Fix IndexTTS-2 compile flag handling and failure reporting

Technical details:
- Normalize legacy boolean workflow inputs so omitted IndexTTS-2 compile flags do not become truthy strings
- Fail fast when IndexTTS-2 torch.compile is enabled on CUDA without Triton installed
- Stop returning silent audio for IndexTTS-2 generation failures and surface real execution errors
- Remove stale IndexTTS-2 Python 3.12 VRAM warnings and noisy QwenEmotion debug logging
- Update IndexTTS-2 QwenEmotion loading to prefer dtype over deprecated torch_dtype
2026-04-14 17:29:42 -03:00
diodiogod 10532907f2 Version 4.25.16
Address Higgs Audio v2 download compatibility regression

Technical details:
- Pin official managed Higgs Audio v2 downloads to pre-2026-04-04 Hugging Face revisions
- Auto-restore pinned compatible Higgs metadata when newer transformers 5.3-native files were already downloaded
- Add revision-aware Hugging Face downloads and reduce repeated Higgs managed-path logging
2026-04-14 17:03:23 -03:00
diodiogod 9095d748b6 Version 4.25.15
Fix Higgs Audio tokenizer compatibility with latest upstream tokenizer metadata

Technical details:
- Add local PreTrainedTokenizerFast fallback for upstream TokenizersBackend metadata
- Preserve existing AutoTokenizer and legacy tokenizer paths
- Keep Higgs tokenizer loading compatible with older transformers support window
2026-04-13 10:42:32 -03:00
diodiogod ef82e69f28 Version 4.25.14
Fix RVC training FAISS stub-package handling

Technical details:
- Validate working FAISS APIs instead of treating plain faiss import success as sufficient
- Remove broken or conflicting faiss packages and namespace-only stubs before reinstalling faiss-cpu
- Detect missing FAISS APIs in RVC index building and degrade cleanly instead of crashing training
2026-04-12 16:01:20 -03:00
diodiogod 1020d64cbf Version 4.25.13
Fix RVC training checkpoint resume loading on PyTorch 2.6+

Technical details:
- Set weights_only=False in bundled RVC training checkpoint loaders used by resume
- Align resume checkpoint loading with the already-fixed initial checkpoint loader
- Prevent RVC resume from silently falling back to fresh training when PyTorch blocks checkpoint unpickling
2026-04-12 14:26:09 -03:00
diodiogod 2add12df54 Version 4.25.12
Fix Higgs Audio compatibility with latest upstream model release

Technical details:
- Restore CharacterParser language display helper after parser modularization
- Add compatibility shim for flat Higgs Audio v2 Hugging Face config schema
- Add loader diagnostics for legacy and v2 Higgs config detection
2026-04-12 11:19:30 -03:00
diodiogod 98f05aab66 Version 4.25.11
Fix RVC training resume job matching

Technical details:
- Match resume jobs by stable dataset, sample rate, and output-name identity instead of the full config hash
- Select the newest compatible job directory that actually contains G_*.pth and D_*.pth checkpoints
- Clarify unified model training resume tooltip and failure messaging
2026-04-12 10:55:01 -03:00
diodiogod 9b83253b87 Version 4.25.10
Improve RVC training Linux FAISS dependency recovery

Technical details:
- Validate faiss imports after Linux CUDA GPU package installation
- Fall back to faiss-cpu when faiss-gpu installs but cannot import
- Log broken faiss import errors more clearly during installation
2026-04-12 10:49:28 -03:00
diodiogod ec6f127136 Version 4.25.9
Improve RVC model training dataset prep reliability

Technical details:
- auto-resolve RMVPE for rmvpe-based RVC training pitch extraction
- stop RVC dataset prep when F0 extraction produces no usable voiced entries
- surface the underlying FAISS import error in RVC index build diagnostics
2026-04-11 17:05:24 -03:00
diodiogod 19d7f7bd83 Harden RVC training dataset preparation
Technical details:
- auto-resolve RMVPE for RVC training pitch extraction when rmvpe-based methods are selected
- fail dataset prep loudly when voiced F0 extraction produces no usable entries
- include the underlying FAISS import error in RVC index build diagnostics
2026-04-11 17:05:18 -03:00
diodiogod c632da3588 Version 4.25.8
Improve RVC model training dependency compatibility

Technical details:
- remove ffmpeg-python from the bundled RVC training runtime
- keep RVC model training explicit when FAISS index build dependencies are unavailable
- record index build status and warnings in RVC training artifacts
- install faiss-cpu with --no-deps to avoid NumPy constraint churn
2026-04-11 15:54:02 -03:00
diodiogod 4b3c050ddd Improve RVC training dependency handling
Technical details:
- remove ffmpeg-python from the bundled RVC training runtime
- keep RVC model training explicit when FAISS index build deps are missing
- record index build status and warnings in RVC training artifacts
- install faiss-cpu with --no-deps to avoid NumPy constraint churn
2026-04-11 15:53:42 -03:00
diodiogod 1438f150d0 Version 4.25.7
Fix RVC training bundled runtime packaging

Technical details:
- Track the missing bundled_runtime/lib training source tree in git
- Keep bytecode artifacts ignored while shipping the real bundled runtime files
- Force RVC training worker processes to use spawn for Linux and ROCm compatibility
2026-04-11 13:02:14 -03:00
diodiogod a76003e2a4 Fix bundled RVC training runtime packaging
Include the missing bundled RVC training runtime lib package in the repository so RVC training no longer fails with 'Missing bundled RVC runtime lib package'. Also force training workers to use an explicit spawn multiprocessing context to avoid ROCm/Linux forked CUDA reinitialization failures.

Addresses issues #284 and #285.
2026-04-11 13:01:02 -03:00
diodiogod 1ea6f182fb Version 4.25.6
Fix ChatterBox optional perth import crashing model load on Python 3.13

Technical details:
- Move perth import behind lazy watermark initialization
- Prevent module load from touching fragile librosa resample path by default
- Keep watermarking available only when explicitly enabled
2026-04-10 21:43:54 -03:00
diodiogod 5185a1678b Version 4.25.5
Improve Multiline TTS Tag Editor editing workflow

Technical details:
- Add find and replace UI with navigation, replace all, and match options
- Persist Multiline TTS Tag Editor search preferences without storing query text
- Add collapsible left panel behavior while preserving splitter resizing
- Refine left panel handle styling and drag sync during resize
2026-04-08 10:14:35 -03:00
diodiogod 47f09ba89b Improve Multiline TTS Tag Editor find and replace UX
- add inline find/replace controls with keyboard shortcuts and match navigation
- add replace and replace all flows with persistent search options and highlighting
- add collapsible left panel control while preserving existing resize behavior
- refine panel handle interaction and styling to avoid splitter conflicts during drag
2026-04-08 10:14:25 -03:00
diodiogod 04da79f0a8 Version 4.25.4
Fix TTS Audio Suite settings API deprecation warnings

Technical details:
- Replace deprecated settings defaultValue fields with default
- Stop passing fallback values to getSettingValue on newer ComfyUI builds
- Keep backend settings sync behavior unchanged while removing console warning spam
2026-04-08 00:31:48 -03:00
diodiogod 460d091442 Update Unified ASR Transcribe SRT Builder workflow 2026-04-08 00:24:30 -03:00
diodiogod e736822a01 Version 4.25.3
Redesign SRT Advanced Options interface

Technical details:
- Replace the old flat widget stack with a compact custom panel
- Add preset seeding, modified-state tracking, and adaptive field locking
- Clarify SRT mode and heuristic profile behavior in tooltips and UI text
2026-04-08 00:20:26 -03:00
diodiogod 7bc5e4bcaa Redesign SRT advanced options UI 2026-04-08 00:19:52 -03:00
diodiogod 3f0f0d42c5 Checkpoint SRT advanced options UI rewrite 2026-04-07 17:52:07 -03:00
diodiogod 7f4f6862d2 Fix ONNX Runtime installer preserving GPU setups
Technical details:
- Detect existing onnxruntime-gpu or CPU onnxruntime before install
- Skip Python 3.13 CPU onnxruntime force-reinstall when GPU runtime already exists
- Keep OpenSeeFace compatibility without clobbering shared ComfyUI environments
- Address issue #283 installer conflict report
2026-04-06 10:57:32 -03:00
diodiogod 7ef2f8718c Version 4.25.1
Fix RVC training runtime shim packaging

Implementation details:
- Move the RVC training spawn shim under the engine training module instead of a repo-root package
- Make engines.rvc package import lazy to avoid Windows multiprocessing spawn side effects
- Track the runtime shim bridge files explicitly despite the broad lib ignore rule
2026-04-05 13:41:50 -03:00
diodiogod 775cbd037c Track RVC training runtime shim support files 2026-04-05 13:41:44 -03:00
diodiogod 75053cc2d8 Fix RVC training runtime shim import path 2026-04-05 13:41:24 -03:00
diodiogod 5d93512f9c Version 4.25.0
Add RVC training workflow and dashboard

Implementation details:
- Add unified model training entry point with RVC dataset prep and training config nodes
- Integrate resumable RVC training, continue-from-model support, and safe interrupt checkpoint saving
- Add live training dashboard, training progress registry, and RVC workflow/docs updates
- Improve RVC loader, auto-index handling, pitch controls, and downloader/model layout integration
2026-04-05 13:32:19 -03:00
diodiogod 861e1ac254 Move RVC training workflow out of unified README section 2026-04-05 13:32:12 -03:00
diodiogod d408715120 Update README.md 2026-04-05 13:30:31 -03:00
diodiogod f6ca9525a2 Refine generated training docs tables 2026-04-05 13:12:14 -03:00
diodiogod f1f05ae290 Document RVC training workflow and capabilities 2026-04-05 13:08:45 -03:00
diodiogod 4521ffbc79 Add unified RVC training workflow and improve RVC UX 2026-04-05 11:09:06 -03:00
diodiogod 8ee1845ecb Version 4.24.20
Address F5-TTS model loading issue #206

Technical details:
- Fix vendored F5-TTS bundled vocab fallback path resolution on custom-node installs
- Pass an explicit bundled vocab path when organized model download falls back to HF cache loading
- Preserve the normal organized-download flow while preventing broken infer/infer/examples path construction
2026-04-03 22:45:47 -03:00
diodiogod ddc87f4ad8 Version 4.24.19
Address Step Audio EditX Mac compatibility issue #245

Technical details:
- Route bundled Step Audio EditX audio loading through the shared librosa fallback layer
- Preserve original librosa.load behavior by keeping 22050 Hz mono loading defaults
- Add bundled trim fallback when fragile numba/librosa environments crash with get_call_template errors
2026-04-03 22:32:53 -03:00
diodiogod 1df7df8b77 Version 4.24.18
Improve VibeVoice and KugelAudio model support

Technical details:
- Add kugel-2 support as a VibeVoice KugelAudio variant
- Normalize KugelAudio v2 config fields for transformers 4.x compatibility
- Fix UnifiedDownloader alt_repo handling for VibeVoice Hindi tokenizer downloads
- Improve VibeVoice pause-tag tracing in custom character switching
- Update VibeVoice model registry and generated docs
2026-04-03 22:12:25 -03:00
diodiogod 8c6e7f358d Version 4.24.17
Improve Multiline TTS Tag Editor gutter spacing and Library guide navigation

Technical details:
- switch the gutter width to digit-based sizing so the line column matches the actual count range
- tighten left padding and gutter spacing while keeping the integrated aligned line model
- add internal Library sub-tabs so guide sections render as focused panels instead of one long block
2026-04-01 22:08:04 -03:00
diodiogod b7f27093b7 Refine Multiline TTS Tag Editor gutter spacing and Library navigation
Implementation details:
- switch gutter width sizing to a digit-based column instead of font-size estimates
- tighten the left gutter padding and spacing without changing the aligned row model
- split the Library view into internal guide tabs for characters, parameters, inline tags, and SRT editing
2026-04-01 22:07:58 -03:00
diodiogod 1498ac0b53 Version 4.24.16
Fix Multiline TTS Tag Editor gutter alignment drift

Technical details:
- replace the separate gutter measurement model with integrated logical-line numbering
- keep gutter width tied to the real line count through editor CSS variables
- remove scroll-sync and wrapped-height guesswork from the editor gutter path
2026-04-01 21:56:19 -03:00
diodiogod 06428ba6d0 Rework Multiline TTS Tag Editor gutter alignment
Implementation details:
- remove the separate gutter DOM and its height/scroll sync logic
- render line numbers on the logical line blocks that already own wrap layout
- size the gutter column from line count through editor CSS variables
2026-04-01 21:56:14 -03:00
diodiogod 1d90607bd6 Fix Multiline TTS Tag Editor plain-text clipboard handling
Implementation details:
- use plain-text selection ranges for copy, cut, and paste in the editor
- avoid browser DOM serialization that breaks SRT timing lines on the clipboard
- reuse the same selection model for selection-based editor actions
2026-04-01 18:14:51 -03:00
diodiogod c9431fdec6 Version 4.24.15
Update Multiline TTS Tag Editor tag highlighting reliability

Release details:
- fix residual internal marker cleanup in highlighted editor output
- add a fallback square-bracket tag highlight pass for missed raw tags
- improve SRT subtitle editing reliability when repeated pause tags are present
2026-04-01 00:52:32 -03:00
diodiogod 1a67f20117 Fix Multiline TTS Tag Editor residual tag highlighting
Implementation details:
- strip leftover internal marker tokens before rendering highlighted editor HTML
- add a fallback square-bracket tag highlight pass for missed raw tags
- preserve the existing SRT timing and cue-number render flow
2026-04-01 00:52:21 -03:00
diodiogod 86992ff44e Fix Multiline TTS Tag Editor guide links
Implementation details:
- convert related documentation paths to real Markdown links
- keep the guide content unchanged aside from link formatting
2026-03-31 23:47:17 -03:00
diodiogod ec5a5212f5 Version 4.24.14
Update Multiline TTS Tag Editor SRT editing workflow

Release details:
- add cue-number merge actions for combining adjacent subtitle blocks
- add caret-based cue splitting with proportional subtitle timing estimation
- improve in-editor documentation with a new SRT editing library section and dedicated user guide
- update README links so Multiline TTS Tag Editor documentation is easier to find
2026-03-31 23:42:51 -03:00
diodiogod 362d4959e0 Add Multiline TTS Tag Editor SRT cue merge and split tools
Implementation details:
- add cue-number merge actions for combining neighboring SRT subtitles
- add caret-based cue splitting with proportional timing estimation
- document SRT editing workflows in the new guide, library tab, and README
2026-03-31 23:42:37 -03:00
diodiogod 19b88e615a Version 4.24.13
Update Multiline TTS Tag Editor timing and layout behavior

Release details:
- add draggable SRT timing handles for start, end, and full-range adjustments
- preserve subtitle gaps when shift-dragging linked timing boundaries
- fix wrapped-line gutter sync by using actual rendered editor line heights
- keep the editor scrollbar overlayed so text no longer shifts when focus changes
2026-03-31 23:07:20 -03:00
diodiogod 5d8d1a78eb Fix Multiline TTS Tag Editor gutter sync for wrapped lines
Implementation details:\n- render logical editor lines as block elements for direct height measurement\n- align gutter rows to actual rendered line heights instead of hidden clone estimates\n- preserve caret mapping and custom scrollbar behavior with the new DOM structure
2026-03-31 23:07:11 -03:00
diodiogod c336e01faa Add draggable SRT timing handles to tag editor 2026-03-30 22:49:10 -03:00
diodiogod 44b7e38853 Version 4.24.12
Fix VibeVoice KugelAudio parameter handling and SageAttention support

Technical details:
- Apply KugelAudio 4-bit quantization instead of silently ignoring the flag
- Honor KugelAudio inference_steps, top_p, and attention_mode settings
- Add SageAttention patch support for KugelAudio and generalize attention restoration
- Align KugelAudio logging with normal VibeVoice attention messages
2026-03-30 22:08:02 -03:00
diodiogod d017f66969 Version 4.24.11
Fix Character Voices restore and CosyVoice model cache invalidation

Technical details:
- Fix Character Voices startup restore so saved voices resolve correctly after reload
- Add path-only narrator fallback when local audio decoding fails
- Restore Character Voices preview hydration after workflow load
- Fix CosyVoice cached engine reuse when switching between 0.5B and 0.5B RL variants
2026-03-29 11:33:19 -03:00
diodiogod cd8f2683b0 Fix CosyVoice cache invalidation on model switch 2026-03-29 11:07:44 -03:00
diodiogod 668ed764eb Fix Character Voices restore and fallback loading 2026-03-29 10:40:23 -03:00
diodiogod 186b9ec7d4 Version 4.24.10
Fix VibeVoice local custom model support

Technical details:
- fix local community model discovery for both TTS roots and direct vibevoice paths
- support nested local snapshot layouts such as VibeVoice7b-low-vram/4bit
- align dropdown discovery with runtime custom model resolution
- make VibeVoice package loading lazy to avoid startup import poisoning
- apply transformers compatibility patches on the unified load path
- fix quantized VibeVoice buffer alignment on the writable inner model
2026-03-28 22:21:05 -03:00
diodiogod 832503712a Merge pull request #274 from Golorom/my-fixes
Fixes for VibeVoice local models + tokenizer + quantization
2026-03-28 22:20:34 -03:00
diodiogod 984eaf1600 Fix VibeVoice local custom model loading
- discover local community VibeVoice folders from both TTS roots and direct vibevoice paths
- support nested snapshot layouts such as VibeVoice7b-low-vram/4bit
- align node dropdown discovery with runtime model path resolution
- avoid startup-time VibeVoice import poisoning by switching to lazy package loading
- apply transformers compatibility patches on the unified VibeVoice load path
- move quantized inner-model buffers on the real writable object instead of the read-only wrapper property
2026-03-28 22:17:11 -03:00
Golorom 2646148ea9 Fixes for VibeVoice local models + tokenizer + quantization (FabioSarracino and DevParker support) 2026-03-28 22:17:11 -03:00
diodiogod f09ae688c5 Version 4.24.9
Expand SRT Advanced Options heuristic language profiles

Implementation details:
- Add first-pass heuristic defaults for Spanish, French, Italian, German, Dutch, Russian, Romanian, Indonesian, Malay, Turkish, Polish, Czech, Swedish, Danish, Finnish, and Greek
- Extend Auto language resolution for the new profile set
- Update the SRT Advanced Options widget so the dropdown seeds matching defaults in the UI
2026-03-28 13:47:37 -03:00
diodiogod 2540583f1d Version 4.24.8
Improve Text to SRT Builder TTS-ready segmentation and ASR timing reuse

Implementation details:
- Add language-aware subtitle heuristic profiles with Portuguese (Brazil) defaults
- Add TTS-ready subtitle mode and preset for single-line semantic cue building
- Improve dangling-tail and incomplete-thought merging in SRT builder
- Serialize ASR timing output as reusable JSON and rename builder input to asr_timing_data
- Remove the duplicate SRT Advanced Options node alias and update bundled workflows
2026-03-28 01:59:36 -03:00
diodiogod 273229976b Version 4.24.7
Fix CosyVoice3 narrator reference and interrupt handling

Implementation details:
- Restore connected opt_narrator resolution in unified text and SRT nodes
- Propagate connected reference text and character names through CosyVoice3 processing
- Add mid-generation interrupt polling for CosyVoice3
- Patch bundled Qwen3-ASR generate loop for reliable ComfyUI interrupts
- Mark vendored Qwen3-ASR interrupt logic as a TTS Audio Suite patch
2026-03-27 22:05:56 -03:00
diodiogod 28fcba7958 Release Multiline TTS Tag Editor redesign
Technical details:
- package the completed node redesign into a patch release
- include functional History, Presets, and Library tab workflows
- ship inline restore guidance, sidebar tooltips, and visual polish refinements
2026-03-27 14:27:06 -03:00
diodiogod 42fdfd6e07 Complete Multiline TTS Tag Editor redesign
Technical details:
- add functional History, Presets, and Library tab views
- expand inline tag and restore guidance with in-editor documentation
- refine sidebar controls, slider fills, syntax highlights, and restore tooltips
- align the node palette and interaction polish across the redesigned editor
2026-03-27 14:23:20 -03:00
diodiogod 4af18f7b9f Redesign multiline tag editor UI shell 2026-03-27 12:01:27 -03:00
diodiogod f285fa8bd2 Fix ComfyUI model wrapper compatibility with ComfyUI 0.18 memory management
Technical details:
- Add model_mmap_residency fallback to the shared ComfyUI wrapper
- Add pinned memory, dtype, patcher, and device compatibility methods
- Return full model memory for non-mmap-backed engines instead of a zero stub
- Add regression coverage for wrapper compatibility behavior

Fixes #276
Fixes #278
2026-03-27 09:25:38 -03:00
768 changed files with 157366 additions and 11444 deletions
+14
View File
@@ -0,0 +1,14 @@
# Registry archives contain runtime files and user documentation.
# These exclusions do not remove anything from the Git repository.
/.github/
/tests/
/scripts/
/PROJECT_INDEX.md
/AGENTS.md
/CLAUDE.md
/docs/BUMP_SCRIPT_INSTRUCTIONS.md
# Keep the metadata YAML files: the auxiliary registry reads its YAML at runtime.
/docs/Dev reports/*
!/docs/Dev reports/tts_audio_suite_engines.yaml
!/docs/Dev reports/tts_audio_suite_aux_models.yaml
+3
View File
@@ -14,6 +14,9 @@ jobs:
steps:
- name: Check out code
uses: actions/checkout@v4
- name: Preserve Unicode filenames in Registry archives
# comfy-cli reads git ls-files as plain paths; Git must not escape emoji.
run: git config core.quotepath false
- name: Publish Custom Node
uses: Comfy-Org/publish-node-action@main
with:
+15
View File
@@ -21,6 +21,10 @@ eggs/
.eggs/
lib/
!engines/rvc/impl/lib/
!engines/rvc/training/bundled_runtime/lib/
!engines/rvc/training/bundled_runtime/lib/**
engines/rvc/training/bundled_runtime/lib/**/__pycache__/
engines/rvc/training/bundled_runtime/lib/**/*.py[cod]
lib64/
parts/
sdist/
@@ -135,6 +139,9 @@ celerybeat.pid
# Environments
.env
.env.local
tests/.env
tests/.env.local
.venv
env/
venv/
@@ -142,6 +149,7 @@ ENV/
env.bak/
venv.bak/
comfyui_env/
/runtimes/
# Spyder project settings
.spyderproject
@@ -238,3 +246,10 @@ tests/output/
# TTS Audio Suite settings cache (user preferences)
.inline_tag_settings_cache.json
# AI Agent skills
.agents/
skills-lock.json
# others
node_modules/
+1173
View File
File diff suppressed because it is too large Load Diff
+23 -16
View File
@@ -27,21 +27,28 @@ Third-Party Model Licenses
The project code is MIT. Model weights carry their own licenses:
──────────────── ───────────────────────────────────────── ──────────────
Engine License Commercial Use
──────────────── ───────────────────────────────────────── ──────────────
F5-TTS CC-BY-NC-4.0 No
ChatterBox MIT Yes
ChatterBox 23L MIT Yes
VibeVoice MIT (research-only per model card) No
Higgs Audio 2 Boson Higgs Audio 2 Community License Conditional
IndexTTS-2 bilibili Model Use License Conditional
CosyVoice3 Apache-2.0 Yes
Qwen3-TTS Apache-2.0 Yes
Granite ASR Apache-2.0 Yes
Step Audio EditX Apache-2.0 (verify before commercial use) Conditional
Echo-TTS CC-BY-NC-SA-4.0 No
RVC MIT (framework); community models vary Varies
──────────────── ───────────────────────────────────────── ──────────────
─────────────────── ──────────────────────────────────────────────────────── ──────────────
Engine License Commercial Use
─────────────────── ──────────────────────────────────────────────────────── ──────────────
F5-TTS CC-BY-NC-4.0 No
ChatterBox MIT Yes
ChatterBox 23L MIT Yes
VibeVoice MIT (research-only per model card) No
Higgs Audio 2 Boson Higgs Audio 2 Community License Conditional
Higgs Audio v3 Boson Higgs Audio v3 Research and Non-Commercial License No
IndexTTS 2 / 2.5 bilibili Model Use License Conditional
CosyVoice3 Apache-2.0 Yes
Qwen3-TTS Apache-2.0 Yes
Granite ASR Apache-2.0 Yes
Step Audio EditX Apache-2.0 (verify before commercial use) Conditional
Echo-TTS CC-BY-NC-SA-4.0 No
Fish Audio S2 Pro Fish Audio Research License No
Dots TTS Apache-2.0 Yes
DramaBox LTX-2 Community License Conditional
OmniVoice Apache-2.0 Yes
MOSS-TTS Apache-2.0 Yes
MOSS-SoundEffect v2 Apache-2.0 Yes
RVC MIT (framework); community models vary Varies
─────────────────── ──────────────────────────────────────────────────────── ──────────────
Users are responsible for complying with respective model licenses.
+116 -53
View File
@@ -4,57 +4,72 @@
## Architecture Overview
**Unified modular architecture** - all engines share the same layered pattern:
1. **Engine Node** (`nodes/engines/<engine>_engine_node.py`) - ComfyUI configuration UI
2. **Unified Nodes** (`nodes/unified/`) - thin delegation layer, routes to engine processors
3. **Processor** (`nodes/<engine>/<engine>_processor.py`) - orchestration, chunking, character/pause/tag handling
4. **SRT Processor** (`nodes/<engine>/<engine>_srt_processor.py`) - SRT-specific orchestration
5. **Adapter** (`engines/adapters/<engine>_adapter.py`) - bridges processor to engine implementation
6. **Engine Implementation** (`engines/<engine>/`) - actual model inference
**Key architectural rules:**
- Chunking happens in the **processor**, not the adapter (`generate_single()` on adapter = raw single call)
**Unified modular architecture** - all engines share the same layered pattern:
1. **Engine Node** (`nodes/engines/<engine>_engine_node.py`) - ComfyUI configuration UI
2. **Unified Nodes** (`nodes/unified/`) - thin delegation layer, routes to engine processors
3. **Processor** (`nodes/<engine>/<engine>_processor.py`) - orchestration, chunking, character/pause/tag handling
4. **SRT Processor** (`nodes/<engine>/<engine>_srt_processor.py`) - SRT-specific orchestration
5. **Adapter** (`engines/adapters/<engine>_adapter.py`) - bridges processor to engine implementation
6. **Engine Implementation** (`engines/<engine>/`) - actual model inference
7. **Optional Isolated Runtime** (`utils/runtimes/`) - worker subprocess layer for fragile engine stacks
**Key architectural rules:**
- Chunking happens in the **processor**, not the adapter (`generate_single()` on adapter = raw single call)
- Runtime routing happens through `ModelLoadConfig.runtime_mode` + `runtime_profile`, not ad-hoc subprocess calls
- Shared runtime workers are currently used for fragile engine families such as Qwen3-TTS / ASR, Granite forced alignment, and Higgs Audio 2. VibeVoice/Kugel and other engines that support the modern stack run natively in the main Transformers 5 environment.
- YAML (`docs/Dev reports/tts_audio_suite_engines.yaml`) is source of truth for engine doc tables → run `python3 scripts/generate_engine_tables.py --readme` to regenerate
- Auxiliary YAML (`docs/Dev reports/tts_audio_suite_aux_models.yaml`) is source of truth for helper/post-process model docs → run `python3 scripts/generate_aux_model_docs.py`
- All models download to `ComfyUI/models/TTS/<model-name>/`
- Engine registry: `utils/models/engine_registry.py`
- All models download to `ComfyUI/models/TTS/<model-name>/`
- Engine registry: `utils/models/engine_registry.py`
## Engines
12 engines follow the pattern above:
19 engines follow the pattern above:
| Engine | Adapter | Processor | SRT Processor | Engine Node |
|--------|---------|-----------|---------------|-------------|
| ChatterBox | `chatterbox_adapter.py` | `nodes/chatterbox/chatterbox_tts_node.py` | `chatterbox_srt_node.py` | `chatterbox_engine_node.py` |
| ChatterBox 23-Lang | `chatterbox_streaming_adapter.py` | `nodes/chatterbox_official_23lang/` | same folder | `chatterbox_engine_node.py` |
| F5-TTS | `f5tts_adapter.py` | `nodes/f5tts/f5tts_node.py` | `f5tts_srt_node.py` | `f5tts_engine_node.py` |
| Higgs Audio 2 | `higgs_audio_adapter.py` | — | `nodes/higgs_audio/higgs_audio_srt_processor.py` | `higgs_audio_engine_node.py` |
| VibeVoice | `vibevoice_adapter.py` | `nodes/vibevoice/vibevoice_processor.py` | — | `vibevoice_engine_node.py` |
| ChatterBox | `chatterbox_adapter.py` | `nodes/chatterbox/chatterbox_tts_node.py` | `chatterbox_srt_node.py` | `chatterbox_engine_node.py` |
| ChatterBox 23-Lang | `chatterbox_official_23lang_adapter.py` | `nodes/chatterbox_official_23lang/` | same folder | `chatterbox_official_23lang_engine_node.py` |
| F5-TTS | `f5tts_adapter.py` | `nodes/f5tts/f5tts_node.py` | `f5tts_srt_node.py` | `f5tts_engine_node.py` |
| Higgs Audio 2 | `higgs_audio_adapter.py` | — | `nodes/higgs_audio/higgs_audio_srt_processor.py` | `higgs_audio_engine_node.py` |
| Higgs Audio v3 | `higgs_audio_v3_adapter.py` | `nodes/higgs_audio_v3/higgs_audio_v3_processor.py` | `higgs_audio_v3_srt_processor.py` | `higgs_audio_v3_engine_node.py` |
| VibeVoice | `vibevoice_adapter.py` | `nodes/vibevoice/vibevoice_processor.py` | — | `vibevoice_engine_node.py` |
| IndexTTS-2 | — | `engines/index_tts/` | — | `index_tts_engine_node.py` |
| Step Audio EditX | `step_audio_editx_adapter.py` | `nodes/step_audio_editx/step_audio_editx_processor.py` | `step_audio_editx_srt_processor.py` | `step_audio_editx_engine_node.py` |
| CosyVoice3 | `cosyvoice_adapter.py` | `engines/processors/cosyvoice_processor.py` | `nodes/cosyvoice/cosyvoice_srt_processor.py` | `cosyvoice_engine_node.py` |
| Qwen3-TTS | `qwen3_tts_adapter.py` | `nodes/qwen3_tts/qwen3_tts_processor.py` | same folder | `qwen3_tts_engine_node.py` |
| MOSS-TTS | `moss_tts_adapter.py` | `nodes/moss_tts/moss_tts_processor.py` | `moss_tts_srt_processor.py` | `moss_tts_engine_node.py` |
| Granite ASR | `asr_granite_adapter.py` | — | — | `granite_asr_engine_node.py` |
| Echo-TTS | `echo_tts_adapter.py` | `nodes/echo_tts/echo_tts_processor.py` | `echo_tts_srt_processor.py` | `echo_tts_engine_node.py` |
| Fish Audio S2 Pro | `fish_audio_s2_adapter.py` | `nodes/fish_audio_s2/fish_audio_s2_processor.py` | `fish_audio_s2_srt_processor.py` | `fish_audio_s2_engine_node.py` |
| Dots TTS | `dots_tts_adapter.py` | `nodes/dots_tts/dots_tts_processor.py` | `dots_tts_srt_processor.py` | `dots_tts_engine_node.py` |
| DramaBox | `dramabox_adapter.py` | `nodes/dramabox/dramabox_processor.py` | `dramabox_srt_processor.py` | `dramabox_engine_node.py` |
| OmniVoice | `omnivoice_adapter.py` | `nodes/omnivoice/omnivoice_processor.py` | `omnivoice_srt_processor.py` | `omnivoice_engine_node.py` |
| MOSS-SoundEffect v2 | `moss_soundeffect_v2_adapter.py` | — | — | `moss_soundeffect_v2_engine_node.py` |
| RVC | — | `engines/rvc/` | — | `rvc_engine_node.py` |
**Engine implementations live in:**
- `engines/chatterbox/`, `engines/chatterbox_official_23lang/`, `engines/f5tts/`, `engines/higgs_audio/`, `engines/vibevoice_engine/`, `engines/step_audio_editx/`, `engines/cosyvoice/`, `engines/qwen3_tts/`, `engines/qwen3_asr/`, `engines/granite_asr/`, `engines/rvc/`
**Engine implementations live in:**
- `engines/chatterbox/`, `engines/chatterbox_official_23lang/`, `engines/f5tts/`, `engines/higgs_audio/`, `engines/higgs_audio_v3/`, `engines/vibevoice_engine/`, `engines/step_audio_editx/`, `engines/cosyvoice/`, `engines/qwen3_tts/`, `engines/qwen3_asr/`, `engines/moss_tts/`, `engines/moss_soundeffect_v2/`, `engines/granite_asr/`, `engines/echo_tts/`, `engines/fish_audio_s2/`, `engines/dots_tts/`, `engines/dramabox/`, `engines/omnivoice/`, `engines/rvc/`
## Documentation Files
**README.md** - Main project docs, installation, features overview
**CLAUDE.md** - Dev guidelines for Claude Code
**CHANGELOG.md** - Full version history
**README.md** - Main project docs, installation, features overview
**CLAUDE.md** - Dev guidelines for Claude Code
**CHANGELOG.md** - Full version history
**docs/BUMP_SCRIPT_INSTRUCTIONS.md** - Version bump process
### User Docs (`docs/`)
- `CHARACTER_SWITCHING_GUIDE.md` - [CharacterName] tag system
- `PARAMETER_SWITCHING_GUIDE.md` - Per-segment parameter override syntax
- `INLINE_EDIT_TAGS_USER_GUIDE.md` - Step Audio EditX inline tags
- `COSYVOICE3_TAGS_GUIDE.md` - CosyVoice3 native paralinguistic tags
- `INLINE_EDIT_TAGS_USER_GUIDE.md` - Step Audio EditX inline tags
- `HIGGS_AUDIO_V3_INLINE_TAGS.md` - Higgs Audio v3 native paralinguistic tags
- `OMNIVOICE_TAGS_GUIDE.md` - OmniVoice native non-verbal tags and pronunciation overrides
- `MOSS_TTS_PROMPT_FIELDS_GUIDE.md` - Official MOSS whole-segment prompt fields and inline `<>` translation limits
- `DRAMABOX_PROMPTING_GUIDE.md` - DramaBox expressive scene prompts, voice references, controls, hardware, and license
- `COSYVOICE3_TAGS_GUIDE.md` - CosyVoice3 native paralinguistic tags
- `CHATTERBOX_V2_SPECIAL_TOKENS.md` - ChatterBox v2 emotion tokens
- `IndexTTS2_Emotion_Control_Guide.md` - IndexTTS-2 emotion vectors
- `IndexTTS2_Emotion_Control_Guide.md` - IndexTTS-2 vector, text, audio, and blended emotion controls
- `VOCAL_REMOVAL_GUIDE.md` - Vocal separation guide
- `qwen3_tts_optimizations.md` - Qwen3-TTS torch.compile setup
- `MODEL_DOWNLOAD_SOURCES.md` - All HF repo links (auto-generated)
@@ -62,12 +77,25 @@
- `AUX_MODEL_SOURCES.md`, `AUX_MODEL_LAYOUTS.md` - Auxiliary helper-model registries (auto-generated)
- `ENGINE_COMPARISON.md`, `LANGUAGE_SUPPORT.md`, `FEATURE_COMPARISON.md` - Auto-generated tables
### Dev Docs (`docs/Dev reports/`)
### Dev Docs (`docs/Dev reports/`)
- `REGISTRY_COMPATIBILITY_PLAN.md` - File access, safe loading, config parsing, runtime installation/preferences, archive fixes, verification and Registry approval evidence
- `tts_audio_suite_engines.yaml` - **Source of truth** for all engine metadata
- `tts_audio_suite_aux_models.yaml` - **Source of truth** for helper/post-process model metadata
- `BUMP_SCRIPT_INSTRUCTIONS.md` - Version bump process
- `SRT_IMPLEMENTATION.md` - SRT timing technical details
- `ISOLATED_RUNTIMES_PLAN.md` - original runtime isolation plan and scope
- `TRANSFORMERS_5_QWEN3_TTS_REPORT.md` - why Qwen3-TTS moved to shared legacy T4 runtime
- `TRANSFORMERS_5_HIGGS_AUDIO_REPORT.md` - why Higgs Audio 2 moved to shared legacy T4 runtime
### New Engine Guides (`docs/New Engines Guides/`)
- `README.md` - Start-here friendly workflow for users guiding LLMs through new engine integrations
- `01_FIRST_ASK_THE_LLM_TO_RESEARCH_THE_OFFICIAL_MODEL.md` - Required official model capability audit before coding
- `02_CHECK_EXISTING_COMFYUI_IMPLEMENTATIONS.md` - How to use existing ComfyUI implementations as references, not source of truth
- `03_DECIDE_ENGINE_SCOPE.md` - Decide whether the engine needs TTS, Voice Changer, ASR, or special nodes; TTS implies SRT support
- `04_IMPLEMENTATION_ORDER_FOR_LLM.md` - Step-by-step implementation order with Qwen3-TTS and Step Audio EditX reference rules
- `05_REQUIRED_PARITY_CHECKLIST.md` - Must-pass architecture, lifecycle, cache, tag, SRT, docs, and UX checklist
- `06_USER_PROMPTS_TO_COPY_PASTE.md` - Ready prompts for users instructing an LLM
- `07_PR_REVIEW_CHECKLIST.md` - Maintainer/contributor checklist before merge
## Core Files
**`__init__.py`** - ComfyUI entry point
@@ -76,41 +104,59 @@
## Node Implementations
### Unified Interface
- `nodes/unified/tts_text_node.py` - Universal TTS text node
- `nodes/unified/tts_srt_node.py` - Universal SRT node
- `nodes/unified/voice_changer_node.py` - Universal voice conversion
- `nodes/unified/asr_transcribe_node.py` - Universal ASR node
### Unified Interface
- `nodes/unified/tts_text_node.py` - Universal TTS text node
- `nodes/unified/tts_srt_node.py` - Universal SRT node
- `nodes/unified/voice_changer_node.py` - Universal voice conversion
- `nodes/unified/asr_transcribe_node.py` - Universal ASR node
### Shared / Special Nodes
### Shared / Special Nodes
- `nodes/shared/character_voices_node.py` - Character voice management (NARRATOR_VOICE output)
- `nodes/shared/unified_voice_designer_node.py` - Unified Qwen VoiceDesign, MOSS VoiceGenerator, and reference-free OmniVoice design
- `nodes/shared/save_character_voice_node.py` - Explicit output node for saving any NARRATOR_VOICE into the established voice library
- `nodes/omnivoice/omnivoice_instruction_builder_node.py` - OmniVoice voice-design instruction helper with custom visual builder UI
- `nodes/text/phoneme_text_normalizer_node.py` - Multilingual text preprocessing
- `nodes/text/asr_punctuation_truecase_node.py` - Standalone punctuation / truecase cleanup for raw ASR text
- `nodes/subtitles/text_to_srt_builder_node.py` - Build SRT from transcript text plus timing data, or estimate timings from plain text
- `nodes/subtitles/srt_advanced_options_node.py` - Subtitle readability / segmentation policy options
- `nodes/text/tts_tag_editor_node.py` - 🏷️ Multiline TTS Tag Editor: rich text editor with character/language/parameter dropdowns, preset system, syntax highlighting, undo/redo — pairs with `web/string_multiline_tag_editor.js`
- `nodes/step_audio_editx_special/step_audio_editx_audio_editor_node.py` - 🎨 Audio Editor: post-process ANY engine's audio with Step Audio EditX (14 emotions, 32 styles, paralinguistic effects like `<Laughter>`, speed control) — universal, not just for Step Audio EditX engine
- `nodes/step_audio_editx_special/step_audio_editx_audio_editor_node.py` - 🎨 Audio Editor: post-process ANY engine's audio with Step Audio EditX (emotion/style, paralinguistic effects, speed); dated checkpoint choices and tags documented in `docs/STEP_AUDIO_EDITX_CHECKPOINTS.md`
- `nodes/engines/index_tts_emotion_options_node.py` - IndexTTS-2 emotion radar chart
- `nodes/qwen3_tts/qwen3_tts_voice_designer_node.py` - Qwen3 voice-from-text-description
### Audio / Video Nodes
- `nodes/audio/analyzer_node.py` - Audio Wave Analyzer
- `nodes/audio/vocal_removal_node.py` - Vocal/instrumental separation
- `nodes/audio/recorder_node.py` - Microphone recording
- `nodes/audio/recorder_node.py` - Saved microphone clip output with independent recording, preview, and trimming controls
- `nodes/audio/merge_audio_node.py` - Audio mixing
- `nodes/video/mouth_movement_analyzer_node.py` - Silent speech timing extractor
- `nodes/models/load_rvc_model_node.py` - RVC model loader
## Utility Systems
### Model Management (`utils/models/`)
- `unified_model_interface.py` - Universal factory pattern for all engines
- `engine_registry.py` - Engine capability definitions
- `manager.py` - Model discovery and caching
- `comfyui_model_wrapper/` - ComfyUI native model management integration
- `extra_paths.py` - extra_model_paths.yaml support
### Model Management (`utils/models/`)
- `unified_model_interface.py` - Universal factory pattern for all engines
- `engine_registry.py` - Engine capability definitions
- `factory_config.py` - standardized model load config, runtime mode/profile normalization
- `step_audio_editx_checkpoints.py` - Dated Step Audio EditX choices and pinned Hugging Face revisions
- `manager.py` - Model discovery and caching
- `comfyui_model_wrapper/` - ComfyUI native model management integration
- `extra_paths.py` - extra_model_paths.yaml support
### Isolated Runtimes (`utils/runtimes/`)
- `profiles.py` - named runtime profiles (shared/dedicated legacy Transformers 4 environments and engine-specific runtimes)
- `launcher.py` - worker environment construction, suite-first import paths, Windows toolchain env setup
- `session.py`, `protocol.py` - JSONL worker transport and message protocol
- `bootstrap.py` - shared runtime bootstrap helpers
- `vibevoice_proxy.py`, `qwen3_tts_proxy.py`, `qwen3_asr_proxy.py`, `higgs_audio_proxy.py` - parent-process proxies
- `workers/` - worker subprocess entrypoints for VibeVoice, Qwen3-TTS, Qwen3-ASR/aligner, Higgs Audio, Fish Audio S2, and Step Audio EditX; each prioritizes suite imports before third-party dependencies
- Current shared legacy T4 runtime profile is reused by:
- Qwen3-TTS
- Qwen3-ASR and Granite's optional Qwen forced aligner
- Higgs Audio 2
### Audio (`utils/audio/`)
### Audio (`utils/audio/`)
- `voice_capture.py`, `voice_capture_api.py` - Background microphone recording and standalone capture controls; saved clips in ComfyUI input/voice_capture/
- `trim.py` - Shared audio-range selection for Character Voices and Voice Capture
- `processing.py` - Tensor manipulation, normalization, format conversion
- `cache.py` - Unified TTS caching system
- `chunk_combiner.py` - Smart chunk combination (auto/concatenate/crossfade/silence_padding)
@@ -134,9 +180,12 @@
- `parser.py` - SRT parsing and validation
- `reporting.py` - Timing report generation
### Other Utils
- `utils/voice/discovery.py` - Voice file discovery with multi-path fallback
- `utils/downloads/unified_downloader.py` - Centralized HF download system
### Other Utils
- `utils/voice/discovery.py` - Voice discovery, user-voice priority, and multi-path fallback
- `utils/voice/designers.py` - Whitelisted voice-designer provider registry
- `utils/voice/character_saver.py` - Shared `.wav` / `.reference.txt` / `.txt` character persistence
- `utils/voice/character_logging.py` - Shared resolved voice labels and boxed prompt previews
- `utils/downloads/unified_downloader.py` - Centralized HF download system
- `utils/compatibility/transformers_patches.py` - transformers version compatibility patches
- `utils/compatibility/numba_compat.py` - Numba/Librosa Python 3.13+ compatibility
- `utils/ffmpeg_utils.py` - FFmpeg with graceful fallback
@@ -150,13 +199,27 @@
### Audio Analyzer
`web/audio_analyzer_*.js` (core, ui, visualization, regions, controls, widgets, drawing, events, layout, node_integration)
### Other Web Files
- `web/chatterbox_voice_capture.js` - Microphone recording UI
- `web/index_tts_emotion_radar.js` + `emotion_radar_canvas_widget.js` - IndexTTS-2 radar chart
- `web/qwen3_tts_widgets.js` - Qwen3 conditional instruction field
- `web/asr_srt_preset_widgets.js` - ASR SRT preset locking
### Other Web Files
- `web/chatterbox_voice_capture.js` - Microphone recording UI
- `web/index_tts_emotion_radar.js` + `emotion_radar_canvas_widget.js` - IndexTTS-2 radar chart
- `web/qwen3_tts_widgets.js` - Qwen model-specific widget enablement and legacy workflow migration
- `web/asr_srt_preset_widgets.js` - ASR SRT preset locking
## Scripts & Config
- `.comfyignore` - Registry archive exclusions for development tools and reports; keeps runtime metadata YAML, user documentation, and engine/training code
- `scripts/bump_version_enhanced.py` - Version bump with changelog (use `patch`/`minor`/`major`)
- `scripts/generate_engine_tables.py` - Regenerate all docs from YAML (`--readme` flag for README too)
- `requirements.txt`, `pyproject.toml` - Dependencies and project metadata
## Registry compatibility helpers
- `utils/security/path_access.py` - Configured data-folder containment, link checks and safe output paths
- `utils/security/data_loading.py` - Restricted checkpoint/dictionary/NumPy loading and scoped dependency adapters
- `utils/security/demucs_loading.py`, `audio_separator_loading.py` - Preserve legacy separator metadata with restricted dependency loaders
- `utils/downloads/unified_downloader.py` - Model download destinations contained in registered data folders
- `utils/security/config_values.py` - Data-only config expressions with bounded list arithmetic
- `utils/runtimes/bootstrap.py` - Runtime lookup/readiness only; generation never installs packages
- `utils/runtimes/installation.py` - Installation-time preparation invoked by `install.py`
- `utils/runtimes/settings.py`, `settings_api.py` - One persisted installation preference and same-origin UI writes
- `web/runtime_installation_settings.js` - Shared runtime installation setting and status/cleanup details
- `web/runtime_mode_compatibility.js` - Dedicated-to-shared migration for older workflows
+451 -342
View File
File diff suppressed because it is too large Load Diff
+342 -215
View File
@@ -15,7 +15,48 @@ Unified architecture supporting ChatterBox, F5-TTS, and future engines like RVC:
import importlib.util
import os
import sys
import time
# ComfyUI 0.12+ owns a top-level ``utils`` package, while this long-standing
# node pack also imports its helpers through ``utils.*``. Preserve ComfyUI's
# loaded package and extend only its module search path with this pack's utils.
_project_root = os.path.dirname(__file__)
_suite_utils_root = os.path.abspath(os.path.join(_project_root, "utils"))
if _project_root in sys.path:
sys.path.remove(_project_root)
sys.path.insert(0, _project_root)
_loaded_utils = sys.modules.get("utils")
if _loaded_utils is None:
import utils as _loaded_utils
_utils_search_path = getattr(_loaded_utils, "__path__", None)
if _utils_search_path is None:
raise ImportError(
"TTS Audio Suite cannot extend the loaded top-level 'utils' module because it is not a package"
)
_normalized_utils_paths = {os.path.normcase(os.path.abspath(path)) for path in _utils_search_path}
if os.path.normcase(_suite_utils_root) not in _normalized_utils_paths:
_utils_search_path.insert(0, _suite_utils_root)
# When this pack is imported before ComfyUI imports its own helpers, locate the
# active ComfyUI utils directory by its stable core modules and add it as the
# fallback side of the same package search path.
for _search_root in sys.path:
_candidate_utils = os.path.abspath(os.path.join(_search_root or os.curdir, "utils"))
_normalized_candidate = os.path.normcase(_candidate_utils)
if _normalized_candidate in _normalized_utils_paths or _normalized_candidate == os.path.normcase(_suite_utils_root):
continue
if all(os.path.isfile(os.path.join(_candidate_utils, filename)) for filename in ("extra_config.py", "install_util.py")):
_utils_search_path.append(_candidate_utils)
_normalized_utils_paths.add(_normalized_candidate)
from utils.hf_download_logging import configure_hf_download_logging
# Keep every engine's Hugging Face download output readable. Download failures
# are still reported by the suite's downloader error handling.
configure_hf_download_logging()
# Note: PyTorch inductor patches removed - not needed for PyTorch 2.10+ with triton-windows 3.6+
# Qwen3-TTS torch.compile optimizations require:
@@ -70,25 +111,25 @@ def _apply_transformers_patches_once():
except Exception as e:
print(f"⚠️ Warning: Could not apply Transformers patches: {e}")
# Numba/Librosa compatibility check at startup.
# Do NOT force NUMBA_DISABLE_JIT on Python 3.13 anymore:
# newer stacks (for example numba 0.64 + librosa 0.11) can work normally,
# and forcing the env var can itself trigger the get_call_template crash.
# For older Python + NumPy 2.x, keep the existing thorough compatibility test.
if sys.version_info < (3, 13):
try:
import numpy as _np
if int(_np.__version__.split('.')[0]) >= 2:
numba_compat_path = os.path.join(os.path.dirname(__file__), "utils", "compatibility", "numba_compat.py")
_spec = importlib.util.spec_from_file_location("numba_compat_module", numba_compat_path)
_numba_compat = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(_numba_compat)
_numba_compat.setup_numba_compatibility(quick_startup=False, verbose=True)
except Exception:
# If the compatibility test itself crashes, that means numba JIT is broken —
# disable it and warn the user.
os.environ['NUMBA_DISABLE_JIT'] = '1'
print("⚠️ TTS Audio Suite: Numba JIT crash detected at startup — disabling JIT (NUMBA_DISABLE_JIT=1)")
# Numba/Librosa compatibility check at startup.
# Do NOT force NUMBA_DISABLE_JIT on Python 3.13 anymore:
# newer stacks (for example numba 0.64 + librosa 0.11) can work normally,
# and forcing the env var can itself trigger the get_call_template crash.
# For older Python + NumPy 2.x, keep the existing thorough compatibility test.
if sys.version_info < (3, 13):
try:
import numpy as _np
if int(_np.__version__.split('.')[0]) >= 2:
numba_compat_path = os.path.join(os.path.dirname(__file__), "utils", "compatibility", "numba_compat.py")
_spec = importlib.util.spec_from_file_location("numba_compat_module", numba_compat_path)
_numba_compat = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(_numba_compat)
_numba_compat.setup_numba_compatibility(quick_startup=False, verbose=True)
except Exception:
# If the compatibility test itself crashes, that means numba JIT is broken —
# disable it and warn the user.
os.environ['NUMBA_DISABLE_JIT'] = '1'
print("⚠️ TTS Audio Suite: Numba JIT crash detected at startup — disabling JIT (NUMBA_DISABLE_JIT=1)")
# TorchCodec note: Removed torchcodec dependency to eliminate FFmpeg system requirement
# torchaudio.load() works fine with fallback backends (soundfile, scipy)
@@ -115,8 +156,9 @@ def check_dependencies():
print(f"{'='*80}")
print(f"The following required packages are missing: {', '.join(missing)}")
print(f"")
print(f"Please run the installation script or install them manually:")
print(f"pip install -r requirements.txt")
install_script = os.path.join(os.path.dirname(__file__), "install.py")
print(f"Please run the TTS Audio Suite installation script:")
print(f'"{sys.executable}" "{install_script}"')
print(f"{'='*80}\n")
# Version disclosure for troubleshooting
@@ -150,33 +192,7 @@ def print_critical_versions():
print(f"ℹ️ Critical package versions: {', '.join(version_info)}")
def warn_transformers_5_unsupported():
"""Warn when Transformers 5.x is installed (Qwen3-TTS tokenizer is incompatible).
NOTE: This check uses sys.modules to avoid eagerly importing transformers (~1.3s).
If transformers hasn't been imported yet (e.g. by the version-printing function above),
we skip the check -- it will be caught later when an engine actually loads transformers.
"""
try:
# Only check if transformers is already loaded (avoids ~1.3s eager import)
import sys as _sys
if 'transformers' not in _sys.modules:
return
transformers = _sys.modules['transformers']
try:
from packaging.version import Version
version = Version(transformers.__version__)
is_5x = version >= Version("5.0.0")
except Exception:
parts = transformers.__version__.split(".")
is_5x = int(parts[0]) >= 5 if parts and parts[0].isdigit() else False
if is_5x:
print("⚠️ Transformers 5.x detected: Qwen3-TTS tokenizer is incompatible.")
print(" Please downgrade to transformers<=4.57.3 (see requirements.txt).")
except Exception:
pass
def check_ffmpeg_availability():
def check_ffmpeg_availability():
"""Check ffmpeg availability and log status"""
try:
# Load ffmpeg_utils directly by file path to avoid package import issues
@@ -205,10 +221,9 @@ def check_ffmpeg_availability():
print("⚠️ FFmpeg not found - using fallback audio processing (reduced quality)")
print("💡 Install FFmpeg for optimal performance: https://ffmpeg.org/download.html")
# Print versions and check dependencies immediately for troubleshooting
check_dependencies()
print_critical_versions()
warn_transformers_5_unsupported()
# Print versions and check dependencies immediately for troubleshooting
check_dependencies()
print_critical_versions()
check_ffmpeg_availability()
# Check for old ChatterBox extension conflict
@@ -283,24 +298,33 @@ if 'utils' in sys.modules:
for key in to_delete:
del sys.modules[key]
# Get the path to the nodes.py file
nodes_py_path = os.path.join(os.path.dirname(__file__), "nodes.py")
# In pytest harness mode, avoid bootstrapping full ComfyUI node graph.
if os.environ.get("COMFYUI_TESTING") == "1":
IS_DEV = False
VERSION = "test"
SEPARATOR = "=" * 70
VERSION_DISPLAY = "test"
NODE_CLASS_MAPPINGS = {}
NODE_DISPLAY_NAME_MAPPINGS = {}
else:
# Get the path to the nodes.py file
nodes_py_path = os.path.join(os.path.dirname(__file__), "nodes.py")
# Load nodes.py as a module
spec = importlib.util.spec_from_file_location("nodes_main", nodes_py_path)
nodes_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(nodes_module)
# Load nodes.py as a module
spec = importlib.util.spec_from_file_location("nodes_main", nodes_py_path)
nodes_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(nodes_module)
# Import constants and utilities
IS_DEV = nodes_module.IS_DEV
VERSION = nodes_module.VERSION
SEPARATOR = nodes_module.SEPARATOR
VERSION_DISPLAY = nodes_module.VERSION_DISPLAY
# Import constants and utilities
IS_DEV = nodes_module.IS_DEV
VERSION = nodes_module.VERSION
SEPARATOR = nodes_module.SEPARATOR
VERSION_DISPLAY = nodes_module.VERSION_DISPLAY
# The new unified architecture handles all node registration in nodes.py
# Just import the mappings that nodes.py creates
NODE_CLASS_MAPPINGS = nodes_module.NODE_CLASS_MAPPINGS
NODE_DISPLAY_NAME_MAPPINGS = nodes_module.NODE_DISPLAY_NAME_MAPPINGS
# The new unified architecture handles all node registration in nodes.py
# Just import the mappings that nodes.py creates
NODE_CLASS_MAPPINGS = nodes_module.NODE_CLASS_MAPPINGS
NODE_DISPLAY_NAME_MAPPINGS = nodes_module.NODE_DISPLAY_NAME_MAPPINGS
# Extension info
__version__ = VERSION_DISPLAY
@@ -310,138 +334,143 @@ __description__ = "Universal multi-engine TTS extension for ComfyUI with unified
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
# Define web directory for JavaScript files (settings UI)
WEB_DIRECTORY = "./web"
def _debug_timing_log(label: str, start_time: float, extra: str = ""):
"""Print consistent startup timing diagnostics for debug branches."""
elapsed = time.perf_counter() - start_time
suffix = f" | {extra}" if extra else ""
print(f"⏱️ [startup-debug] {label}: {elapsed:.3f}s{suffix}")
# Register API endpoint for widget data
WEB_DIRECTORY = "./web"
# Register API endpoint for widget data
def setup_api_routes():
"""Setup API routes for widget communication"""
try:
import json
from server import PromptServer
from aiohttp import web
"""Setup API routes for widget communication"""
try:
import json
import folder_paths
from server import PromptServer
from aiohttp import web
from utils.security.path_access import resolve_input_path, filename_component, child_path
def _get_ui_data_dir():
base_dir = os.path.join(folder_paths.get_system_user_directory("tts_audio_suite"), "ui")
os.makedirs(base_dir, exist_ok=True)
return base_dir
def _get_omnivoice_preset_library_path():
return os.path.join(_get_ui_data_dir(), "omnivoice_instruction_builder_presets.json")
from utils.voice.alias_api import register_character_alias_routes
register_character_alias_routes(PromptServer.instance.routes, web)
from utils.runtimes.settings_api import register_runtime_settings_routes
register_runtime_settings_routes(PromptServer.instance.routes, web)
from utils.audio.voice_capture_api import register_voice_capture_routes
register_voice_capture_routes(PromptServer.instance.routes, web)
@PromptServer.instance.routes.get("/api/tts-audio-suite/index-tts-emotion-presets")
async def get_index_tts_emotion_presets_endpoint(request):
"""Return presets stored beside the IndexTTS resources under models/TTS."""
try:
from .utils.text.index_tts_emotion import load_emotion_presets
return web.json_response({"presets": load_emotion_presets()})
except Exception as e:
print(f"⚠️ Error retrieving IndexTTS emotion presets: {e}")
return web.json_response({"presets": {}, "error": str(e)}, status=500)
@PromptServer.instance.routes.post("/api/tts-audio-suite/index-tts-emotion-presets")
async def save_index_tts_emotion_presets_endpoint(request):
"""Atomically persist the IndexTTS emotion preset library."""
try:
from .utils.text.index_tts_emotion import save_emotion_presets
data = await request.json()
presets = data.get("presets", {})
path = save_emotion_presets(presets)
return web.json_response({"status": "success", "count": len(presets), "path": path})
except ValueError as e:
return web.json_response({"error": str(e)}, status=400)
except Exception as e:
print(f"⚠️ Error saving IndexTTS emotion presets: {e}")
return web.json_response({"status": "error", "error": str(e)}, status=500)
@PromptServer.instance.routes.get("/api/tts-audio-suite/available-characters")
async def get_available_characters_endpoint(request):
"""API endpoint to get available TTS character voices including aliases"""
start_time = time.perf_counter()
try:
# Load voice discovery directly by file path to avoid package import issues
voice_discovery_path = os.path.join(os.path.dirname(__file__), "utils", "voice", "discovery.py")
spec = importlib.util.spec_from_file_location("voice_discovery_module", voice_discovery_path)
voice_discovery_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(voice_discovery_module)
characters = list(voice_discovery_module.get_available_characters())
# Also get character aliases
aliases = list(voice_discovery_module.voice_discovery._character_aliases.keys()) if hasattr(voice_discovery_module.voice_discovery, '_character_aliases') else []
from utils.voice import discovery as voice_discovery_module
characters = list(voice_discovery_module.get_available_characters())
aliases = list(voice_discovery_module.voice_discovery.get_character_aliases().keys())
# Combine and deduplicate
all_chars = sorted(set(characters + aliases))
_debug_timing_log("/available-characters", start_time, f"count={len(all_chars)}")
return web.json_response({"characters": all_chars})
except Exception as e:
print(f"⚠️ Error retrieving available characters: {e}")
_debug_timing_log("/available-characters", start_time, f"error={type(e).__name__}")
return web.json_response({"characters": [], "error": str(e)})
return web.json_response({"characters": all_chars})
except Exception as e:
print(f"⚠️ Error retrieving available characters: {e}")
return web.json_response({"characters": [], "error": str(e)})
@PromptServer.instance.routes.get("/api/tts-audio-suite/available-languages")
async def get_available_languages_endpoint(request):
"""API endpoint to get available language codes from the canonical language mapper"""
start_time = time.perf_counter()
try:
# Load language_mapper directly by file path to avoid package import issues
language_mapper_path = os.path.join(os.path.dirname(__file__), "utils", "models", "language_mapper.py")
"""API endpoint to get available language codes from the canonical language mapper"""
try:
# Load language_mapper directly by file path to avoid package import issues
language_mapper_path = os.path.join(os.path.dirname(__file__), "utils", "models", "language_mapper.py")
spec = importlib.util.spec_from_file_location("language_mapper_module", language_mapper_path)
language_mapper_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(language_mapper_module)
# Get all unique canonical language codes (the values in LANGUAGE_ALIASES)
languages = sorted(set(language_mapper_module.LANGUAGE_ALIASES.values()))
_debug_timing_log("/available-languages", start_time, f"count={len(languages)}")
return web.json_response({"languages": languages})
except Exception as e:
# Get all unique canonical language codes (the values in LANGUAGE_ALIASES)
languages = sorted(set(language_mapper_module.LANGUAGE_ALIASES.values()))
return web.json_response({"languages": languages})
except Exception as e:
print(f"⚠️ Error retrieving available languages: {e}")
_debug_timing_log("/available-languages", start_time, f"error={type(e).__name__}")
# Fallback list
return web.json_response({"languages": ["en", "de", "fr", "ja", "es", "it", "pt", "th", "no"], "error": str(e)})
@PromptServer.instance.routes.get("/api/tts-audio-suite/voice-input-devices")
async def get_voice_input_devices_endpoint(request):
"""Return input devices without risking a main-process PortAudio hang."""
start_time = time.perf_counter()
@PromptServer.instance.routes.get("/api/tts-audio-suite/omnivoice-presets")
async def get_omnivoice_presets_endpoint(request):
"""Return the persisted OmniVoice instruction builder preset library."""
try:
import json
import subprocess
probe_script = r"""
import json
import sounddevice as sd
devices = []
seen = set()
for device in sd.query_devices():
try:
max_input_channels = int(device.get("max_input_channels", 0))
except Exception:
max_input_channels = 0
if max_input_channels <= 0:
continue
name = str(device.get("name", "")).strip()
if not name or name in seen:
continue
seen.add(name)
devices.append(name)
print(json.dumps({"devices": devices}))
"""
result = subprocess.run(
[sys.executable, "-c", probe_script],
capture_output=True,
text=True,
timeout=8,
check=False,
)
if result.returncode != 0:
stderr = (result.stderr or "").strip()
stdout = (result.stdout or "").strip()
error_message = stderr or stdout or f"device probe exited with code {result.returncode}"
return web.json_response({"devices": [], "error": error_message}, status=500)
payload = json.loads(result.stdout or "{}")
devices = payload.get("devices", [])
if not isinstance(devices, list):
devices = []
_debug_timing_log("/voice-input-devices", start_time, f"count={len(devices)}")
return web.json_response({"devices": devices})
except subprocess.TimeoutExpired:
_debug_timing_log("/voice-input-devices", start_time, "timeout")
return web.json_response(
{"devices": [], "error": "Timed out while probing audio input devices. Leaving the dropdown on system default avoids startup hangs."},
status=504,
)
library_path = _get_omnivoice_preset_library_path()
if not os.path.exists(library_path):
return web.json_response({"presets": [], "builtinStates": {}, "builtinLayouts": {}})
with open(library_path, "r", encoding="utf-8") as f:
payload = json.load(f)
presets = payload.get("presets", []) if isinstance(payload, dict) else []
builtin_states = payload.get("builtinStates", {}) if isinstance(payload, dict) else {}
builtin_layouts = payload.get("builtinLayouts", {}) if isinstance(payload, dict) else {}
if not isinstance(presets, list):
presets = []
if not isinstance(builtin_states, dict):
builtin_states = {}
if not isinstance(builtin_layouts, dict):
builtin_layouts = {}
return web.json_response({"presets": presets, "builtinStates": builtin_states, "builtinLayouts": builtin_layouts})
except Exception as e:
print(f"⚠️ Error retrieving voice input devices: {e}")
_debug_timing_log("/voice-input-devices", start_time, f"error={type(e).__name__}")
return web.json_response({"devices": [], "error": str(e)}, status=500)
print(f"⚠️ Error retrieving OmniVoice preset library: {e}")
return web.json_response({"presets": [], "builtinStates": {}, "builtinLayouts": {}, "error": str(e)}, status=500)
@PromptServer.instance.routes.post("/api/tts-audio-suite/settings")
async def set_inline_tag_settings_endpoint(request):
"""API endpoint to receive settings from frontend for inline edit tags and restore VC"""
start_time = time.perf_counter()
print("🔧 Settings endpoint called") # Immediate print to verify endpoint is reached
@PromptServer.instance.routes.post("/api/tts-audio-suite/omnivoice-presets")
async def save_omnivoice_presets_endpoint(request):
"""Persist the OmniVoice instruction builder preset library."""
try:
data = await request.json()
presets = data.get("presets", [])
builtin_states = data.get("builtinStates", {})
builtin_layouts = data.get("builtinLayouts", {})
if not isinstance(presets, list):
return web.json_response({"error": "presets must be a list"}, status=400)
if not isinstance(builtin_states, dict):
return web.json_response({"error": "builtinStates must be an object"}, status=400)
if not isinstance(builtin_layouts, dict):
return web.json_response({"error": "builtinLayouts must be an object"}, status=400)
library_path = _get_omnivoice_preset_library_path()
payload = {"presets": presets, "builtinStates": builtin_states, "builtinLayouts": builtin_layouts}
with open(library_path, "w", encoding="utf-8") as f:
json.dump(payload, f, ensure_ascii=False, indent=2)
return web.json_response({"status": "success", "count": len(presets)})
except Exception as e:
print(f"⚠️ Error saving OmniVoice preset library: {e}")
return web.json_response({"status": "error", "error": str(e)}, status=500)
@PromptServer.instance.routes.post("/api/tts-audio-suite/settings")
async def set_inline_tag_settings_endpoint(request):
"""API endpoint to receive settings from frontend for inline edit tags and restore VC"""
print("🔧 Settings endpoint called") # Immediate print to verify endpoint is reached
try:
data = await request.json()
precision = data.get("precision", "auto")
device = data.get("device", "auto")
vc_engine = data.get("vc_engine", "chatterbox_23lang")
@@ -452,32 +481,51 @@ print(json.dumps({"devices": devices}))
# Import edit_post_processor using normal import to ensure we get the same module instance
# that will be used during workflow execution
# CRITICAL: Must use the same module instance, not create a new one via importlib!
try:
import_start = time.perf_counter()
from utils.audio import edit_post_processor as edit_post_processor_module
_debug_timing_log("/settings import edit_post_processor", import_start, "import_mode=normal")
except ImportError:
# Fallback: Load directly by file path if normal import fails
edit_post_processor_path = os.path.join(os.path.dirname(__file__), "utils", "audio", "edit_post_processor.py")
spec = importlib.util.spec_from_file_location("utils.audio.edit_post_processor", edit_post_processor_path)
edit_post_processor_module = importlib.util.module_from_spec(spec)
sys.modules["utils.audio.edit_post_processor"] = edit_post_processor_module # Register in sys.modules!
import_start = time.perf_counter()
spec.loader.exec_module(edit_post_processor_module)
_debug_timing_log("/settings import edit_post_processor", import_start, "import_mode=fallback")
# Store in global settings that edit_post_processor can access
edit_post_processor_module.set_inline_tag_settings(precision=precision, device=device, vc_engine=vc_engine, cosyvoice_variant=cosyvoice_variant)
_debug_timing_log("/settings", start_time, "status=success")
return web.json_response({"status": "success", "precision": precision, "device": device, "vc_engine": vc_engine, "cosyvoice_variant": cosyvoice_variant})
except Exception as e:
print(f"⚠️ Error setting inline tag settings: {e}")
_debug_timing_log("/settings", start_time, f"error={type(e).__name__}")
return web.json_response({"status": "error", "error": str(e)})
try:
from utils.audio import edit_post_processor as edit_post_processor_module
except ImportError:
# Fallback: Load directly by file path if normal import fails
edit_post_processor_path = os.path.join(os.path.dirname(__file__), "utils", "audio", "edit_post_processor.py")
spec = importlib.util.spec_from_file_location("utils.audio.edit_post_processor", edit_post_processor_path)
edit_post_processor_module = importlib.util.module_from_spec(spec)
sys.modules["utils.audio.edit_post_processor"] = edit_post_processor_module # Register in sys.modules!
spec.loader.exec_module(edit_post_processor_module)
@PromptServer.instance.routes.get("/api/tts-audio-suite/voice-preview")
async def get_voice_preview_endpoint(request):
# Store in global settings that edit_post_processor can access
edit_post_processor_module.set_inline_tag_settings(precision=precision, device=device, vc_engine=vc_engine, cosyvoice_variant=cosyvoice_variant)
return web.json_response({"status": "success", "precision": precision, "device": device, "vc_engine": vc_engine, "cosyvoice_variant": cosyvoice_variant})
except Exception as e:
print(f"⚠️ Error setting inline tag settings: {e}")
return web.json_response({"status": "error", "error": str(e)})
def get_voice_discovery_module():
"""Return the shared discovery module used by nodes and save notifications."""
from utils.voice import discovery as voice_discovery_module
return voice_discovery_module
def resolve_character_voice(voice_name):
"""Resolve a dropdown key through the shared discovery cache."""
voice_discovery_module = get_voice_discovery_module()
voice_discovery_module.get_available_voices(force_refresh=False)
return voice_discovery_module.load_voice_reference(voice_name)
@PromptServer.instance.routes.get("/api/tts-audio-suite/voice-library")
async def get_voice_library_endpoint(request):
"""Return current dropdown keys for Character Voices."""
try:
voice_discovery_module = get_voice_discovery_module()
force_refresh = request.query.get("refresh", "0").strip().lower() in {"1", "true", "yes"}
voices = voice_discovery_module.get_available_voices(force_refresh=force_refresh)
response = web.json_response({"voices": voices})
response.headers["Cache-Control"] = "no-store, no-cache, must-revalidate"
return response
except Exception as e:
print(f"⚠️ Error serving voice library: {e}")
return web.json_response({"error": str(e)}, status=500)
@PromptServer.instance.routes.get("/api/tts-audio-suite/voice-preview")
async def get_voice_preview_endpoint(request):
"""
Stream selected Character Voices dropdown audio for browser preview playback.
@@ -489,15 +537,7 @@ print(json.dumps({"devices": devices}))
if not voice_name or voice_name == "none":
return web.json_response({"error": "voice_name is required and cannot be 'none'"}, status=400)
# Load voice discovery directly by file path to avoid package import issues
voice_discovery_path = os.path.join(os.path.dirname(__file__), "utils", "voice", "discovery.py")
spec = importlib.util.spec_from_file_location("voice_discovery_module", voice_discovery_path)
voice_discovery_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(voice_discovery_module)
# Use cached discovery for fast preview playback.
voice_discovery_module.get_available_voices(force_refresh=False)
audio_path, _ = voice_discovery_module.load_voice_reference(voice_name)
audio_path, _ = resolve_character_voice(voice_name)
if not audio_path or not os.path.exists(audio_path):
return web.json_response({"error": f"Voice file not found: {voice_name}"}, status=404)
@@ -507,12 +547,99 @@ print(json.dumps({"devices": devices}))
response.headers["Cache-Control"] = "no-store, no-cache, must-revalidate"
return response
except Exception as e:
print(f"⚠️ Error serving voice preview audio: {e}")
return web.json_response({"error": str(e)}, status=500)
print(f"⚠️ Error serving voice preview audio: {e}")
return web.json_response({"error": str(e)}, status=500)
@PromptServer.instance.routes.get("/api/tts-audio-suite/voice-info")
async def get_voice_info_endpoint(request):
"""Return canonical metadata for a Character Voices dropdown entry."""
try:
voice_name = request.query.get("voice_name", "").strip()
if not voice_name or voice_name == "none":
return web.json_response({"error": "voice_name is required and cannot be 'none'"}, status=400)
audio_path, reference_text = resolve_character_voice(voice_name)
if not audio_path or not os.path.exists(audio_path):
return web.json_response({"error": f"Voice file not found: {voice_name}"}, status=404)
response = web.json_response({
"voice_name": voice_name,
"reference_text": reference_text or "",
})
response.headers["Cache-Control"] = "no-store, no-cache, must-revalidate"
return response
except Exception as e:
print(f"⚠️ Error serving voice metadata: {e}")
return web.json_response({"error": str(e)}, status=500)
@PromptServer.instance.routes.post("/api/tts-audio-suite/audio-analyzer-preview")
async def audio_analyzer_preview_endpoint(request):
"""
Analyze file-based Audio Wave Analyzer inputs without queueing the ComfyUI graph.
Connected AUDIO inputs still require graph execution because the browser/backend
route cannot access an upstream tensor that has not been computed.
"""
try:
data = await request.json()
audio_file = (data.get("audio_file") or "").strip()
if not audio_file:
return web.json_response({"error": "audio_file is required for preview analysis"}, status=400)
audio_file = resolve_input_path(audio_file)
node_id = filename_component(data.get("node_id") or "preview")
analyzer_node_path = os.path.join(os.path.dirname(__file__), "nodes", "audio", "analyzer_node.py")
spec = importlib.util.spec_from_file_location("tts_audio_suite_audio_analyzer_node", analyzer_node_path)
analyzer_node_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(analyzer_node_module)
analyzer_node = analyzer_node_module.AudioAnalyzerNode()
analysis_result = analyzer_node.analyze_audio(
audio_file=audio_file,
analysis_method=data.get("analysis_method", "silence"),
precision_level=data.get("precision_level", "milliseconds"),
visualization_points=int(data.get("visualization_points", 2000)),
audio=None,
options=data.get("options"),
manual_regions=data.get("manual_regions", ""),
region_labels=data.get("region_labels", ""),
export_format=data.get("export_format", "f5tts"),
node_id=node_id,
)
import folder_paths
if analysis_result[1].startswith("Error:"):
return web.json_response({"error": analysis_result[1]}, status=400)
cache_file = child_path(folder_paths.get_output_directory(), f"audio_analyzer_cache_{node_id}.json")
with open(cache_file, "r", encoding="utf-8") as f:
payload = json.load(f)
response = web.json_response(payload)
response.headers["Cache-Control"] = "no-store, no-cache, must-revalidate"
return response
except Exception as e:
print(f"⚠️ Audio analyzer preview failed: {e}")
return web.json_response({"error": str(e)}, status=400 if isinstance(e, (ValueError, FileNotFoundError)) else 500)
@PromptServer.instance.routes.get("/api/tts-audio-suite/training-progress")
async def get_training_progress_endpoint(request):
"""Return live training progress snapshots for one or all tracked training nodes."""
try:
from engines.training.progress_registry import get_training_progress_snapshot
node_id = request.query.get("node_id")
snapshot = get_training_progress_snapshot(node_id=node_id)
response = web.json_response({"nodes": snapshot})
response.headers["Cache-Control"] = "no-store, no-cache, must-revalidate"
return response
except Exception as e:
print(f"⚠️ Error retrieving training progress: {e}")
return web.json_response({"nodes": {}, "error": str(e)}, status=500)
except Exception as e:
print(f"⚠️ Could not setup API routes: {e}")
# Setup API routes when extension loads
setup_api_routes()
# nodes.py already handles all the startup output and status reporting
# nodes.py already handles all the startup output and status reporting
+9 -5
View File
@@ -2,7 +2,7 @@
## Overview
Advanced multiline string editor node that extends ComfyUI's standard multiline widget with a sidebar containing context-aware controls for TTS-specific tags (character switching, parameters, pauses). The node features real-time tag generation, preset management, and intelligent syntax support based on the CHARACTER_SWITCHING_GUIDE and PARAMETER_SWITCHING_GUIDE.
Advanced multiline string editor node that extends ComfyUI's standard multiline widget with a sidebar containing context-aware controls for TTS-specific tags (character switching, parameters, pauses, and engine-native inline controls). The node features real-time tag generation, preset management, engine-aware quick swapping, and intelligent syntax support based on the CHARACTER_SWITCHING_GUIDE and PARAMETER_SWITCHING_GUIDE.
---
@@ -85,10 +85,14 @@ Advanced multiline string editor node that extends ComfyUI's standard multiline
- Split by paragraph breaks or sentence punctuation
- Batch apply parameters across multiple `[Character]text` blocks
#### Tag Inspector
- Show existing tags in selection/current line
- Checkbox UI to toggle tags on/off temporarily
- Quick-edit dialog for existing tag values
#### Tag Inspector
- Show existing tags in selection/current line
- Checkbox UI to toggle tags on/off temporarily
- Click a character, language, audio-reference, parameter, or supported engine-native inline tag to open a color-coded quick-swap palette
- Click a palette option once to keep the palette open, then click again to commit; press-and-hold, drag, and release selects in one gesture
- Palette choices follow the selected inline engine (including IndexTTS-2, Higgs Audio v3, Step Audio EditX, CosyVoice3, and OmniVoice)
- Quoted IndexTTS-2 text emotion tags remain direct editable text and do not open a replacement palette
- Long bracket and angle tags wrap inside the editor rather than overflowing horizontally
#### Auto-Formatting
- Button: "Auto-Format Tags" → organize tags consistently
+38 -31
View File
@@ -12,21 +12,26 @@
**⚠️ IMPORTANT: Use positional arguments, NOT --commit/--changelog flags**
```bash
# EASIEST: Just use 'patch' - script auto-increments the version
python3 scripts/bump_version_enhanced.py patch "<commit_desc>" "<changelog_desc>"
```powershell
# Windows: use the canonical ComfyUI environment for this project
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' patch "<commit_desc>" "<changelog_desc>"
# OR: Specify exact version if needed
python3 scripts/bump_version_enhanced.py <version> "<commit_desc>" "<changelog_desc>"
# OR: specify an exact version if needed
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' <version> "<commit_desc>" "<changelog_desc>"
```
```bash
# Linux/macOS: use the available Python 3 interpreter
python3 scripts/bump_version_enhanced.py patch "<commit_desc>" "<changelog_desc>"
```
### Examples
#### Multiline Format (Recommended Standard)
```bash
```powershell
# Patch release (bug fixes) - CORRECT FORMAT
python3 scripts/bump_version_enhanced.py 3.2.9 "Fix character alias resolution
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' 3.2.9 "Fix character alias resolution
Technical details:
- Fix parser bypassing character tags in single mode
@@ -37,8 +42,8 @@ Technical details:
- Improve character name recognition accuracy
- Better error handling for invalid character names"
# Minor release (new features) - CORRECT FORMAT
python3 scripts/bump_version_enhanced.py 3.3.0 "Add Higgs Audio 2 TTS engine
# Minor release (new features) - CORRECT FORMAT
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' 3.3.0 "Add Higgs Audio 2 TTS engine
Implementation details:
- Integrate boson_multimodal voice cloning system
@@ -50,7 +55,7 @@ Implementation details:
- Multiple built-in voice presets available"
# Major release (breaking changes) - CORRECT FORMAT
python3 scripts/bump_version_enhanced.py 4.0.0 "Complete unified architecture implementation
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' 4.0.0 "Complete unified architecture implementation
Breaking changes:
- Migrate all nodes to unified interface pattern
@@ -64,22 +69,22 @@ Breaking changes:
```
#### Auto-Increment Examples (Recommended)
```bash
# Auto-increment patch version (4.5.25 → 4.5.26) - CORRECT FORMAT
python3 scripts/bump_version_enhanced.py patch "Fix character parsing issues" "Fix character name handling in TTS generation"
```powershell
# Auto-increment patch version (4.5.25 → 4.5.26) - Windows
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' patch "Fix character parsing issues" "Fix character name handling in TTS generation"
# Auto-increment minor version (4.5.25 → 4.6.0) - CORRECT FORMAT
python3 scripts/bump_version_enhanced.py minor "Add new TTS engine support" "Add Higgs Audio 2 TTS engine with voice cloning"
# Auto-increment minor version (4.5.25 → 4.6.0) - Windows
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' minor "Add new TTS engine support" "Add Higgs Audio 2 TTS engine with voice cloning"
```
#### Single-Line Format (Only for Super Minor Changes)
```bash
python3 scripts/bump_version_enhanced.py patch "Fix typo in node tooltip" "Fix typo in audio analyzer tooltip"
```powershell
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' patch "Fix typo in node tooltip" "Fix typo in audio analyzer tooltip"
```
#### Dry-Run Preview (Test Before Committing)
```bash
python3 scripts/bump_version_enhanced.py patch "Fix preview issues" "Fix preview not reflecting filter parameters" --dry-run
```powershell
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' patch "Fix preview issues" "Fix preview not reflecting filter parameters" --dry-run
```
#### Auto-Categorization System
@@ -99,21 +104,22 @@ python3 scripts/bump_version_enhanced.py patch "Fix preview issues" "Fix preview
- **Commit**: Technical implementation details for developers
- **Changelog**: User-facing benefits and impacts
**Bash Syntax Notes:**
**Command Syntax Notes:**
- Multiline strings need proper quoting (opening quote on first line, closing quote on last line)
- Use `\` (backslash) for line continuation in bash commands
- The Windows command uses PowerShell's `&` call operator and the canonical project Python path
- The Linux/macOS command uses `python3`
- Don't add manual category prefixes like "Fixed:" - script handles categorization automatically!
### Interactive Mode (Recommended for Complex Changes)
```bash
python3 scripts/bump_version_enhanced.py 3.2.9 --interactive
```powershell
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' 3.2.9 --interactive
```
### Legacy Mode (Same Description for Both)
```bash
python3 scripts/bump_version_enhanced.py 3.2.9 "Fix bugs and improve stability"
```powershell
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' 3.2.9 "Fix bugs and improve stability"
```
### What the Script Does
@@ -239,13 +245,14 @@ git commit -m "Prepare for version bump"
- Use semantic versioning: `4.5.25` (not `v4.5.25` or `4.5`)
- Or use auto-increment: `patch`, `minor`, `major`
**Bash syntax errors with multiline**
- Make sure opening quote is on same line as `--commit` or `--changelog`
**Command syntax errors with multiline**
- Make sure opening quote is on same line as `--commit` or `--changelog`
- Make sure closing quote is on its own line
- Use `\` for line continuation
- On Windows, use the PowerShell command shown above
- On Linux/macOS, use `python3`
**Want to see what will happen before committing?**
```bash
```powershell
# Add --dry-run to preview changelog categorization
python3 scripts/bump_version_enhanced.py patch "description" "changelog" --dry-run
```
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' patch "description" "changelog" --dry-run
```
+79 -410
View File
@@ -1,410 +1,79 @@
# 🎭 Character & Narrator Switching Guide
## Overview
ChatterBox Voice now supports seamless character and narrator switching with **language-aware processing** for both **F5TTS** and **ChatterBox TTS** engines. Use `[CharacterName]` or `[language:CharacterName]` tags in your text to automatically switch between different voices and languages, creating dynamic multilingual multi-character audio content.
---
## ✨ Key Features
- **🎭 Universal Character Parsing** - Works with both F5TTS and ChatterBox engines
- **🌍 Language-Aware Switching** - Switch both character AND language with `[lang:character]` syntax
- **⚙️ Per-Segment Parameters** - Control seed, temperature, cfg, and other parameters per segment with `[Alice|seed:42|temperature:0.5]` syntax (see [Parameter Switching Guide](PARAMETER_SWITCHING_GUIDE.md))
- **🔄 Robust Fallback System** - No errors when characters or languages not found
- **📁 Voice Folder Integration** - Organized character voice management
- **📺 SRT Support** - Character and language switching in subtitle timing
- **🗂️ Default Language Settings** - Set default languages per character in alias files
- **⚡ Performance Optimized** - Preserves all existing caching systems
- **🔙 Backward Compatible** - Existing workflows work unchanged
---
## 🚀 Quick Start
### 1. Text Format
Use `[CharacterName]` tags to switch voices, with optional language specification:
#### Basic Character Switching
```
Hello! This is the narrator speaking.
[Alice] Hi there! I'm Alice, nice to meet you.
[Bob] And I'm Bob! Great to meet you both.
Back to the narrator for the conclusion.
```
#### Language-Aware Character Switching (NEW!)
```
Hello! This is English narrator.
[de:Alice] Hallo! Ich bin Alice und spreche Deutsch.
[fr:Bob] Bonjour! Je suis Bob et je parle français.
[Alice] Alice now speaks in her default language.
Back to the narrator in default language.
```
**Language Tag Format:** `[language:character]`
- `[de:Alice]` - Alice speaks in German
- `[fr:Bob]` - Bob speaks in French
- `[en:narrator]` - Narrator speaks in English
- `[Alice]` - Alice uses her default language (from alias settings or global default)
### 2. Supported Languages
#### F5-TTS Language Models
- **English** (`en`) - F5TTS_Base, F5TTS_v1_Base, E2TTS_Base
- **German** (`de`) - F5-DE
- **Spanish** (`es`) - F5-ES
- **French** (`fr`) - F5-FR
- **Japanese** (`jp`, `ja`) - F5-JP
- **Italian** (`it`) - F5-IT
- **Thai** (`th`) - F5-TH
- **Portuguese Brazilian** (`pt`, `pt-br`) - F5-PT-BR
#### ChatterBox Language Models
- **English** (`en`) - English
- **German** (`de`) - German
- **Norwegian** (`no`, `nb`, `nn`) - Norwegian
**Note:** If a language model is not available, the system will fallback to the default model with a warning.
### 3. Voice File Structure
Organize character voices using filenames in `models/voices` (preferred) or in the custom_node TTS Audio Suite folder under `voices_examples/`:
```
voices_examples/
├── narrator.wav (or .mp3, .flac, .ogg, .m4a, .aac)
├── narrator.reference.txt (for F5TTS only)
├── alice.mp3
├── alice.reference.txt (for F5TTS only)
├── bob.flac
├── bob.reference.txt (for F5TTS only)
└── characters/ (folders for organization)
├── female_01.wav
├── female_01.reference.txt
├── male_01.ogg
└── male_01.reference.txt
```
**Character names are determined by the audio filename, not folder names. Folders are for organization only.**
**Supported audio formats:** `.wav`, `.mp3`, `.flac`, `.ogg`, `.m4a`, `.aac`
### 3. Engine Differences
- **F5TTS**: Requires both **audio files** and `.reference.txt` files
- **ChatterBox**: Only needs **audio files** (simpler setup)
---
## 📖 Detailed Usage
### Character Tag Syntax
- **Format**: `[CharacterName]Text content here`
- **Case**: Character names are case-insensitive (`[Alice]` = `[alice]`)
- **Punctuation**: Automatically cleaned (`[Alice:]` → `alice`)
- **Fallback**: Unknown characters use narrator voice automatically
### Example Multi-Character Script
```
Welcome to our story! This is the narrator.
[Alice] Hello everyone! I'm excited to be here.
[Bob] Nice to meet you, Alice. I'm Bob.
[Alice] Great to meet you too, Bob!
[Wizard] I am the ancient wizard...
And the narrator concludes the tale.
```
### SRT Subtitle Example
```srt
1
00:00:01,000 --> 00:00:04,000
Hello! This is F5-TTS SRT with character switching.
2
00:00:04,500 --> 00:00:09,500
[Alice] Hi there! I'm Alice speaking with precise timing.
3
00:00:10,000 --> 00:00:14,000
[Bob] And I'm Bob! The audio matches these exact SRT timings.
```
---
## 🛠️ Setup Instructions
### For F5TTS Nodes
1. **Add Character Voice Files**:
```
voices_examples/alice.wav (or .mp3, .flac, etc.)
voices_examples/alice.reference.txt
voices_examples/bob.mp3
voices_examples/bob.reference.txt
```
2. **Voice File Requirements**:
- `alice.wav` - Audio sample of Alice's voice (5-15 seconds recommended)
- `alice.reference.txt` - Transcript of what Alice says in the audio
- **Supported formats:** `.wav`, `.mp3`, `.flac`, `.ogg`, `.m4a`, `.aac`
3. **⚠️ F5-TTS Best Practices**: Follow [F5-TTS inference guidelines](#f5-tts-inference-guidelines) to avoid generation failures
4. **Reference Text Example**:
```
Hello, this is Alice speaking clearly and naturally.
```
### For ChatterBox Nodes
1. **Add Audio Files Only**:
```
voices_examples/alice.wav (or .mp3, .flac, etc.)
voices_examples/bob.ogg
```
- No text files needed!
- **Supported formats:** `.wav`, `.mp3`, `.flac`, `.ogg`, `.m4a`, `.aac`
2. **Flexible Organization**:
```
voices_examples/
├── main_characters/
│ ├── alice.wav
│ └── bob.mp3
└── background_voices/
├── shopkeeper.flac
└── guard.ogg
```
### Alternative Voice Sources
- **ComfyUI Models**: `models/voices/` directory (same filename-based system)
- **Flexible Organization**: Any subfolder structure supported for organization
### 🏷️ Character Aliases with Language Defaults (Optional)
Create a `#character_alias_map.txt` file to use friendly names and set default languages:
```
# Character Alias Map
# Supported formats:
# Alias = Character_Name
# Alias = Character_Name, default_language
# Alias[TAB]Character_Name[TAB]default_language
# Main Characters with language defaults:
Alice female_01 de
Bob male_01 fr
Narrator david_attenborough_cc3
# Supporting Cast with language defaults:
Girl = female_01, en
Woman = female_02, en
# Supporting Cast:
Girl = female_01
Woman = female_02
Cowboy = clint_eastwood_cc3_enhanced2
# Background Voices:
Old Man male_01
Shopkeeper male_02
```
**Benefits:**
- Use `[Alice]` instead of `[female_01]` in your text
- Multiple aliases can point to the same voice file
- Flexible format: supports both `=` and tab separation
- Comments and empty lines are ignored
---
## 🔄 Fallback System
The system gracefully handles missing characters:
1. **Character Found**: Uses character-specific voice
2. **Character Not Found**:
- ⚠️ Warning message: `Using main voice for character 'Unknown' (not found in voice folders)`
- 🔄 Automatically uses narrator/main reference voice
- ✅ **No errors or workflow interruption**
### Example Fallback Behavior
```
[Alice] This uses Alice's voice (if available)
[UnknownCharacter] This falls back to narrator voice
[Bob] This uses Bob's voice (if available)
```
---
## 💡 Tips & Best Practices
### Voice Recording
- **Duration**: 5-15 seconds per character voice but it's not mandatory
- **Quality**: Clear, noise-free recordings
- **Content**: Natural speech that represents the character
- **Format**: WAV, MP3, FLAC supported
### Character Naming
- **Consistency**: Use the same character names throughout
- **Simplicity**: Avoid special characters in names
- **Organization**: Group related characters in subfolders
### Reference Text (F5TTS)
- **Accuracy**: Must match the audio exactly
- **Clarity**: Write exactly what is spoken
- **Length**: Should match audio duration
### Performance Optimization
- **Caching**: Character voices are cached automatically
- **Chunking**: Long character segments are chunked intelligently
- **Reuse**: Same character voices used across multiple generations
---
## 🐛 Troubleshooting
### Common Issues
#### "Character not found" warnings
- **Cause**: Character audio file missing or incorrectly named
- **Solution**: Check that audio filename matches character name used in `[Character]` tags
- **Note**: Supports `.wav`, `.mp3`, `.flac`, `.ogg`, `.m4a`, `.aac` formats
- **Result**: Uses fallback voice (no workflow break)
#### F5TTS missing reference text
- **Cause**: `.reference.txt` file missing for character
- **Solution**: Add reference text file matching audio
- **Alternative**: Use ChatterBox engine (no text required)
#### Audio quality inconsistent
- **Cause**: Different recording conditions per character
- **Solution**: Record all characters with similar setups
- **Tip**: Use consistent volume and background noise levels
### Debugging
Detailed logging to see character and language detection:
- Character switching mode messages: `🎭 F5-TTS: Character switching mode`
- Language switching mode messages: `🌍 F5-TTS: Language switching mode`
- Voice loading messages: `🎭 Using character voice for 'Alice'`
- Model switching messages: `🌍 Loading F5-TTS model 'F5-DE' for language 'de'`
---
## 📈 Advanced Usage
### Nested Character Organization
```
voices_examples/
├── story1/
│ ├── hero.wav
│ ├── hero.reference.txt
│ ├── villain.mp3
│ └── villain.reference.txt
├── story2/
│ ├── alice.flac
│ ├── alice.reference.txt
│ ├── bob.ogg
│ └── bob.reference.txt
└── narrator.m4a
└── narrator.reference.txt
```
### Language Priority System
The language selection follows this priority order:
1. **Explicit language tag**: `[de:Alice]` (highest priority)
2. **Character default language**: Set in alias file (medium priority)
3. **Global default language**: From node settings (lowest priority)
Example:
```
# If Alice has default language 'de' in alias file:
[Alice] Uses German (from alias default)
[en:Alice] Uses English (explicit override)
[fr:Alice] Uses French (explicit override)
```
### Mixed Character Scenes
```
[Narrator] The scene opens in a busy marketplace.
[Merchant] Fresh apples! Get your fresh apples here!
[Customer] How much for a dozen?
[Merchant] Two coins, good sir!
[Narrator] The customer nodded and made the purchase.
```
### Multilingual Character Scenes (NEW!)
```
[en:Narrator] Welcome to our international marketplace!
[de:Merchant] Frische Äpfel! Holt euch frische Äpfel!
[fr:Customer] Combien pour une douzaine?
[de:Merchant] Zwei Münzen, mein Herr!
[en:Narrator] The transaction concluded successfully.
```
### Dynamic Character Assignment
- Characters are detected automatically from text
- No pre-configuration needed
- Add new characters by adding audio files with matching names
- Remove characters by deleting audio files
- Character name = audio filename (without extension)
- **Supports all audio formats:** `.wav`, `.mp3`, `.flac`, `.ogg`, `.m4a`, `.aac`
---
## 🎯 Integration Examples
### Story Narration
Perfect for audiobooks, stories, and educational content with multiple speakers.
### Dialogue Systems
Ideal for game dialogue, chatbots, and interactive applications.
### Educational Content
Great for language learning with different character voices.
### Accessibility
Helps distinguish speakers in audio content for better comprehension.
---
## 🔗 Related Features
- **[Voice Discovery System](../core/voice_discovery.py)**: Automatic character voice detection
- **[Audio Processing](../core/audio_processing.py)**: Smart audio chunking and combining
- **[SRT Integration](../chatterbox_srt/)**: Subtitle timing with character voices
- **[Caching System](../core/)**: Performance optimizations for character voices
---
## 📋 F5-TTS Inference Guidelines
To avoid possible inference failures when using F5-TTS character voices, make sure you follow these optimization guidelines:
1. **Reference Audio Duration**: Use reference audio <12s and leave proper silence space (e.g. 1s) at the end. Otherwise there is a risk of truncating in the middle of word, leading to suboptimal generation.
2. **Letter Case Handling**: Uppercased letters (best with form like K.F.C.) will be uttered letter by letter, and lowercased letters used for common words.
3. **Pause Control**: Add some spaces (blank: " ") or punctuations (e.g. "," ".") to explicitly introduce some pauses.
4. **Punctuation Spacing**: If English punctuation marks the end of a sentence, make sure there is a space " " after it. Otherwise not regarded as sentence chunk.
5. **Number Processing**: Preprocess numbers to Chinese letters if you want to have them read in Chinese, otherwise they will be read in English.
These guidelines help ensure optimal F5-TTS generation quality and prevent common audio artifacts.
---
## 📝 Version History
- **v3.0.13**: Initial character switching implementation
- **Future**: Enhanced character management UI, voice cloning improvements
---
*For technical support or feature requests, please check the main README or create an issue on GitHub.*
# Character and Narrator Switching
Use `[Alice]` and `[Bob]` tags in **TTS Text** or **TTS SRT** to select character voices. This is shared TTS Audio Suite syntax; reference requirements, languages, and native multi-speaker modes depend on the selected engine.
## Add character voices
Save your voices under `ComfyUI/models/voices/`:
```text
ComfyUI/models/voices/
├── alice.wav
├── alice.reference.txt
├── bob.wav
└── bob.reference.txt
```
- The audio filename gives the character its name: `alice.wav` becomes `[Alice]`. Names are case-insensitive. Subfolders are for organization; use unique filenames across your library.
- For engines that use a transcript, put the exact spoken words in `name.reference.txt`. A matching `name.txt` is also accepted when no reference transcript is found. Keep metadata in a separate file from the transcript.
- Transcript requirements vary by engine and mode; see **Reference Transcript** in the [Feature Comparison](FEATURE_COMPARISON.md).
The suite also searches `models/TTS/voices/`, bundled `voices_examples/`, and voice paths configured in `extra_model_paths.yaml`. Supported audio formats include WAV, MP3, FLAC, OGG, M4A, and AAC.
You can create library voices with **Save Character Voice**, using the `opt_narrator` output from **Character Voices** or **Voice Designer**.
## Set the narrator and write your script
Connect an engine to **TTS Text** or **TTS SRT**. For reference-based voice cloning, select the engine's cloning mode when it offers multiple modes. Choose the default voice with `narrator_voice`, or connect **Character Voices** to `opt_narrator`.
```text
Welcome to our story. This line uses the narrator.
[Alice] Hi, Bob!
[Bob] Hello, Alice.
[Alice] Shall we begin? [narrator] And so their journey started.
The next line uses the narrator again.
```
**Each new line starts with the narrator.** A character tag applies until the next tag on that line. Repeat `[Alice]` at the start of each line Alice should speak; use `[narrator]` to switch back within a line.
Use the same tags inside SRT subtitle text. Subtitle timing is controlled separately by the TTS SRT node's timing mode.
## Languages and segment settings
| Tag | Meaning |
| --- | --- |
| `[Alice]` | Select Alice's voice. |
| `[narrator]` | Select the narrator. |
| `[de:Alice]` | Select Alice and request German. |
| `[fr:]` | Request French with the narrator. |
| `[Alice\|seed:42]` | Select Alice and override the segment's seed. |
Language tags do not translate text or add languages to a model. Some engines infer language from the text; others accept an explicit language or load a language-specific model. See [Language Support](LANGUAGE_SUPPORT.md) for engine coverage and the [Parameter Switching Guide](PARAMETER_SWITCHING_GUIDE.md) for supported segment settings.
Language priority is: explicit tag → character's alias default → node/model default.
## Aliases (optional)
**Character Alias Manager demo (v5.5.3):**
https://github.com/user-attachments/assets/960a27d8-c0d8-45e6-b343-7d18bf85d1d6
Open the manager from **Character Voices** or the **Multiline TTS Tag Editor**, or use the **Open Character Alias Manager** command. Assign friendly names, choose voice files, and set optional default languages. Save changes to apply your mappings.
Alternatively, place `#character_alias_map.txt` in a voice-library root. For example, to address `female_01.wav` as Alice:
```text
Alice = female_01, de
Bob = male_01, en
```
`[Alice]` then selects `female_01` with German as its default language; `[en:Alice]` overrides that language. The language field is optional. Alias Manager user overrides take priority over mappings loaded from files.
## If the wrong voice speaks
- Check the audio filename or alias target. Unknown character names fall back to the narrator and produce a console warning.
- Check the transcript requirements for the engine and mode. If you trim a reference, update its transcript to match the retained speech.
- After adding or changing library files, use **Refresh Voice Cache** to rescan them.
For engine-specific audio preparation, see [F5-TTS Reference and Inference Guidelines](F5_TTS_REFERENCE_GUIDE.md). For visual tag insertion, see the [Multiline TTS Tag Editor Guide](MULTILINE_TTS_TAG_EDITOR_GUIDE.md).
+4 -2
View File
@@ -4,7 +4,9 @@
CosyVoice3 has **native paralinguistic tag support** built into the model. Unlike Step Audio EditX post-processing, these tags are processed **during TTS generation** for natural integration.
**⚠️ IMPORTANT**: Step Audio EditX post-processing is **DISABLED** for CosyVoice3. EditX syntax like `<Laughter:2>`, `<emotion:happy>`, `<style:whisper>` will be stripped but **not processed**. Use CosyVoice3 native tags instead.
**⚠️ IMPORTANT**: Step Audio EditX post-processing is **DISABLED** for CosyVoice3. EditX syntax like `<Laughter:2>`, `<emotion:happy>`, `<style:whisper>` will be stripped but **not processed**. Use CosyVoice3 native tags instead.
The multiline TTS tag editor has an engine-aware `Inline Tags` panel with a dedicated `CosyVoice3` mode that inserts these native tags directly.
---
@@ -216,7 +218,7 @@ A: Yes! Include tags in subtitle text.
## Related Guides
- **[Inline Edit Tags Guide](INLINE_EDIT_TAGS_USER_GUIDE.md)** - Step Audio EditX tags for other engines
- **[Step Audio EditX Inline Tags Guide](INLINE_EDIT_TAGS_USER_GUIDE.md)** - Step Audio EditX tags for other engines
- **[Parameter Switching Guide](PARAMETER_SWITCHING_GUIDE.md)** - Per-segment parameter control
---
+99
View File
@@ -0,0 +1,99 @@
# DramaBox LoRA training
TTS Audio Suite exposes the official DramaBox audio-branch IC-LoRA trainer
through the unified `🎓 Model Training` flow. The bundled scripts are pinned to
the same upstream DramaBox revision as the inference implementation.
See the official DramaBox
[LoRA training guide](https://github.com/resemble-ai/DramaBox#training-a-lora-on-top-of-dramabox)
for the upstream dataset format and training behavior.
## Workflow
1. Build a `⚙️ DramaBox Engine`.
2. Create the dataset either externally or entirely inside ComfyUI:
`🎞️ Training Clip Staging` → `🧾 DramaBox Dataset Rows`.
3. Connect the resulting manifest to `📦 DramaBox Dataset Prep` and keep
`dataset_type` set to `manifest`.
4. Provide at least two clips per speaker.
5. Connect the dataset to `🎛️ DramaBox Training Config` and then to `🎓 Model Training`.
6. Select the resulting adapter in the DramaBox engine, or enter its path in the
advanced LoRA override field.
The dataset node accepts:
- JSONL/JSON manifests with `audio_filepath` (or `audio_path`) and `text` (or
`transcript`)
- TSV rows with audio path and text
- the official `gemini_synthetic` and `libriheavy` index formats
Manifest rows may include `speaker`, `speaker_id`, `language`, and `duration`.
If `speaker` is omitted, rows are grouped as `speaker_1`. Duration and audio
metadata are measured without loading the waveform into the GPU. The suite
converts all accepted formats into the `~`-delimited speaker index required by
the upstream training loop. Clips are restricted to 2–20 seconds by default.
For an all-ComfyUI dataset, connect one or more `AUDIO` sources to
`🎞️ Training Clip Staging`, then enter one transcript per clip in
`🧾 DramaBox Dataset Rows`. Speaker and language lines are optional; shared
defaults are used when those lines are blank.
### Transcripts and scene descriptions
The official trainer accepts either plain spoken transcripts or the same
scene-style prompt format used for inference. For example, both of these are
valid training text:
```text
This is the spoken sentence.
A woman speaks warmly, "This is the spoken sentence."
```
Use scene descriptions only when they accurately describe the clip. Plain
transcripts remain valid and are the safer choice when no reliable style or
scene annotation is available.
## What training does
The first preprocessing pass uses Gemma and the DramaBox audio VAE to create
cached conditions and audio latents. The training process then attaches a LoRA
to the audio transformer branch. It saves periodic checkpoints and exports the
selected adapter to:
```text
ComfyUI/models/TTS/dramabox/loras/<adapter_name>/
```
The job directory, normalized index, preprocessing cache, progress file, and
logs are stored under:
```text
ComfyUI/output/tts_audio_suite_training/dramabox/
```
`continue_from` is a warm start from an existing LoRA checkpoint; it is not an
exact optimizer-state resume. Use saved checkpoints to compare quality rather
than assuming the last step is best. Optional upstream validation can be
enabled with a `val_config` YAML path, but it launches full DramaBox inference
at each save step. It requires a second GPU: set `validation_gpu` to that
physical CUDA device index. The suite rejects validation on the training GPU
instead of allowing both full model processes to compete for the same VRAM.
DramaBox LoRA inference supports normal transformer precision, `fp8_cast`, and
the optional `torch.compile` path. With normal precision the live adapter is
reversibly merged for fast inference. With FP8 storage the BF16 adapter remains
unmerged above the immutable FP8 base weights, avoiding unsafe mixed-dtype
weight fusion while retaining the main FP8 memory saving.
The base DramaBox runtime is reused when the selected adapter or LoRA strength
changes. Strength updates are applied directly to the live PEFT adapter, while
the generated-audio cache still treats adapter path, file revision, and strength
as distinct generation settings. Replacing an adapter with a different rank may
retrace compiled transformer blocks, but does not reload the base checkpoint.
## CPU-safe preflight
Training and Gemma/VAE preprocessing are GPU workloads. For development or
validation without touching CUDA, enable `dry_run` in the training config and
the dataset node's `dry_run`/`preprocess_now` controls. This writes the
normalized index and official command/config without loading DramaBox weights.
+163
View File
@@ -0,0 +1,163 @@
# DramaBox Prompting Guide
DramaBox is an English expressive TTS engine. It accepts ordinary narration,
dialogue in quotation marks, and natural-language stage directions in one
prompt.
## Basic prompts
Use quoted text for speech and surrounding prose for delivery:
```text
A tired detective speaks quietly in a rain-soaked office. "I knew this case would find me again."
```
`prompt_template` defaults to `"{seg}"`. `{seg}` is replaced by the current
plain fragment, so the default marks the whole fragment as literal spoken
dialogue. For example, `Hello.` becomes `"Hello."`. Clear the template field to
send plain text unchanged.
Customize the template to add delivery context, for example
`A man speaks warmly, "{seg}"`. Quote-only input is normalized without adding
another pair of quotes. Complete scene prompts with directions outside their
quotation marks remain unchanged. Every non-empty template must contain `{seg}`.
If it is omitted accidentally, TTS Audio Suite warns once and appends
`"{seg}"` automatically instead of failing generation.
For a one-segment override, `prompt_template` (or its `template` alias)
automatically enables templating for that segment and then reverts to the node
setting:
```text
[Narrator|template:A woman whispers, "{seg}"] This line uses a custom wrapper.
```
DramaBox can render non-verbal and delivery cues when they are described
naturally:
```text
She tries to stay serious, then breaks into a short laugh. "That is the worst excuse I have ever heard." She sighs and continues more gently. "But I believe you."
```
Write one continuous scene paragraph. DramaBox does not require newlines as
prompt syntax. In TTS Audio Suite, an untagged newline starts another generated
segment, so use prose action directions and quoted dialogue in the same
paragraph when they should remain one coherent DramaBox scene.
Do not use ChatterBox V2 special tokens such as `[giggle]`. DramaBox was
trained for prose-style scene direction, not that token vocabulary.
## Voice references
Reference audio is optional. Connect narrator audio or use a character voice
file to clone its speaker and delivery. Upstream uses the first 10 seconds, so
a clean single-speaker clip is the useful input; a transcript is not required.
Without a reference, DramaBox uses its built-in voice behavior.
DramaBox can occasionally produce a near-silent sample for a particular
combination of reference audio, reference duration, generation duration, and
seed. TTS Audio Suite checks the decoded waveform and prints a warning when both
its RMS and peak levels are conservatively near silence. The audio is preserved;
the suite does not retry or change parameters automatically. Try another
generation duration, reference duration/audio, guidance setting, or seed for
the affected segment. A different seed can help some combinations but is not a
guaranteed fix.
The warning is also propagated to node outputs. TTS Text includes affected
segments in `generation_info`. TTS SRT marks affected subtitle numbers in
`timing_report`, including the parameters that may be worth testing for that
segment.
## Character and pause tags
TTS Audio Suite character tags still work. Each tagged character is generated
as a separate DramaBox segment:
```text
[Alice] "We should leave now."
[Bob] He answers without looking up. "Give me one minute."
[pause:0.8]
[Alice] "You said that five minutes ago."
```
Suite pause tags create exact silence outside the model. Natural pauses inside
a spoken scene are better expressed in the prose prompt.
## Engine controls
- `cfg_scale`: text/prompt guidance. Official default: `2.5`.
- `stg_scale`: skip-token guidance. Official default: `1.5`.
- `duration_multiplier`: scales the estimated speaking duration. Official
default: `1.1`.
- `gen_duration`: explicit generated-audio duration from `0` to `60` seconds.
`0` keeps automatic prompt-based estimation.
- `ref_duration`: uses the first `3` to `30` seconds of a voice reference.
The default is `10`; audio later in the source file is ignored.
- `rescale_scale`: CFG latent rescaling. Use `auto` or a fixed value from
`0` to `1`.
- `watermark`: enables the optional official Perth output watermark. It is off
by default and requires Perth.
- `seed`: supplied by the unified TTS Text or SRT node.
Segment overrides support `seed`, `cfg_scale`, `stg_scale`, and
`duration_multiplier`, `gen_duration`, `ref_duration`, and `rescale_scale`.
Watermarking remains a whole-engine setting rather than a segment override.
DramaBox performs its own duration-aware long-form chunking. The suite does
not split a DramaBox scene by character count before passing it to the model.
Automatically estimated scenes above 45 seconds use text chunking. A nonzero
`gen_duration` remains one native generation so its explicit 0–60 second
target is preserved.
The unified SRT node's **Native Duration Targeting** option passes each
subtitle's duration to DramaBox before final timing assembly. For subtitles
containing multiple character or pause-separated fragments, the available
speech time is allocated proportionally after explicit pause durations and
inline `gen_duration` overrides are accounted for. The selected SRT timing
mode still performs its normal final correction.
## Negative Prompt and Segment Switching
DramaBox uses CFG and exposes its negative prompt in the engine node. The
default discourages robotic, distorted, noisy, muffled, unclear, and monotone
speech. Override it for one character segment with:
```text
[Alice|negative:robotic, muffled] "Keep this line clean and intimate."
[Bob|neg:noise, static] "This line uses a different negative prompt."
```
The segment override ends at the next character tag.
## Memory and Performance
- `fast` keeps all components on CUDA for the fastest repeated generation.
- `staged` is an experimental strategy for lowering peak VRAM. It loads and
releases Gemma, the voice encoder, and audio decoder by stage, at the cost of
reloading them for each generated segment or long-form chunk.
- `sequential` is a more aggressive experimental strategy for lowering peak
VRAM. It additionally keeps the diffusion transformer in system RAM while
another major stage uses CUDA. It transfers the transformer for every
generated segment or long-form chunk and is therefore substantially slower.
Actual peak usage varies with the environment, generation settings, and
other loaded components; no minimum GPU size is guaranteed.
System RAM must hold the offloaded transformer (about 3.4GB with FP8 or
6.6GB without it).
- `fp8_cast` uses the official LTX FP8 transformer weight-storage policy and
upcasts linear weights during inference. It can lower VRAM and may be slower.
- `compile_model` compiles the diffusion transformer blocks with DramaBox's
bundled LTX compilation path. The first generation can take substantially
longer while kernels compile; later denoising may be faster.
## Requirements and license
The full download is approximately 16.4GB and the official runtime requires
an NVIDIA CUDA GPU. Fast mode targets roughly 24GB VRAM; the experimental
staged modes can run with less memory at a speed cost. Output is 48kHz stereo.
The optional official Perth watermark is applied only when enabled and the
dependency is available.
DramaBox uses the LTX-2 Community License. Entities with at least USD 10
million in annual revenue require a separate paid commercial license. Review
the bundled license before production use.
@@ -0,0 +1,183 @@
# DramaBox and Chatterbox Multilingual V3 Capability and Scope
Research date: 2026-07-25
## Official references
- DramaBox code: `resemble-ai/DramaBox` at
`a70a5818e103c1c9fef22409c1e0c707ebf4f8a7`
- DramaBox weights: `ResembleAI/Dramabox` at
`404f967f653fa1170dc15a9d1ddd3fdb9a0a842d`
- Chatterbox code: `resemble-ai/chatterbox` at
`5de7a54aa4e5e2baadb0182dde554908b48b85c2`
- Chatterbox weights: `ResembleAI/chatterbox` at
`5bb1f6ee58e50c3b8d408bc82a6d3740c2db6e18`
- ComfyUI reference only: `kat3ri/ComfyUI-DramaBox` at
`715fcb11cc14d8c185438e2319b52fc00163941c`
The repositories were cloned under
`IgnoredForGitHubDocs/For_reference/`.
## DramaBox capability report
### Native scope
- Task: English text-to-speech with optional zero-shot voice cloning.
- Expressive control: prompt-driven speaker description, delivery, emotion,
pauses, laughs, sighs, and transitions.
- Voice input: optional reference audio; upstream uses up to 10 seconds.
- No native voice conversion, ASR, or audio editing API.
- No language control. The official model is English-only.
- No extra special node is required. Its structured scene prompt fits the
existing unified text and SRT nodes.
### Native generation parameters
- `cfg_scale` (official warm-server default `2.5`)
- `stg_scale` (default `1.5`)
- `duration_multiplier` (default `1.1`)
- `seed` (default `42`)
- `ref_duration` (default `10.0` seconds)
- `rescale_scale` (`auto` by default)
- `gen_duration` (`0` means automatic)
- Official long-form chunk limits and crossfade parameters
The initial suite UI exposes `cfg_scale`, `stg_scale`, and
`duration_multiplier`. Seed remains owned by the unified TTS nodes.
Reference duration, rescale, steps, modality guidance, and explicit output
duration stay on official defaults because exposing them would add expert
controls without a demonstrated suite use case. The official duration-aware
long-form path is used automatically instead of adding duplicate chunk UI.
### Audio and generation behavior
- The LTX audio decoder returns stereo audio at 48 kHz.
- The base model was trained on clips around 20 seconds. Current upstream
supports longer clips with a silence-prior correction and automatically
chunks prompts targeting about 37 seconds with a 45-second cap.
- The official long-form chunker preserves the scene/speaker prefix and quote
groups, then joins chunks with a 50 ms equal-power crossfade.
- Upstream applies the Perth watermark only in `generate_to_file()`, not in
the in-memory `generate()` method. The suite wrapper must therefore apply
the watermark to in-memory output explicitly.
### Model layout
Organized destination: `ComfyUI/models/TTS/dramabox/DramaBox/`
- `dramabox-dit-v1.safetensors` — 6,575,225,528 bytes
- `dramabox-audio-components.safetensors` — 1,942,831,020 bytes
- `assets/silence_latent_frame.pt` — 1,501 bytes
- `gemma-3-12b-it-bnb-4bit/`
- two safetensor shards plus tokenizer/config files from
`unsloth/gemma-3-12b-it-bnb-4bit`
The implementation must use the suite downloader with `local_dir`-style
organized downloads and disable Transformers/Hugging Face fallback downloads.
### Dependencies and runtime
The official requirements include Torch/Torchaudio 2.8, Transformers 4.45+,
bitsandbytes 0.45+, Accelerate, PEFT, PyAV, Einops, SentencePiece,
Safetensors, PyYAML, and Perth. The official source imports successfully in
the configured suite validation environment with Torch 2.10 and Transformers
5.10, so DramaBox belongs in the main Transformers 5 environment.
The optional NVIDIA RE-USE reference denoiser is intentionally excluded:
its Mamba dependencies have no practical Windows installation path and its
NSCLv1 non-commercial license is a poor default for the suite.
### License
DramaBox code and weights are under the LTX-2 Community License, not MIT.
The license requires attribution, use restrictions, modified-file notices,
and a separate paid license for entities with at least USD 10 million in
annual revenue. The upstream license must ship beside any bundled inference
code, and the engine UI/docs must disclose the restriction.
## Existing ComfyUI reference notes
`kat3ri/ComfyUI-DramaBox` confirms useful ComfyUI audio-shape handling,
organized model paths, the warm `TTSServer` API, and practical UI ranges.
It must not be copied as architecture:
- It auto-clones source code at runtime.
- It has no unified model lifecycle, cache, character/pause integration, SRT
processor, interrupt handling, or generation report integration.
- It directly calls the engine from a standalone node.
- It patches partially imported bitsandbytes modules globally.
- Its README says output is watermarked, but its node calls the unwatermarked
in-memory upstream path.
## Chatterbox Multilingual V3 capability report
V3 is not a new engine. Official upstream loads it as an opt-in checkpoint through
`ChatterboxMultilingualTTS.from_pretrained(..., t3_model="v3")`; the only
model-family change is selecting `t3_mtl23ls_v3.safetensors` instead of the
V2 T3 checkpoint. Its official generation path also skips the legacy
alignment analyzer, uses repetition penalty `1.2`, and removes the final
degraded pre-EOS speech-token artifact. It keeps
the same 500M architecture, 23-language list, 24 kHz output, voice-reference
mode, tokenizer, voice encoder, S3Gen decoder, and generation parameters:
- `language_id`
- `exaggeration`
- `cfg_weight`
- `temperature`
- `repetition_penalty`
- `min_p`
- `top_p`
The suite forwards V3 `exaggeration` using the upstream/native scale. Manual
testing found little or no audible response across values, so this remains a
current checkpoint limitation rather than a suite-side scaling issue.
The existing `chatterbox_official_23lang` engine already implements Unified
TTS Text, Unified SRT TTS, voice references, caching, character switching,
pause tags, parameter switching, and lifecycle handling. V3 therefore
extends its `model_version` choices and downloader requirements. It must not
create a second engine node or duplicate processors.
## Integration scope
### DramaBox
- Unified TTS Text: yes
- Unified SRT TTS: yes
- Character tags and narrator fallback: yes
- Pause tags: yes
- Segment switching: `seed`, `cfg_scale`, `stg_scale`, and
`duration_multiplier`
- Generated audio cache: yes
- Long-form strategy: official duration-aware chunker; ignore suite
character-count chunking inside each already separated character/pause
segment
- Clear VRAM: full TTSServer teardown and lazy reload because the quantized
Gemma stack should not be copied to system RAM
- Runtime: main environment
- Voice Changer / ASR / editing / special node: no
### Chatterbox Multilingual V3
- Extend existing Official 23-Lang model version control with V3.
- Keep V2 available for backward compatibility.
- Make V3 the suite default for new configurations so the newly requested
version is immediately selected. Upstream still defaults to V2 and exposes
V3 as opt-in.
- Existing saved workflows with V1/V2 values continue to load unchanged.
## Validation matrix
- Static import and registration checks
- Chatterbox V1/V2/V3 file-resolution tests without downloading weights
- DramaBox downloader layout checks without downloading the 16+ GB models
- DramaBox TTS processor tests with a fake adapter for character, pause,
cache-facing parameter, audio-shape, and combination behavior
- SRT processor interrupt and timing-path tests with fake generation
- Live FL-MCP checks after restarting ComfyUI:
- engine and unified nodes register
- smallest DramaBox text workflow loads
- smallest DramaBox SRT workflow loads
- full generation is attempted only if all 16+ GB weights and adequate
VRAM are available
- Human assessment remains required for subjective audio quality.
@@ -0,0 +1,71 @@
# Isolated Runtimes Plan
Historical design. The current implementation uses Main Environment or one Shared Runtime,
prepared by install.py. Dedicated profiles and first-use package installation were removed;
see [Registry compatibility work](REGISTRY_COMPATIBILITY_PLAN.md).
## Goal
Keep ComfyUI in one modern main runtime while routing fragile engines into
their own pinned Python environments.
## Why
- `transformers` major-version breaks now hit engines differently
- one broken HF stack should not poison the whole node runtime
- per-engine rollback is simpler than global dependency rollback
## User impact
- isolated runtimes duplicate dependencies on disk
- large packages such as PyTorch will usually be installed again inside the worker runtime
- this must be warned in:
- docs
- node tooltips
- runtime creation logs
## Contract
ComfyUI process:
- owns UI, node orchestration, model discovery, caching policy
- prepares normalized request payloads
- launches a worker only for engines configured as `runtime_mode=isolated`
Worker process:
- runs in a dedicated Python env
- receives a JSON request
- performs load / generate / unload actions
- returns JSON response + output artifact paths
## Initial profiles
- `vibevoice_transformers4`
- `qwen3_tts_transformers5`
- `step_audio_editx_transformers5`
- `moss_tts_transformers5`
## First target
VibeVoice / KugelAudio
Reason:
- highest dependency fragility
- strongest evidence that global `transformers 5` compatibility is not enough
- likely template for future HF-heavy engines
## Implementation order
1. Add config + registry support for runtime routing
2. Add launcher + request/response protocol
3. Add VibeVoice worker entrypoint
4. Add opt-in routing from unified interface
5. Add user-facing runtime path configuration
6. Expand to Qwen / MOSS / Step Audio only after VibeVoice works
## Non-goals
- no silent env creation
- no arbitrary command execution
- no shell-constructed user input
- no automatic migration of all engines
- no pretending disk duplication does not exist
@@ -0,0 +1,248 @@
# Registry compatibility work
Scope implemented locally on 2026-10-09: file containment, restricted data loading,
configuration parsing, installation-time Shared Runtime setup, user preferences,
dedicated-runtime migration, and Registry archive cleanup. Local verification is recorded below.
These verification records do not constitute Registry approval.
## 1. File access
- Restrict workflow and HTTP-supplied file paths to the configured ComfyUI
input/output/temp or explicitly registered model/voice folders, as appropriate
for the operation. Resolve links before checking containment.
- Keep the audio analyzer's existing widget and browser upload/drag-and-drop.
Uploaded audio is already placed in input by ComfyUI's upload endpoint.
- Reject paths outside the permitted folders. Copying an arbitrary server-side
path into input would still read that path and expose its contents.
- Audit the analyzer preview route and node execution together, plus the suite's
other file-backed widgets and routes. Protect generated cache/output names
from raw node IDs and other request values.
- Preserve explicitly configured external model and voice directories; do not
turn their normal use into an unrestricted path exception.
- Restrict model download destinations before network access, writes, or replacement;
reject traversal and links escaping the permitted data roots. Custom-node code
folders are excluded from data roots even though ComfyUI registers them.
## 2. Checkpoint and dictionary loading
- Prefer restricted PyTorch loading for state dictionaries; keep existing model
formats where safe loading supports them. Do not require conversion or copies
of upstream weights as the default solution.
- The local format audit covered 52 PyTorch files and one plain-pickle language
dictionary: 50 checkpoints opened directly with weights_only=True, two Dots
latent-statistics files needed a scoped NumPy type allowlist, and the dictionary
loaded with a data-only unpickler that refuses class/global lookups.
- The audit used the configured test interpreter with Torch 2.10.0+cu130. It did
not establish full engine inference, every remote variant, TorchScript safety,
or compatibility with old whole-model HuBERT files and all training resumes.
- Review RVC inference/separation/training loaders, DramaBox training, Dots
integration, optional Russian stress dictionaries, and remaining unsafe
helpers. No automatic fallback to unrestricted loading for an untrusted file.
- Keep dependency patches scoped to suite integration rather than changing
unrelated installed packages or globally weakening deserialization.
## 3. Configuration expressions
- Replace the Qwen tokenizer's constant eval with its literal value.
- Replace bundled FunASR configuration eval calls with restricted data parsing.
Preserve required list/tuple expressions, including existing list addition and
repetition; literal_eval alone does not cover those expressions.
- Keep neural-network model.eval() calls. Add the required adjacent TTS Audio
Suite patch comments for bundled-code changes.
## 4. Optional isolated environments
The maintainer rejected a manual setup command as the normal user experience.
Required behavior: users get automatic setup or a clear install action inside
ComfyUI, without editing files, running terminal commands, or learning about
Python environments. Keep existing workflow widgets and their order. Generation now checks prepared-runtime metadata and never installs packages.
The installer prepares or reuses Shared Runtime independently of its main-dependency fast path.
Implemented: prepare the single shared legacy environment automatically from the existing install.py hook during normal
Manager installation/update. Default Qwen3-TTS, Qwen ASR, Step Audio EditX, and
Higgs Audio 2 already select this environment. Users keep the existing experience
without terminal setup or additional support packages. Model weights still
download only when the corresponding engine is used.
The local shared environment measured 139.1 MB (132.7 MiB), including 6,429
files. It inherits the main environment's heavy packages rather than installing
another PyTorch copy. This measures the current machine, not download size or
a universal upper bound; missing/incompatible host dependencies can change it.
Keep an advanced opt-out for automatic shared-environment preparation. A local
installer configuration can apply before first installation; a ComfyUI setting
can control subsequent installation/update/repair runs. Skipping preparation
must neither delete an existing environment nor silently switch engines to an
incompatible main environment. A missing required environment should report
that support must be enabled and installation repaired through Manager.
### Implemented advanced controls
- Add a default-on "Install shared runtime automatically" setting under
ComfyUI Settings > TTS Audio Suite > Runtime installation.
- Store the preference in one server-side JSON file read by the installer.
The UI edits that same file and shows its actual location; a hand-edited
preference must not be overwritten with stale browser defaults on page load.
Prefer the suite's existing ComfyUI System User storage, outside package files
replaced during updates. Resolve the configured user directory consistently
in the running suite and the separate installer process.
Default location: ComfyUI/user/__tts_audio_suite/runtime_settings.json.
Document the equivalent location under a custom user directory so pre-install
opt-out does not depend on opening a UI that is not installed yet.
- File schema:
```json
{
"install_shared_runtime": false
}
```
- Only accept the defined boolean. Do not
accept executable paths, package names, URLs, or shell commands from this file,
a workflow, or a management request. Protect settings-management writes.
- Turning shared preparation off affects later installer runs. It does not
remove the installed environment, stop its use by existing workflows, or
select Main Environment instead. The file option supports pre-install opt-out;
the UI becomes available after the suite is installed.
- Do not add automatic deletion on restart. Show whether the environment is
installed, its folder location, and instructions to close ComfyUI before
manually deleting runtimes/shared_legacy_t4. Explain that engines still set
to Shared Runtime then need that environment restored or another prepared,
compatible runtime selected. Model weights and voice files are separate.
### Runtime choices implemented locally
- Remove Dedicated Runtime from Qwen3-TTS and Step Audio EditX. Runtime widgets
stay in their existing positions with Main Environment and Shared Runtime.
- Old Dedicated Runtime widget/API values and old dedicated profile names map
to Shared Runtime. Frontend migration also updates saved workflow widgets.
- Remove the separate Qwen, Step, and VibeVoice legacy profile definitions; keep
the shared profile unchanged so existing installations can be reused.
- Leave existing dedicated environment folders on disk. Do not delete user files
as part of this change. Users can remove unused folders after closing ComfyUI.
- No dedicated installation controls or file settings are needed. Reuse valid
shared runtime metadata on later updates, checking profile changes separately
from the main dependency installer's fast path.
Manager can defer its installation jobs until restart, including on Windows.
That is still Manager-requested package installation. A workflow node recording
flags and our prestartup/import code calling pip on every boot is a different
boundary; timing alone does not make it acceptable. No new installer node is
needed for the recommended default. The implementation still requires Registry review; it does not assert approval under the latest policy.
Reuse existing compatible environments. An update must not silently delete a
working environment during generation. Model downloads remain separate from
installing executable Python dependencies. An HTTP route or workflow node that
invokes pip after a click would still be runtime installation; renaming pip or
hiding it in a helper is not a policy fix.
Evidence checked:
- [Official standards](https://docs.comfy.org/registry/standards) prohibit runtime
package installation through subprocess calls.
- [Manager's install queue](https://github.com/Comfy-Org/ComfyUI-Manager/blob/main/glob/manager_server.py)
accepts registered node IDs/versions and delegates to its package installer.
Its [special-purpose files documentation](https://github.com/Comfy-Org/ComfyUI-Manager#custom-node-support)
documents automatic install.py execution during installation.
- [Manager's deferred installer](https://github.com/Comfy-Org/ComfyUI-Manager/blob/main/glob/manager_core.py#L2070)
schedules installation jobs for restart when appropriate, rather than requiring
each node pack to create an independent startup installer.
- [SAM3 Registry versions](https://api.comfy.org/nodes/comfyui-sam3/versions?include_status_reason=true):
0.1.21 is Active, with a manual SAFE decision by drltdata@comfy.org under
policy-v0.1. Its [published archive](https://cdn.comfy.org/pznodes/comfyui-sam3/0.1.21/node.zip)
contains install.py calling comfy_env.install(), plus isolated-environment
dependency configuration. This is an accepted installation-time precedent,
not a blanket exception for our current first-use bootstrap or newer policies.
- [comfy-env installation documentation](https://docs.comfy-forge.org/comfy-env/install/)
describes installation-time creation rather than environment creation during
node execution. Adoption of that library is not required for this plan.
- Registry review requests [226](https://github.com/Comfy-Org/registry-backend/issues/226),
[261](https://github.com/Comfy-Org/registry-backend/issues/261), and
[210](https://github.com/Comfy-Org/registry-backend/issues/210) are unanswered;
they do not prove approval of optional runtime installers.
### User documentation
README.md now describes automatic installation, the UI/file opt-out, custom user
directories, repair through Manager, and manual cleanup. The details command shows
the actual preference file and runtime folder; no UI route or node runs pip.
## 5. Registry archive contents
- Added .comfyignore to exclude root tests/scripts, GitHub automation, the project
index, and development reports from Registry archives. Repository contents
remain available for development.
- Keep the two metadata YAML files in docs/Dev reports. The auxiliary model
registry reads its YAML at runtime. Keep user guides, example workflows,
frontend assets, and all engine/training code.
- The publisher's current Git filename handling skips quoted emoji paths.
Configure core.quotepath=false in publishing so those guides/examples ship.
## Verification for the implementation
- Compare an archive with an independently collected, NUL-delimited Git source
manifest (git ls-files -z), not the packer's own parsed filename list. Require
every missing file to have an intentional exclusion and verify retained bytes.
This catches silent omissions from quoting, decoding, and path parsing.
- Exercise uploads, allowed typed paths, rejected traversal/link escapes, and
malformed node IDs through both node execution and HTTP preview routes.
- Check restricted loaders against the actual local formats, then exercise the
affected engine inference and training paths using the configured test setup.
- Check existing prepared environments and missing/stale profiles. Confirm that
generation cannot install packages or replace an environment.
- Recheck the published Registry status separately from successful upload;
packaging cleanup by itself does not fix the runtime findings.
## Local checks for dedicated runtime removal
- The existing runtime unit test module passed all 27 cases, including legacy
mode/profile migration and rejection of unsupported values.
- JavaScript migration checks covered 20 hook/value combinations and confirmed
other widget positions/values and unrelated nodes remain unchanged.
- The restarted ComfyUI server exposes only Main Environment and Shared Runtime
for Qwen3-TTS and Step Audio EditX, with the original input order. Its prompt
validation accepts old dedicated values and rejects unknown modes. Requests
deliberately omitted required downstream inputs, so no generation was queued.
- ComfyUI serves the new frontend extension. Full-fix checks are recorded below.
- Source rewrites must match AST call identity. Compare preserved attribute-call
identities against the original source, independently of edit selection; builtin
eval removal must not rename model.eval() or other unrelated methods.
## Completed local verification
- 59 targeted existing unit cases passed, covering runtime/profile migration,
installer repairs, audio processing, voice compatibility, and dependency guidance.
- An empty temporary Shared Runtime was created by the real installer using the
configured test Python. It installed Transformers 4.57.3, inherited host PyTorch,
passed import/readiness/reuse checks, and left the host packages unchanged. The
temporary environment was removed. Windows console logging uses ASCII output.
- Installer hook checks covered enabled preparation and file-based opt-out.
Missing/stale runtime checks cannot run subprocess installers during generation;
simulated failed installation restores the previous runtime.
- Live API checks passed for uploaded audio analysis, permitted paths, blocked
traversal/external paths/node IDs, strict settings values, same-origin writes,
preference persistence, and preservation of installed runtime files.
- The real browser showed the default-on setting and the details menu with the
correct file/folder paths and manual cleanup instructions. Backend file edits
override stale browser preferences. Frontend checks also covered rapid toggles
and restoring the checkbox after a failed save.
- Live Step TTS followed by a giggle edit succeeded through the Shared Runtime,
including migration from the old dedicated API value. The final run produced
non-silent 24 kHz audio lasting 3.12 seconds.
- Restricted Dots latent statistics, the 3,060,182-entry Russian dictionary, NumPy
data/DAC formats, and Demucs architecture metadata passed. A small real Demucs
model restored its state through audio-separator's existing load_model entry
point. Malicious pickle/NumPy/Torch payloads and model-constructor REDUCE calls
were rejected. Dependency patches preserve global torch/importlib references.
- Python source compilation and independent AST comparison preserved every
existing neural model.eval() call. Required config list arithmetic remains usable.
- The candidate archive includes intended uncommitted source without staging it:
1,936 packaged files and 104 intentional exclusions. Runtime files, Unicode
paths, user guides, and both metadata YAML files retain exact bytes. The auxiliary
model registry reads its YAML successfully from the packaged layout.
Limits: Manager is absent from the test installation, so its real queue/restart
flow was not exercised; the install.py hook and fresh runtime installation were
tested directly. This does not establish every engine's inference/training resume,
support arbitrary whole-object checkpoints, or constitute Registry approval.
+12 -5
View File
@@ -2,10 +2,17 @@
This document tracks architectural issues and inconsistencies that need refactoring to improve modularity and reduce code duplication.
## SRT Processing Architecture Issues
### Problem: Inconsistent SRT Implementation Approaches
Different engines use completely different patterns for SRT processing:
## SRT Processing Architecture Issues
### Problem: VibeVoice Native Multi-Speaker SRT Uses Non-Contiguous Global Slots
- **Issue**: Later subtitles can request global `Speaker 2`/`Speaker 3` IDs while VibeVoice truncates and renumbers the supplied voice prompts from zero.
- **Impact**: A subtitle containing only later global speakers can bind the wrong reference or leave a requested speaker without a matching voice prompt.
- **Solution**: Use the global character map only to select references, then compact each subtitle to request-local speakers `0..N-1` with references in the same order.
- **Regression test**: Cover subtitle 1 `[Alice]...[Bob]...`, followed by subtitle 2 `[Bob]...[Rick]...`.
- **Priority**: High
### Problem: Inconsistent SRT Implementation Approaches
Different engines use completely different patterns for SRT processing:
1. **ChatterBox (Old)**: Uses `ChatterboxSRTTTSNode` class (should be processor)
2. **VibeVoice**: Uses proper `VibeVoiceSRTProcessor` class with full implementation
@@ -136,4 +143,4 @@ Each engine reimplements:
---
*Last Updated: 2025-01-XX*
*Add new issues to this file as they are discovered during development*
*Add new issues to this file as they are discovered during development*
@@ -0,0 +1,123 @@
TTS Audio Suite - Transformers 5.x Higgs Audio 2 Investigation Report
Scope
- Problem: Higgs Audio 2 regressed after the suite moved the main environment to Transformers 5.x.
- Goal: determine whether Higgs Audio 2 is a good native Transformers 5 patch target, or whether it should move to runtime isolation.
Environment
- Windows / ComfyUI runtime.
- Main env on Transformers 5.x.
- Direct repro and validation performed in the real Windows Python environment outside the ComfyUI UI wrapper.
Findings Summary
- Higgs Audio 2 is not blocked at the shallow import/API level only.
- Some real compatibility fixes were possible:
- bundled cache API adaptation
- cache-arg rename fixes
- major cold-load regression fix
- However, the remaining failure is decode-path semantics, especially around StaticCache and CUDA graph replay.
- Higgs can be made to load and "run" on Transformers 5, but the high-performance path is not trustworthy and produced incorrect audio.
- This is not a good short-path patch target like IndexTTS.
Reproduction
1. Transformers 5 main environment.
2. Higgs Audio 2 loaded through the normal TTS Text path with voice cloning.
3. Observed one or more of:
- multi-minute cold-load stalls
- StaticCache API crashes
- CUDA graph capture/runtime failures
- runaway/gibberish generation
- near-empty or silent output
Key Diagnostic Evidence
- Initial cold-load timing showed:
- `⏱️ Higgs Audio: Core model load took 440.61s`
- Tokenizer/HuBERT stages were not the bottleneck:
- text tokenizer around 1-2s
- audio tokenizer around 2-5s
- The 440s stall was inside `HiggsAudioModel.from_pretrained(...)`, specifically Hugging Face missing-key initialization.
- Direct checkpoint/model key comparison in runtime showed:
- `missing_count 0`
- `unexpected_count 0`
- After bypassing unnecessary missing-key initialization for complete local checkpoints, direct timing dropped to:
- `⏱️ Higgs Audio: Core model load took 58.46s`
- later direct runs around `33-58s` depending on cache state
- StaticCache/API drift evidence:
- `'StaticCache' object is not subscriptable`
- later mask/cache shape mismatches around static cache width `1024`
- Cache-arg drift evidence:
- bundled Higgs/LLaMA path mixed `past_key_value` and `past_key_values`
- led to duplicated kwarg failures and broken decode semantics
- CUDA graph replay evidence:
- direct runtime probe with graph replay active produced:
- immediate or near-immediate audio termination
- tiny garbage clips
- or invalid decode behavior
- direct runtime probe with StaticCache but without CUDA graph replay produced:
- normal-length audio token generation
- reasonable token spread
- real waveform output
Attempts and Results (chronological)
1) StaticCache compatibility shims
- Added bundled helpers to access cache tensors/device across old tuple caches and new Cache/StaticCache APIs.
- Result: fixed immediate StaticCache indexing crashes.
2) CUDA graph failure fallback cleanup
- On graph capture failure, forced cache recreation and switched to DynamicCache instead of continuing with stale StaticCache assumptions.
- Result: removed some follow-on crashes after graph failure, but not the core correctness issue.
3) Attention backend adjustment
- Stopped forcing eager attention on CUDA and preferred SDPA instead.
- Result: better alignment with the intended high-performance path, but did not fully restore correctness.
4) TTS defaults forcing audio generation
- Set Higgs TTS defaults to force audio generation rather than letting the model drift in text mode.
- Result: prevented obvious free-running text-style generation, but did not solve bad output.
5) Cold-load regression fix
- Added a local complete-checkpoint detector and skipped Hugging Face missing-key initialization when the local Higgs checkpoint was complete.
- Result: large real improvement to cold-load time.
6) Cache-argument adaptation
- Patched bundled Higgs/LLaMA call sites for Transformers 5 cache-arg naming.
- Result: moved runtime further, but surfaced deeper semantic issues instead of fixing them outright.
7) CUDA graph runner PyTorch compatibility
- Wrapped input-buffer updates in inference mode to satisfy current PyTorch restrictions.
- Result: removed one explicit runtime crash, but not the underlying decode corruption.
8) StaticCache without CUDA graph replay direct comparison
- Direct Windows probes showed:
- `enable_cuda_graphs=True` with graph replay: corrupted/too-short audio output
- `enable_cuda_graphs=False`: long audio-token generation and real waveform output
- `StaticCache` with graph replay skipped: closer to sane behavior than replay path
- Result: isolated the remaining failure to the CUDA graph replay path on Transformers 5.
Interpretation
- Higgs Audio 2 on Transformers 5 is not blocked by a single missing symbol or a few renamed APIs.
- The remaining problem is decode/runtime behavior, especially the interaction between:
- StaticCache
- bundled Higgs decode logic
- LLaMA attention/cache semantics on current Transformers 5
- CUDA graph replay
- That is behavior-level drift, not just compatibility-shim work.
Decision
- Higgs Audio 2 on Transformers 5 is still theoretically patchable.
- It is not a good use of time right now.
- The engine reached the same bad zone as Qwen3-TTS, but with even more sensitivity around CUDA graphs and cache behavior.
- Runtime isolation is the correct path for now.
Operational Recommendation
- Do not continue native Transformers 5 Higgs patching in the main branch.
- Route Higgs Audio 2 into an isolated legacy runtime instead.
- Prefer a dedicated Higgs runtime first rather than assuming shared legacy parity.
- Revisit native Transformers 5 support only if Higgs becomes important enough to justify deeper decode/cuda-graph compatibility work.
Notes
- The cold-load fix was a real improvement and may still be worth revisiting later, even outside the final T5 decision.
- The rest of this investigation should be treated as a branch-local investigation record, not as a stable main-branch solution.
@@ -1,4 +1,4 @@
TTS Audio Suite - Transformers 5.0.0 Qwen3-TTS Investigation Report
TTS Audio Suite - Transformers 5.x Qwen3-TTS Investigation Report
Scope
- Problem: Qwen3-TTS garbled audio or load failure on Transformers 5.0.0.
@@ -54,3 +54,65 @@ Recommendation
Notes
- No changes in this report alter project behavior.
Addendum - Transformers 5.10.2 Re-check
Scope
- Problem: determine whether newer Transformers 5.10.2 improves Qwen3-TTS enough to avoid runtime isolation.
- Goal: distinguish shallow API drift from deeper generation/runtime incompatibility.
Environment
- Windows / ComfyUI runtime.
- Main env on Transformers 5.10.2.
- Direct repro also performed in the real Windows Python environment outside the ComfyUI UI wrapper.
What Changed vs 5.0.0
- The failure mode is different.
- The first blocker is no longer the tokenizer collapse described above.
- Qwen3-TTS can be pushed significantly further on 5.10.2 with local compatibility shims.
Key Diagnostic Evidence
- Direct model load initially failed with:
- `KeyError: 'default'`
- source: `ROPE_INIT_FUNCTIONS[self.rope_type]`
- Current Transformers 5.10.2 no longer exposes a legacy `"default"` RoPE entry:
- available keys observed in runtime: `dynamic`, `linear`, `llama3`, `longrope`, `proportional`, `yarn`
- After restoring a local `"default"` compatibility entry, model load succeeded.
- Generation then failed on changed Hugging Face masking APIs:
- `create_causal_mask() got an unexpected keyword argument 'input_embeds'`
- After adapting those call sites to the current HF signature, generation progressed further.
- Final direct generation failure on 5.10.2 was:
- `RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [16, 277] but got: [16, 139].`
- source: SDPA attention path during generation.
Attempts and Results (5.10.2)
1) Legacy RoPE compatibility restore
- Added a local fallback for legacy `"default"` RoPE initialization.
- Result: Qwen3-TTS load progressed and completed successfully.
2) Mask helper API adaptation
- Updated bundled Qwen mask call sites to match current HF `create_causal_mask` / `create_sliding_window_causal_mask` signatures.
- Result: generation progressed beyond the earlier immediate crash.
3) Direct generation validation with real voice-clone inputs
- Used real reference audio and transcript in the Windows environment.
- Result: no longer a trivial import/load/signature issue; generation hit a tensor-shape mismatch inside SDPA.
Interpretation
- Transformers 5.10.2 is not blocked in the same way as 5.0.0.
- Some problems are still shallow compatibility issues and can be adapted cleanly.
- However, once generation reaches SDPA tensor-shape mismatch territory, this stops being a simple API-shim problem.
- At that point, continuing the patch path means re-validating model behavior against newer Hugging Face generation/mask/cache semantics, not just restoring removed symbols.
Decision
- Qwen3-TTS on Transformers 5.10.2 is still technically patchable in principle.
- It is not a good short-path target like IndexTTS.
- The suite should prefer runtime isolation for Qwen3-TTS instead of continuing deep Transformers 5 surgery.
Operational Recommendation
- Treat Qwen3-TTS as a legacy-stack engine for now.
- Route it into the shared legacy Transformers 4 runtime used by compatible engines (the same runtime family already proven for Qwen in prior testing).
- Keep the main environment on Transformers 5.
- Revisit native Transformers 5 support only if Qwen3-TTS becomes strategically important enough to justify deeper behavior-level compatibility work.
Notes
- The transient 5.10.2 patch attempts were not kept as the main solution path.
- The current project direction is runtime isolation, not continued in-place Transformers 5 patching for Qwen3-TTS.
@@ -55,7 +55,6 @@ categories:
languages:
- "47 languages"
notes: "Heavier multilingual fallback with the widest language coverage supported by the punctuators package."
model_layouts_markdown: |
# Auxiliary Model Layouts
File diff suppressed because it is too large Load Diff
+23 -14
View File
@@ -2,17 +2,26 @@
## Engine Comparison
| Engine | Models | Size | TTS | SRT | VC | ASR | License | Special Features | Languages |
| ------------------ | ----------------------------------------- | ------------ | --- | --- | --- | --- | ------------------------ | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| **F5-TTS** | Base, v1, E2TTS + 8 lang models | ~1.2GB each | ✅ | ✅ | ❌ | ❌ | CC-BY-NC-4.0 | Targeted Word/Speech Editing, Speed control | 🇺🇸​🇩🇪​🇪🇸​🇫🇷​🇮🇹​🇯🇵​🇧🇷​🇵🇱​🇮🇳​🇹🇭 |
| **ChatterBox** | EN, DE×3, IT, FR, RU, HY, KA, JA, KO, NO | ~4.3GB | ✅ | ✅ | ✅ | ❌ | MIT | Expressiveness slider | 🇺🇸​🇩🇪​🇫🇷​🇮🇹​🇯🇵​🇰🇷​🇷🇺​🇳🇴​🇦🇲​🇬🇪 |
| **ChatterBox 23L** | v1, v2, Vietnamese (Viterbox), Egyptian Arabic (oddadmix) | ~4.3GB | ✅ | ✅ | ✅ | ❌ | MIT | 24 languages in single model, emotion tokens (v2 - doesn't work) | 🇺🇸​🇨🇳​🇩🇪​🇪🇸​🇫🇷​🇮🇹​🇯🇵​🇰🇷​🇷🇺​🇵🇹​🇵🇱​🇮🇳​🇪🇬​🇹🇷​🇹🇭​🇳🇴​🇻🇳​🇩🇰​🇫🇮​🇬🇷​🇮🇱​🇲🇾​🇳🇱​🇸🇪​🇰🇪(+9) |
| **VibeVoice** | 1.5B, 7B, KugelAudio-0 (7B), Hindi-1.5B/7B | 5.4GB / 18GB | ✅ | ✅ | ❌ | ❌ | MIT (research-only per model card) | 90-min long-form, Native 4-speaker (Base models), Multilingual (KugelAudio), 4-bit quantization | 🇺🇸​🇨🇳​🇩🇪​🇪🇸​🇫🇷​🇮🇹​🇯🇵​🇰🇷​🇷🇺​🇧🇷​🇵🇱​🇮🇳​🇪🇬​🇹🇷​🇹🇭​🇳🇴​🇻🇳​🇦🇲​🇬🇪​🇩🇰​🇫🇮​🇬🇷​🇮🇱​🇲🇾​🇳🇱​🇸🇪​🇰🇪 |
| **Higgs Audio 2** | 3B | ~9GB | ✅ | ✅ | ❌ | ❌ | Boson Higgs Audio 2 Community License | 3 multi-speaker, CUDA graphs (55+ tokens/sec) | 🇺🇸​🇨🇳​🇩🇪​🇪🇸​🇰🇷 |
| **IndexTTS-2** | IndexTTS-2 | ~4.7GB | ✅ | ✅ | ❌ | ❌ | bilibili Model Use License | Emotion Control: 8 vectors, Text as reference, Audio as reference | 🇺🇸​🇨🇳​🇯🇵 |
| **CosyVoice3** | 0.5B, 0.5B-RL | ~5.4GB | ✅ | ✅ | ✅ | ❌ | Apache-2.0 | Paralinguistic tags | 🇺🇸​🇨🇳​🇯🇵​🇰🇷 |
| **Qwen3-TTS** | 0.6B, 1.7B (CustomVoice/VoiceDesign/Base) | ~3-6GB | ✅ | ✅ | ❌ | ✅ | Apache-2.0 | Voice design, ASR (Automatic Speech Recognition) | 🇺🇸​🇨🇳​🇩🇪​🇪🇸​🇫🇷​🇮🇹​🇯🇵​🇰🇷​🇷🇺​🇵🇹 |
| **Granite ASR** | granite-4.0-1b-speech | ~4.6GB | ❌ | ✅ | ❌ | ✅ | Apache-2.0 | ASR (Automatic Speech Recognition), Custom timestamps/SRT via reused Qwen forced aligner, Speech translation (experimental) | 🇺🇸​🇩🇪​🇪🇸​🇫🇷​🇯🇵​🇵🇹 |
| **Step Audio EditX** | 3B LLM + CosyVoice | ~7GB | ✅ | ✅ | ❌ | ❌ | Apache-2.0 (verify before commercial use) | Second Pass Speech Editing Node: 14 emotions, 32 speaking styles, Paralinguistic effects | 🇺🇸​🇨🇳​🇯🇵​🇰🇷 |
| **Echo-TTS** | echo-tts-base + fish-s1-dac-min | ~5.3GB + ~1.8GB | ✅ | ✅ | ❌ | ❌ | CC-BY-NC-SA-4.0 | Diffusion-based (~30s best), Force Speaker KV (speaker drift control) | 🇺🇸 |
| **RVC** | Community .pth | 100-300MB | ❌ | ❌ | ✅ | ❌ | MIT (framework); community models vary | Real-time VC, Pitch shift (±14), 6 HuBERT models, Language-independent | 🇺🇸​🇨🇳​🇩🇪​🇪🇸​🇫🇷​🇮🇹​🇯🇵​🇰🇷​🇷🇺​🇧🇷​🇵🇱​🇮🇳​🇪🇬​🇹🇷​🇹🇭​🇳🇴​🇻🇳​🇦🇲​🇬🇪​🇩🇰​🇫🇮​🇬🇷​🇮🇱​🇲🇾​🇳🇱​🇸🇪​🇰🇪 |
| Engine | Isolation | Models | Size | TTS | SRT | VC | ASR | Sound Effects | Training | License | Special Features | Languages |
| ------------------ | --------- | ----------------------------------------- | ------------ | :-: | :-: | :-: | :-: | :-----------: | :------: | ------------------------ | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| **F5-TTS** | Main | Base, v1, E2TTS + 8 lang models | ~1.2GB each | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | CC-BY-NC-4.0 | Targeted Word/Speech Editing, Speed control | 10 |
| **ChatterBox** | Main | EN, DE×3, IT, FR, RU, HY, KA, JA, KO, NO | ~4.3GB | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | MIT | Expressiveness slider | 10 |
| **ChatterBox 23L** | Main | v1, v2, v3, Vietnamese (Viterbox), Egyptian Arabic (oddadmix) | ~4.3GB | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | MIT | V1, V2, and V3 official checkpoints, Emotion tokens (v2; currently ineffective), V3 skips the legacy alignment analyzer and trims the final token artifact | 25 |
| **VibeVoice** | Main | 1.5B, 7B, KugelAudio-0 (7B), kugel-2 (7B), Hindi-1.5B/7B | 5.4GB / 18GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | MIT (research-only per model card) | 90-min long-form, Native 4-speaker (Base models), Multilingual (KugelAudio variants), 4-bit quantization | 27 |
| **Higgs Audio 2** | Shared | 3B | ~9GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Boson Higgs Audio 2 Community License | 3 multi-speaker, CUDA graphs (55+ tokens/sec) | 5 |
| **Higgs Audio v3** | Main | 4B | ~8GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Boson Higgs Audio v3 Research and Non-Commercial License | Native inline emotion/style/prosody/SFX tags, Zero-shot voice cloning, 100+ language support | 100+ |
| **IndexTTS 2 / 2.5** | Main | IndexTTS-2, IndexTTS-2.5 | ~4.7GB / ~5.49GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | bilibili Model Use License | Emotion Control: 8 vectors, Text as reference, Audio as reference, IndexTTS-2.5 official internal feature-duration scaling (not prosody planning), IndexTTS-2.5 pronunciation annotations | 5 |
| **CosyVoice3** | Main | 0.5B, 0.5B-RL | ~5.4GB | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | Apache-2.0 | Paralinguistic tags | 4 |
| **Qwen3-TTS** | Shared | 0.6B, 1.7B (CustomVoice/VoiceDesign/Base) | ~3-6GB | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | Apache-2.0 | Voice design, ASR (Automatic Speech Recognition) | 10 |
| **Granite ASR** | Main | granite-4.0-1b-speech, granite-speech-4.1-2b, granite-speech-4.1-2b-plus | ~4.6GB | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | Apache-2.0 | Native speaker attribution / diarization (plus model variant), Native word-level timestamps (plus model variant), ASR (Automatic Speech Recognition), Custom timestamps/SRT via reused Qwen forced aligner, Speech translation (experimental), Optional forced aligner auto-routed through shared legacy T4 runtime | 6 |
| **Step Audio EditX** | Shared | 3B LLM + CosyVoice | ~7GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 (verify before commercial use) | Second Pass Speech Editing Node: 14 emotions, 32 speaking styles, Dated checkpoints: 2026-01-23 / 2025-11-28, Expanded January paralinguistic effects with legacy tag compatibility, Selectable main or shared Python runtime (shared Transformers 4 runtime recommended) | 4 |
| **Echo-TTS** | Main | echo-tts-base + fish-s1-dac-min | ~5.3GB + ~1.8GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | CC-BY-NC-SA-4.0 | Diffusion-based (~30s best), Force Speaker KV (speaker drift control) | 1 |
| **Fish Audio S2 Pro** | Main | S2 Pro 4B / FP8 | ~10.3GB / ~8.0GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Fish Audio Research License | Free-form sub-word emotion/prosody tags, Native multi-speaker and multi-turn dialogue with dynamic speaker references, Zero-shot voice cloning, Optional per-segment custom character switching, Configurable 4K-32K native context with reduced KV-cache VRAM, Optional community FP8 weight-only checkpoint with BF16 activations, Optional on-the-fly BitsAndBytes INT8/NF4 for the official checkpoint | 80+ languages |
| **Dots TTS** | Main | dots.tts-base, dots.tts-soar, dots.tts-mf | ~6GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 | Official auto language detect / language control, SOAR and MeanFlow distilled variants | 19 |
| **DramaBox** | Main | DramaBox 3.3B | ~16.4GB | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | LTX-2 Community License | Expressive scene prompting and stage directions, Native and SRT-aware duration targeting, Official duration-aware long-form chunking with scene-prefix preservation, Optional 10-second zero-shot voice reference, CFG negative prompt with per-segment switching, Explicit generation/reference durations, CFG rescale control, and optional Perth watermark, Experimental staged and sequential strategies for lowering peak VRAM; official LTX FP8-cast storage, Optional official torch.compile path, Official audio-branch IC-LoRA training workflow | 1 |
| **OmniVoice** | Main | OmniVoice | ~3.7GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 | Inline non-verbal tags and pronunciation overrides, Reference-free voice design, 600+ language support, Upstream long-form chunk orchestration | 600+ |
| **MOSS-TTS** | Main | Local 1.7B, Delay 8B v1.5/1.0, LAION Voice Acting 8B community fine-tune, VoiceGenerator 1.7B, SoundEffect 8B v1, TTSD 8B | ~8.5GB tokenizer + ~6.1GB/17GB/18GB model | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | Apache-2.0 | Reference-free voice design with MOSS-VoiceGenerator, Native 1-5 speaker TTSD dialogue, 31-language generation with MOSS-TTS-v1.5, Optional LAION community 8B voice-acting fine-tune, Config-based discovery of compatible local MOSS full checkpoints, Prompt-only sound-effect generation with MOSS-SoundEffect v1, Long-form generation (TTSD/Delay), Duration token hint, Local/Delay/TTSD variants, Initial integrated LoRA training workflow (Delay 8B) | 24 |
| **MOSS-SoundEffect v2** | Main | MOSS-SoundEffect-v2.0 | ~11.2GB | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | Apache-2.0 | Durations up to 30 seconds, Native negative prompting, CFG, flow shift, and diffusion-step controls, Prompt-only text-to-sound generation, 48 kHz mono output, Seeded generation | 2 |
| **RVC** | Main | Community .pth | 100-300MB | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | MIT (framework); community models vary | Real-time VC, Integrated training workflow, Pitch shift (±14), 6 HuBERT models, Language-independent | Any |
*Isolation column: `Main` runs in the main ComfyUI environment. `Shared` uses a shared secondary runtime reused by multiple engines.*
+28
View File
@@ -0,0 +1,28 @@
# F5-TTS Reference and Inference Guidelines
These recommendations apply to F5-TTS generation, whether you use a narrator reference or saved character voices. For switching between voices, see the [Character Switching Guide](CHARACTER_SWITCHING_GUIDE.md).
## Prepare the reference
- Use clean speech from one speaker, with the delivery you want to reproduce.
- Keep the reference **shorter than 12 seconds** and leave a little silence at the end, roughly one second. It does not need to be exactly 10 or 12 seconds, or have one second of silence at both ends.
- Supply an exact transcript of the speech in that clip. When trimming audio, update the transcript too.
- Save library references as `name.wav` with `name.reference.txt`. A matching `name.txt` can serve as the transcript when no reference transcript is found. For connected audio, use **Character Voices** to provide both audio and text through `opt_narrator`.
F5-TTS works at **24,000 Hz** and resamples reference audio automatically. Manual conversion to 22,050 Hz is unnecessary. If converting a file yourself, resample it normally; changing only its declared sample rate changes playback speed and pitch.
## If generated speech is too fast
F5 estimates generation duration from the reference audio's duration and transcript length, adjusted by the engine's `speed` setting. Reference delivery and transcript accuracy both affect pacing.
**Long references can create a hidden mismatch:** F5's preprocessing automatically shortens audio to around 12 seconds, but retains a supplied transcript. Text that matched the original recording may no longer match the processed clip, causing rushed or unstable generation. Prepare a short reference and its matching transcript before passing them to F5.
Start with `speed = 1.0`. After checking the reference, lower it for slower speech or raise it for faster speech. Large changes can affect generation quality. Include the reference audio, transcript, and generated output when reporting a pacing problem.
## Text preparation
- Use punctuation and spaces to suggest pauses. Put a space after English sentence-ending punctuation so F5 can recognize sentence boundaries for chunking.
- Use ordinary lowercase words for normal reading; uppercase forms such as `K.F.C.` are intended for letter-by-letter pronunciation.
- For Chinese speech, write numbers in Chinese words when needed to avoid an unintended English reading.
These points follow the [official F5-TTS inference guidance](https://github.com/SWivid/F5-TTS/blob/main/src/f5_tts/infer/README.md). Reference clipping and duration estimation are implemented in the [upstream inference code](https://github.com/SWivid/F5-TTS/blob/main/src/f5_tts/infer/utils_infer.py).
+19 -11
View File
@@ -2,14 +2,22 @@
## Feature Comparison Matrix
| Feature | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | RVC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| **Voice Cloning** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (Base model) | ❌ | ✅ | ✅ | ⚠️ (needs training) |
| **Native Multi-Speaker** | ❌ | ❌ | ❌ | ✅ (Base only, Kugel uses fallback) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **Voice Conversion** | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |
| **ASR (Transcribe)** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| **Emotion Control** | ❌ | ❌ | ⚠️ (v2 tags - doesn't work) | ❌ | ⚠️ (via prompt) | ✅ (8 emotions) | ⚠️ (via instruct) | ⚠️ (via instruct) | ❌ | ✅ (14 emotions) | ❌ | ❌ |
| **Native Long-form (90min)** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | N/A |
| **Community Finetunes** | ✅ | ✅ | ✅ | ✅ KugelAudio, Hindi | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| **VRAM Efficient** | ✅ | ✅ | ✅ | ⚠️ (5-18GB) | ⚠️ (9GB) | ⚠️ (9-12GB) | ✅ (5.4GB) | ✅ (3-6GB) | ✅ (~4.6GB) | ⚠️ (7GB) | ⚠️ (~7GB total) | ✅ |
| **Speed/Performance** | ✅ Very Fast | ✅ Fast | ✅ Fast | ⚠️ | ⚠️ | ⚠️ | ✅ Fast | ⚠️ | ⚠️ Moderate | ⚠️ | ✅ Fast (diffusion, realtime-capable) | ✅ Fast |
| Feature | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS 2 / 2.5 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | DramaBox | OmniVoice | MOSS-TTS | MOSS-SoundEffect v2 | RVC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| **TTS** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| **SRT** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| **Voice Conversion** | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| **ASR (Transcribe)** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **Sound Effects** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ |
| **Training** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ✅ |
| **Voice Cloning** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (Base model) | ❌ | ✅ | ✅ | ✅ Reference audio plus exact transcript | ✅ | ✅ Optional reference audio; configurable 3-30 second window from the beginning of the source, default 10 seconds | ✅ | ✅ | ❌ | ⚠️ (needs training) |
| **Reference Transcript†** | **Required** | Not used | Not used | Not used | Optional | Optional | Not used | Conditional | Conditional | N/A | **Required** | Not used | **Required** | Optional | Not used | **Required** | Conditional | Not used | N/A |
| **Native Multi-Speaker** | ❌ | ❌ | ❌ | ✅ (Base only, Kugel uses fallback) | ✅ | ❌ | ❌ | ❌ | ❌ | ⚠️ (Plus variant speaker attribution / diarization) | ❌ | ❌ | ✅ Native dialogue is default; optional custom mode generates each [Character] segment independently as local speaker 0 | ❌ | ❌ | ❌ | ✅ (TTSD v1.0; 1-5 speakers) | ❌ | ❌ |
| **Emotion Control** | ❌ | ❌ | ⚠️ (v2 tags - doesn't work) | ❌ | ⚠️ (via prompt) | ✅ (native inline tags) | ✅ (8 emotions) | ⚠️ (via instruct) | ⚠️ (via instruct) | ❌ | ✅ (14 emotions) | ❌ | ✅ Free-form inline natural-language tags | ❌ | ✅ Natural-language scene prompt and stage directions | ⚠️ (voice-design instruct + inline non-verbal tags) | ✅ (MOSS-VoiceGenerator instruction-conditioned voice design) | ❌ | ❌ |
| **Native Long-form** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Configurable 4K-32K native context; suite text chunking is bypassed | ❌ | ✅ Official duration-aware quote-group chunking; ~37s target / 45s cap | ✅ (uses upstream audio_chunk_duration / audio_chunk_threshold orchestration; bypasses suite char-based chunk splitting) | ✅ (TTSD/Delay long-form; use chunk orchestration for very long inputs) | ❌ | N/A |
| **Community Finetunes** | ✅ | ✅ | ✅ | ✅ KugelAudio, Hindi | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Official audio-branch IC-LoRA adapters can be trained and loaded | ❌ | ✅ (Compatible full local checkpoints, LAION Voice Acting 8B auto-download, and LoRA adapter inference/training supported) | ❌ | ✅ |
| **VRAM Efficient** | ✅ | ✅ | ✅ | ⚠️ (5-18GB) | ⚠️ (9GB) | ⚠️ (~8-10GB) | ⚠️ (9-12GB) | ✅ (5.4GB) | ✅ (3-6GB) | ✅ (~4.6GB) | ⚠️ (7GB) | ⚠️ (~7GB total) | ⚠️ 8K context measured at ~15.2GB BF16, ~11.2GB FP8, ~11.2GB BNB INT8, or ~8.9GB BNB NF4; BF16 codec and activations; BNB is a load-time option for the official checkpoint | ⚠️ (main env works; 2B-class model is not lightweight) | ✅ Fast mode is ~24GB; experimental FP8, staged, and sequential options can reduce VRAM, but practical peak usage varies by environment and loaded components | ⚠️ (main env works; 2B-class model is not lightweight) | ⚠️ (Local 1.7B smaller; tokenizer is large) | ⚠️ (runs in the main ComfyUI environment but remains GPU-heavy) | ✅ |
| **Speed/Performance** | ✅ Very Fast | ✅ Fast | ✅ Fast | ⚠️ | ⚠️ | ⚠️ CUDA recommended | ⚠️ | ✅ Fast | ⚠️ | ⚠️ Moderate | ⚠️ | ✅ Fast (diffusion, realtime-capable) | ✅ Main-environment subprocess with reliable teardown; local compile measurements: ~40 it/s BF16, ~11.8 it/s NF4 at ~8.9GB VRAM, and ~3.7 it/s INT8 at ~11.2GB VRAM; quality comparison pending | ⚠️ Moderate; mf variant is faster | ⚠️ Fast favors repeated generation; staged reloads temporary components; sequential also transfers the transformer; optional block-level torch.compile accelerates repeat denoising | ✅ Fast; upstream reports sub-realtime RTF | ✅ Fast with CUDA/FlashAttention | ⚠️ (100 diffusion steps by default) | ✅ Fast |
| **No Narrator Required** | ❌ | ✅ (default speaker) | ✅ (default speaker) | ✅ (zero-shot / default speaker) | ✅ (basic TTS if no narrator/reference is provided) | ✅ (zero-shot) | ❌ | ✅ (cross-lingual or instruct mode) | ✅ (Base default voice or CustomVoice presets) | N/A | ❌ | ❌ | ✅ Reference audio is optional | ✅ (default speaker) | ✅ Voice reference is optional | ✅ (native default voice; instruct also works without narrator) | ✅ (direct TTS and prompt-only generation) | ❌ | N/A |
† **Reference Transcript:** Conditional means the transcript is required only for the specific mode: CosyVoice3 zero-shot, Qwen3-TTS full Base cloning, or MOSS-TTSD cloned-speaker dialogue. Higgs Audio 2, Higgs Audio v3, and Dots TTS accept matching text when provided but do not require it.
+35
View File
@@ -0,0 +1,35 @@
# Fish Audio S2 Pro Inline Tags
Use the suite's public angle-bracket syntax for Fish's free-form instructions:
```text
<whisper in small voice>Hello. <professional broadcast tone>Good evening.
```
The Fish adapter converts these to native `[...]` instructions only when text reaches the Fish engine. Do not write Fish tags with square brackets in suite text because `[Character]` is reserved for character switching.
Fish also supports normal suite tags such as `[pause:500ms]`, character switching, and per-segment parameters. Those are processed by the suite before Fish inference.
## Language Prompting
Fish has no native language dropdown or parameter. The engine can instead prepend a natural inline instruction from the suite's resolved segment language:
- `language_prompting = Auto Inline Tag`: non-English resolved languages become tags such as `<German>` or `<French>`
- `language_prompting = Off`: no automatic language instruction is added
English is only added when the user explicitly requested it with a language tag such as `[en:Bob]` or `[English:Bob]`. Implicit/default English stays untagged.
## Character Switching Modes
The engine defaults to `Native Multi-Speaker`. All `[Character]` turns in a generated block are sent through one Fish dialogue request, preserving native multi-turn context and long-form behavior.
Select `Custom Character Switching` to generate every parsed character segment independently. Each call uses only that character's reference and is remapped to Fish speaker 0. This can reduce speaker leakage, but it requires more calls and does not preserve context between character turns. SRT subtitle boundaries and timing are preserved in both modes.
The engine UI separates checkpoint choice from load-time quantization:
- `model_variant`: `s2-pro` or the separate `s2-pro-fp8` checkpoint
- `quantization`: `none`, `bnb_int8`, or `bnb_nf4` for on-the-fly quantization of the official `s2-pro` checkpoint
The BNB options reuse the official files and do not download a second model copy.
The S2 Pro weights use the Fish Audio Research License: research and non-commercial use are allowed; commercial use requires a separate Fish Audio license.
+78
View File
@@ -0,0 +1,78 @@
# Higgs Audio v3 Inline Tags
Higgs Audio v3 uses native inline control tokens in the text prompt. Type these tags directly in **TTS Text** or **TTS SRT** when using the Higgs Audio v3 engine.
## Syntax
Canonical native tags use:
```text
<|category:value|>
```
TTS Audio Suite also accepts a convenience alias form without the bars:
```text
<category:value>
```
Aliases are normalized internally to the canonical Higgs syntax. The editor inserts the canonical `<|...|>` form on purpose.
Examples:
```text
<|emotion:amusement|>Wait, that was actually funny.
<|style:whispering|>Keep your voice down.
I need a second.<|prosody:pause|> Okay, continue.
That was perfect. <|sfx:laughter|>Haha, absolutely perfect.
```
## Supported Categories
Emotion:
```text
elation, amusement, enthusiasm, determination, pride, contentment, affection,
relief, contemplation, confusion, surprise, awe, longing, arousal, anger, fear,
disgust, bitterness, sadness, shame, helplessness
```
Style:
```text
singing, shouting, whispering
```
Prosody:
```text
speed_very_slow, speed_slow, speed_fast, speed_very_fast,
pitch_low, pitch_high, pause, long_pause,
expressive_high, expressive_low
```
Sound effects:
```text
cough, laughter, crying, screaming, burping, humming, sigh, sniff, sneeze
```
## Placement Rules
- Put delivery-wide tags like emotion, style, speed, pitch, and expressiveness at the start of the sentence or turn.
- Put positional tags like `<|prosody:pause|>`, `<|prosody:long_pause|>`, and `<|sfx:...|>` exactly where they should happen.
- Pair SFX tags with written sound text, for example `<|sfx:laughter|>Haha` or `<|sfx:sigh|>Ahh`.
- Canonical Higgs syntax is `<|...|>`. Alias input like `<emotion:amusement>` is accepted by TTS Audio Suite for convenience, but the official form stays `<|emotion:amusement|>`.
- Higgs Audio v3 does not use Step Audio EditX second-pass inline processing here. Use Higgs native tags instead of Step semantics when targeting Higgs.
## Character Voices
Higgs Audio v3 supports reference audio cloning. A transcript of the reference audio strongly improves cloning quality. Character voice `.txt` files are passed as reference transcripts when available.
## Multiline Editor
The multiline TTS tag editor has an `Inline Tags` panel with a dedicated `Higgs Audio v3` mode. That panel inserts canonical native Higgs tags directly.
## License
Higgs Audio v3 is released under the Boson Higgs Audio v3 Research and Non-Commercial License. Production, hosted API, or revenue-generating use requires a separate commercial license from Boson AI.
+38 -4
View File
@@ -1,4 +1,17 @@
# Inline Edit Tags - User Guide
# Step Audio EditX Inline Tags - User Guide
## Scope
This guide is specifically for **Step Audio EditX** inline post-processing tags.
The multiline TTS tag editor now has an engine-aware `Inline Tags` panel with separate modes for:
- `Step Audio EditX`
- `Higgs Audio v3`
- `CosyVoice3`
Use the Step mode when you want post-process editing tags. Use the Higgs or Cosy modes when you want those engines' native generation tags.
The existing **model_path** dropdown now offers `Step-Audio-EditX-2026-01-23` and `Step-Audio-EditX-2025-11-28`. The January checkpoint is the default for new engine nodes and supports the expanded sound vocabulary below. Existing unversioned and local selections keep their files. See [checkpoint selection and compatibility](STEP_AUDIO_EDITX_CHECKPOINTS.md).
## 💡 Inline vs Manual Node Workflow
@@ -15,7 +28,7 @@
## Overview
Inline edit tags allow you to apply **Step Audio EditX** post-processing effects to TTS-generated audio from ANY engine (F5-TTS, ChatterBox, Higgs Audio, VibeVoice, etc.). These tags are automatically stripped before TTS generation and applied as a second processing pass.
Inline edit tags allow you to apply **Step Audio EditX** post-processing effects to TTS-generated audio from engines that support the Step inline workflow. These tags are automatically stripped before TTS generation and applied as a second processing pass.
**Example:**
```
@@ -48,6 +61,19 @@ Insert non-verbal sounds at specific positions in the audio:
| `<Question-ei>` | Question sound | `<Question-ei> Really?` |
| `<Dissatisfaction-hnn>` | Dissatisfied "hnn" | `<Dissatisfaction-hnn> Not good` |
The **2026-01-23** checkpoint additionally supports:
| Tag | Effect |
|---|---|
| `<inhale>`, `<exhale>`, `<breath>` | Breathing sounds |
| `<laugh>`, `<chuckle>`, `<giggle>` | Laughter variants |
| `<clears throat>` or `<clears_throat>` | Throat clearing |
| `<snort>`, `<cough>` | Snorting and coughing |
| `<Surprise-yo>` | Additional surprise vocalization |
| `<Question-ah>`, `<Question-en>`, `<Question-yi>`, `<Question-oh>` | Additional questioning vocalizations |
These support the same iteration and pipe syntax: `<giggle:2>` and `<clears_throat:1|style:serious>`. Tags are case-insensitive. For ChatterBox v2/v3, specify an iteration for Step effects (`<giggle:1>`); bare overlapping tags retain their native meaning.
**Position matters:** The sound is inserted where you place the tag.
**💡 Pro Tip - Stronger Effects:** For more reliable and pronounced effects, include relevant text alongside the tag:
@@ -67,7 +93,7 @@ Apply emotional tone to the entire audio segment:
`<emotion:VALUE>` or `<emotion:VALUE:ITERATIONS>`
**Available emotions:** happy, sad, angry, excited, calm, fearful, surprised, disgusted, confusion, empathy, embarrass, depressed, coldness, admiration
**Available emotions:** happy, sad, angry, excited, fear, humour, surprised, disgusted, confusion, empathy, embarrass, depressed, coldness, admiration. Older suite options `calm` and `fearful` remain accepted.
**Examples:**
```
@@ -612,9 +638,13 @@ Choose appropriate effects for the situation:
### All Paralinguistic Tags
`<Breathing>`, `<Laughter>`, `<Sigh>`, `<Uhm>`, `<Surprise-oh>`, `<Surprise-ah>`, `<Surprise-wa>`, `<Confirmation-en>`, `<Question-ei>`, `<Dissatisfaction-hnn>`
January checkpoint vocabulary: `<inhale>`, `<exhale>`, `<laugh>`, `<chuckle>`, `<clears throat>` (alias `<clears_throat>`), `<snort>`, `<giggle>`, `<cough>`, `<breath>`, `<Surprise-yo>`, `<Question-ah>`, `<Question-en>`, `<Question-yi>`, `<Question-oh>`.
### All Emotions
`<emotion:happy>`, `<emotion:sad>`, `<emotion:angry>`, `<emotion:excited>`, `<emotion:calm>`, `<emotion:fearful>`, `<emotion:surprised>`, `<emotion:disgusted>`, `<emotion:confusion>`, `<emotion:empathy>`, `<emotion:embarrass>`, `<emotion:depressed>`, `<emotion:coldness>`, `<emotion:admiration>`
Also available: `<emotion:fear>` and `<emotion:humour>`.
### All Styles
`<style:whisper>`, `<style:serious>`, `<style:child>`, `<style:older>`, `<style:girl>`, `<style:pure>`, `<style:sister>`, `<style:sweet>`, `<style:exaggerated>`, `<style:ethereal>`, `<style:generous>`, `<style:recite>`, `<style:act_coy>`, `<style:warm>`, `<style:shy>`, `<style:comfort>`, `<style:authority>`, `<style:chat>`, `<style:radio>`, `<style:soulful>`, `<style:gentle>`, `<style:story>`, `<style:vivid>`, `<style:program>`, `<style:news>`, `<style:advertising>`, `<style:roar>`, `<style:murmur>`, `<style:shout>`, `<style:deeply>`, `<style:loudly>`, `<style:arrogant>`, `<style:friendly>`
@@ -638,7 +668,7 @@ A: Slightly. The TTS generates normally, then Step Audio EditX loads once and pr
A: Yes! Batch processing is automatically applied - all subtitles are generated first, then all edits are applied at once.
**Q: What if I make a typo in a tag?**
A: Unknown tags are ignored and stripped from the text. Check console for warnings.
A: The Step parser leaves unknown tags in the text and does not create an edit for them. Correct the spelling before generation; the TTS engine may interpret or speak the remaining text.
**Q: Can I nest tags?**
A: No. Use pipe syntax or separate tags: `<tag1|tag2>` or `<tag1><tag2>`
@@ -661,6 +691,10 @@ A: Just remove them from your text. No special setting needed.
## Support
Related guides:
- `docs/HIGGS_AUDIO_V3_INLINE_TAGS.md` for Higgs Audio v3 native tags
- `docs/COSYVOICE3_TAGS_GUIDE.md` for CosyVoice3 native tags
For issues, feature requests, or questions:
- GitHub Issues: https://github.com/diodiogod/TTS-Audio-Suite/issues
- Check console output for detailed processing information
+138 -28
View File
@@ -12,15 +12,27 @@ IndexTTS-2 supports multiple emotion control methods that can be combined for so
- **Text Emotion**: AI-powered QwenEmotion analysis from text descriptions with dynamic templates
- **Character Tag Emotions**: Per-character emotion control using `[Character:emotion_ref]` syntax
## Emotion Control Priority
You can only connect to the Engine node one source of control emotion: Either audio, text, or vectors.
When using tags on the text iself, **Character tag emotions** (highest priority) - `[Alice:angry_bob]` overrides all other emotion control settings for that character segment
## Emotion Control Inputs and Blending
The IndexTTS-2 Engine has two emotion inputs:
- **`emotion_control`**: vector controls or Qwen text emotion
- **`emotion_audio`**: an audio reference from an AUDIO or Character Voices node
Both inputs may be connected at the same time. Text emotion is analyzed into an
8-value vector; audio emotion remains an audio-derived conditioning signal. The
engine blends the two signals in its latent emotion-conditioning space rather
than converting the audio into the eight visible vector values.
For backward compatibility, the original `emotion_control` socket still accepts
legacy audio connections, but new workflows should use `emotion_audio` for
audio references. A character tag such as `[Alice:angry_bob]` supplies a
segment-local audio reference and can also be combined with vector/text emotion
for that segment.
## Method 1: Direct Audio Reference
Connect any audio file directly to the IndexTTS-2 Engine's `emotion_control` input.
Connect any audio file directly to the IndexTTS-2 Engine's `emotion_audio` input.
**How it works:**
@@ -37,7 +49,7 @@ Connect any audio file directly to the IndexTTS-2 Engine's `emotion_control` inp
**Example:**
```
AUDIO node → IndexTTS-2 Engine (emotion_control)
AUDIO node → IndexTTS-2 Engine (emotion_audio)
```
## Method 2: Character Voices Audio Reference
@@ -48,7 +60,7 @@ Use the `opt_narrator` output from the 🎭 Character Voices node as an emotion
1. Add a 🎭 Character Voices node
2. Select a voice with the desired emotional expression
3. Connect `opt_narrator` output to IndexTTS-2 Engine `emotion_control` input
3. Connect `opt_narrator` output to IndexTTS-2 Engine `emotion_audio` input
**Advantages:**
@@ -59,12 +71,13 @@ Use the `opt_narrator` output from the 🎭 Character Voices node as an emotion
**Example workflow:**
```
🎭 Character Voices (David_Attenborough) → opt_narrator → IndexTTS-2 Engine (emotion_control)
🎭 Character Voices (David_Attenborough) → opt_narrator → IndexTTS-2 Engine (emotion_audio)
```
## Method 3: Emotion Vectors
Use the 🌈 IndexTTS-2 Emotion Vectors node for precise manual control over 8 different emotions.
## Method 3: Emotion Vectors
Use the 🌈 IndexTTS-2 Emotion Vectors node for precise manual control over 8 different emotions.
Connect its `emotion_control` output to the IndexTTS-2 Engine's `emotion_control` input.
**Available emotions:**
@@ -84,9 +97,10 @@ Use the 🌈 IndexTTS-2 Emotion Vectors node for precise manual control over 8 d
- Start with single emotions, then experiment with combinations
- Use the `random` buttom to get a completely random emotion pattern. Might be too strong.
## Method 4: Text Emotion (Dynamic Analysis)
Use the 🌈 IndexTTS-2 Text Emotion node for AI-powered emotion analysis with dynamic templates.
## Method 4: Text Emotion (Dynamic Analysis)
Use the 🌈 IndexTTS-2 Text Emotion node for AI-powered emotion analysis with dynamic templates.
Connect its `emotion_control` output to the IndexTTS-2 Engine's `emotion_control` input.
### Static Text Emotion
@@ -124,11 +138,104 @@ Analysis: "Worried parent speaking: Where have you been?"
Result: Anxious, concerned vocal expression
```
---
## Character Tag Emotion Control
Control emotions per character using inline tags in your text: `[Character:emotion_ref]`
---
## Combining Audio with Vectors or Text
Connect both emotion sources when you want an audio performance to provide the
base delivery while vectors or Qwen text analysis add a targeted emotional
direction:
```text
🎭 Character Voices (opt_narrator) ──→ emotion_audio
🌈 Emotion Vectors or Text Emotion ──→ emotion_control
IndexTTS-2 Engine
```
`emotion_alpha` is the shared overall emotion-intensity control. The audio
reference and vector/text signal are blended during IndexTTS-2 conditioning;
they are not generated as two separate voices and mixed afterward.
For example, an audio reference can provide a natural speaking style while a
`[sad:+0.2|calm:-0.1]` inline adjustment adds a restrained sadness to one
segment. A Qwen text preset can be used the same way.
Character audio and inline emotion parameters can share one tag:
```text
[Bob:br_ivan_raiva3|sad:+0.25|calm:-0.10] Bob speaks with a restrained overlay.
[Bob:br_ivan_raiva3|emotion:"quiet grief masking frustration"] Bob uses Qwen text emotion too.
```
---
## Inline Emotion Switching
Numeric emotion tags can replace or adjust the vector for one text segment:
```text
[sad:0.7|calm:0.2] This uses explicit absolute values.
[sad:+0.3|calm:-0.2] This modifies the connected vector.
[vector:0,0,0.7,0,0,0.4,0,0.2] This supplies all eight absolute values.
[vector:+0,+0,+0.3,+0,+0,+0,+0,-0.2] This supplies eight deltas.
```
The full-vector order is `happy, angry, sad, afraid, disgusted, melancholic,
surprised, calm`. Unsigned named values are absolute; `+` and `-` named values
are relative. A full vector is relative only when every value carries a sign.
Text-emotion controls support saved presets and quoted descriptions:
```text
[emotion:restrained_anger] Text using a saved preset.
[emotion:"Restrained anger masking disappointment"] Direct text control.
[emotion:"Analyze this delivery as nervous anticipation: {seg}"] Dynamic control.
```
Inline controls override connected global vector/text values for that segment.
A character audio emotion reference such as `[Alice:sad_reference]` replaces the
global audio reference for that segment, but it can still blend with the
segment's vector/text control. Inline settings revert at the next segment and do
not mutate the connected vector.
The TTS Tag Editor provides the same interactive radar used by the IndexTTS-2
Emotion Vectors node. Click an existing numeric emotion tag to open its radar
as a contextual popover beside the tag. Create tags from **Inline Tags →
IndexTTS-2**, and use **Manage Emotion Presets** in its Text Emotion section for
the preset library. Text and vector presets are stored in
`models/TTS/IndexTTS/emotion_presets.json`.
Radar changes appear in the editor text immediately. Intermediate drag/input
states are not added to undo history: closing the popover commits one undoable
change, while Cancel or Escape restores the tag exactly as it was when opened.
The editor's **Inline Tags** tab also includes an **IndexTTS-2** engine panel for
inserting full absolute/delta vectors, named emotion values, saved text presets,
quoted descriptions, and dynamic descriptions containing `{seg}`.
Emotion controls can be composed directly on character tags: place the caret on
`[Bob:audio_reference]` and add a vector or text emotion to append it as a pipe
parameter.
The named-emotion panel includes a magnitude slider and a press-drag-release
radial picker: direction chooses the emotion and distance chooses its value.
The operation dropdown determines whether that value is absolute, a positive
delta, or a negative delta. Saved text and vector presets refresh in the sidebar
immediately after they are changed in the preset manager.
Clicking a saved `[emotion:preset_name]` tag in the editor opens a small anchored
preset dropdown, allowing that line's preset to be swapped without opening the
full manager. Adding an emotion control while the caret is inside a pure emotion
tag replaces that tag; when the caret is inside a character/audio tag, the
editor appends or updates the emotion parameter after the existing `|` fields.
Named tags remain readable while only some emotions are active. If radar editing
activates all eight emotions, the editor automatically converts the result to the
shorter ordered `[vector:...]` form.
## Character Tag Emotion Control
Control emotions per character using inline tags in your text: `[Character:emotion_ref]`
**Syntax:**
@@ -145,8 +252,9 @@ Control emotions per character using inline tags in your text: `[Character:emoti
```
Hello everyone! [Alice:happy_sarah] I'm so excited to be here today!
[Bob:angry_tom] That's completely unacceptable behavior.
[Narrator:David] Meanwhile, in a distant galaxy...
[Bob:angry_tom] That's completely unacceptable behavior.
[Narrator:David] Meanwhile, in a distant galaxy...
[Bob:br_ivan_raiva3|sad:+0.25] Bob uses an audio reference plus a vector delta.
*assuming happy_sarah, angry_tom and David are alias or character voices in yout folder with that name
```
@@ -155,11 +263,13 @@ Hello everyone! [Alice:happy_sarah] I'm so excited to be here today!
**Character tag priority:**
- Character tags override ALL other emotion settings for that specific character
- Other characters use global emotion settings
- Allows mixing different emotions in the same audio
**Character tag behavior:**
- Character tags select the speaker and can provide a segment-local audio emotion reference
- A segment-local audio reference replaces the global audio reference for that segment
- Global or inline vector/text controls can still blend with that audio reference
- Other characters use the global audio and vector/text settings
- Allows mixing different emotion sources in the same audio
## Emotion Alpha Control
@@ -211,4 +321,4 @@ Welcome to our show! [Bob:serious_narrator] But first, a serious announcement.
---
This comprehensive emotion control system gives you unprecedented flexibility in creating expressive, emotionally rich TTS audio for any application.
This comprehensive emotion control system gives you unprecedented flexibility in creating expressive, emotionally rich TTS audio for any application.
+108 -30
View File
@@ -2,40 +2,118 @@
## Language Support by Engine
| Language | Code | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | RVC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 🇺🇸 **English** | EN | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| 🇨🇳 **Chinese** | ZH | ❌ | ❌ | ✅ | ✅ | ✅ (Mandarin) | ✅ | ✅ + 18 dialects | ✅ | ❌ | ✅ (Mandarin + Sichuanese, Cantonese) | ❌ | ✅ |
| 🇩🇪 **German** | DE | ✅ | ✅ (×3) | ✅ | ✅ (Kugel) | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ |
| 🇪🇸 **Spanish** | ES | ✅ | ❌ | ✅ | ✅ (Kugel) | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ |
| 🇫🇷 **French** | FR | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ |
| 🇮🇹 **Italian** | IT | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ |
| 🇯🇵 **Japanese** | JA | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ ? | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
| 🇰🇷 **Korean** | KO | ❌ | ✅ | ✅ | ✅ (Kugel) | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ |
| 🇷🇺 **Russian** | RU | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇷 **Portuguese** | PT | ✅ (BR) | ❌ | ✅ | ✅ (Kugel) | ❌ | ❌ | ❌ | ✅ (EU/BR*) | ✅ | ❌ | ❌ | ✅ |
| 🇵🇱 **Polish** | PL | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Hindi** | HI | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇬 **Arabic** | AR | ❌ | ❌ | ✅ (Egyptian) | ✅ (Kugel) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇹🇷 **Turkish** | TR | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇹🇭 **Thai** | TH | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇴 **Norwegian** | NO | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇻🇳 **Vietnamese** | VI | ❌ | ❌ | ✅ (Viterbox) | ✅ (Kugel) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇲 **Armenian** | HY | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇬🇪 **Georgian** | KA | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇩🇰 **Danish** | DA | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇫🇮 **Finnish** | FI | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇬🇷 **Greek** | EL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇱 **Hebrew** | HE | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇾 **Malay** | MS | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇱 **Dutch** | NL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇸🇪 **Swedish** | SV | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇪 **Swahili** | SW | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| Language | Code | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS 2 / 2.5 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | DramaBox | OmniVoice | MOSS-TTS | MOSS-SoundEffect v2 | RVC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 🇺🇸 **English** | EN | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ Tier 1 | ✅ | ✅ Official model is English-only | ✅ | ✅ | ✅ Officially demonstrated prompt language | ✅ |
| 🇨🇳 **Chinese** | ZH | ❌ | ❌ | ✅ | ✅ | ✅ (Mandarin) | ✅ | ✅ | ✅ + 18 dialects | ✅ | ❌ | ✅ (Mandarin + Sichuanese, Cantonese) | ❌ | ✅ Tier 1 | ✅ (Mandarin; official also exposes YUE separately outside this matrix) | ❌ | ✅ | ✅ | ✅ Officially demonstrated prompt language | ✅ |
| 🇩🇪 **German** | DE | ✅ | ✅ (×3) | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇪🇸 **Spanish** | ES | ✅ | ❌ | ✅ | ✅ (Kugel) | ✅ | ✅ | ✅ IndexTTS-2.5 | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇫🇷 **French** | FR | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇮🇹 **Italian** | IT | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇯🇵 **Japanese** | JA | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ✅ IndexTTS-2.5 | ✅ | ✅ | ✅ (4.1-2b supports Japanese; plus variant does not) | ✅ | ❌ | ✅ Tier 1 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇰🇷 **Korean** | KO | ❌ | ✅ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇷🇺 **Russian** | RU | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇧🇷 **Portuguese** | PT | ✅ (BR) | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ (EU/BR*) | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ (Official PT tag is generic; upstream does not expose separate PT-BR/PT-PT tags and it may lean more European Portuguese than Brazilian Portuguese) | ❌ | ✅ (generic PT; official language space is much broader than this matrix) | ✅ | ❌ | ✅ |
| 🇵🇱 **Polish** | PL | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇮🇳 **Hindi** | HI | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇪🇬 **Arabic** | AR | ❌ | ❌ | ✅ (Egyptian) | ✅ (Kugel) | ❌ | ✅ | ✅ IndexTTS-2.5 | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇹🇷 **Turkish** | TR | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇹🇭 **Thai** | TH | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇳🇴 **Norwegian** | NO | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
| 🇻🇳 **Vietnamese** | VI | ❌ | ❌ | ✅ (Viterbox) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇦🇲 **Armenian** | HY | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
| 🇬🇪 **Georgian** | KA | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
| 🇩🇰 **Danish** | DA | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇫🇮 **Finnish** | FI | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇬🇷 **Greek** | EL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇮🇱 **Hebrew** | HE | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇲🇾 **Malay** | MS | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇳🇱 **Dutch** | NL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇸🇪 **Swedish** | SV | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇰🇪 **Swahili** | SW | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇿🇦 **Afrikaans** | AF | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇱 **Albanian** | SQ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Assamese** | AS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Asturian** | AST | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇿 **Azerbaijani** | AZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇺 **Bashkir** | BA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Basque** | EU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇾 **Belarusian** | BE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇩 **Bengali** | BN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇦 **Bosnian** | BS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇬 **Bulgarian** | BG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Catalan** | CA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇭 **Cebuano** | CEB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇶 **Central Kurdish** | CKB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇭🇷 **Croatian** | HR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇿 **Czech** | CS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇺 **Eastern Mari** | MHR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🌍 **Esperanto** | EO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇪 **Estonian** | ET | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Galician** | GL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Gujarati** | GU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇭🇹 **Haitian Creole** | HT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇬 **Hausa** | HA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇭🇺 **Hungarian** | HU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇩 **Indonesian** | ID | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇩 **Javanese** | JV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Kannada** | KN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇿 **Kazakh** | KK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇼 **Kinyarwanda** | RW | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇬 **Kyrgyz** | KY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇱🇻 **Latvian** | LV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇩 **Lingala** | LN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇱🇹 **Lithuanian** | LT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇪 **Luo** | LUO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇰 **Macedonian** | MK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Malayalam** | ML | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇹 **Maltese** | MT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇿 **Māori** | MI | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Marathi** | MR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇳 **Mongolian** | MN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇵 **Nepali** | NE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇫🇷 **Occitan** | OC | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇷 **Persian** | FA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇴 **Romanian** | RO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇦 **Sepedi** | NSO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇸 **Serbian** | SR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇼 **Shona** | SN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇸🇰 **Slovak** | SK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇸🇮 **Slovene** | SL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇭 **Tagalog** | TL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇹🇯 **Tajik** | TG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Tamil** | TA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Telugu** | TE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇺🇦 **Ukrainian** | UK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇰 **Urdu** | UR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇳 **Uyghur** | UG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇺🇿 **Uzbek** | UZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇦 **Xhosa** | XH | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇦 **Zulu** | ZU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇼 **Chichewa/Nyanja** | NY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Eastern Punjabi** | PA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇺🇬 **Ganda** | LG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇸 **Icelandic** | IS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇪 **Irish** | GA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇩🇿 **Kabyle** | KAB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇻 **Kabuverdianu** | KEA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇪 **Kamba** | KAM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇻🇦 **Latin** | LA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇱🇺 **Luxembourgish** | LB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇹 **Oromo** | OM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇫 **Pashto** | PS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇰 **Sindhi** | SD | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇸🇴 **Somali** | SO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇴 **Umbundu** | UMB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🏴 **Welsh** | CY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
**Notes:**
- **CosyVoice3 Chinese**: Includes 18+ dialects (Cantonese, Sichuan, Dongbei, Shanghai, etc.)
- **Higgs Audio 2**: Trained on EN, ZH (Mandarin), KO, DE, ES (English majority) - 10M hours AudioVerse dataset
- **Higgs Audio v3**: Official model card lists 102 languages total: 85 polished-production languages and 17 usable but less polished languages.
- **IndexTTS-2**: Trained on 55K+ hours - ZH, EN, JA primary.
- **RVC**: Language-independent (work as post-processors VC with any TTS output)
- **MOSS-TTS**: Official list includes ZH, EN, DE, ES, FR, JA, IT, HU, KO, RU, FA, AR, PL, PT, CS, DA, SV, EL, TR; HU/FA/CS are not separate columns in this matrix.
- **OmniVoice**: Official model supports 600+ languages; this matrix only shows the suite's comparison subset.
- **RVC**: Language-agnostic voice conversion/post-processing; it is marked supported for every language row.
- Qwen3-TTS: Supports pt-BR only with instruction (voice design or harcoded voices). Base can't do instructions, so it will always ouput pt-PT.
+63 -3
View File
@@ -38,7 +38,7 @@ Use this as the canonical list of model repositories/links for offline setup.
| Component | Source | Size | Auto-Download | Notes |
|---|---|---|---|---|
| Official 23-Lang (v1/v2) | [ResembleAI/chatterbox](https://huggingface.co/ResembleAI/chatterbox) | ~4.3GB | ✅ | v1 + v2 files and tokenizer |
| Official 23-Lang (v1/v2/v3) | [ResembleAI/chatterbox](https://huggingface.co/ResembleAI/chatterbox) | ~4.3GB | ✅ | v1 + v2 + v3 T3 checkpoints with shared tokenizer, voice encoder, and S3Gen |
| Russian stress dictionary (Russian only) | [Vuizur/add-stress-to-epub release](https://github.com/Vuizur/add-stress-to-epub/releases/download/v1.0.1/russian_dict.zip) | ~1.5GB | ✅ | Auxiliary Official 23-Lang Russian stress-labeling data; downloads on demand only when Russian stress support is used |
| Vietnamese (Viterbox) | [dolly-vn/viterbox](https://huggingface.co/dolly-vn/viterbox) | ~4.3GB | ✅ | Vietnamese community finetune used by downloader |
| Egyptian Arabic (oddadmix) | [oddadmix/chatterbox-egyptian-v0](https://huggingface.co/oddadmix/chatterbox-egyptian-v0) | ~4.3GB | ✅ | Egyptian Arabic community finetune (architecture v2) |
@@ -49,6 +49,8 @@ Use this as the canonical list of model repositories/links for offline setup.
|---|---|---|---|---|
| vibevoice-1.5B | [microsoft/VibeVoice-1.5B](https://huggingface.co/microsoft/VibeVoice-1.5B) | ~5.4GB | ✅ | Microsoft official model |
| vibevoice-7B | [aoi-ot/VibeVoice-Large](https://huggingface.co/aoi-ot/VibeVoice-Large) | ~18GB | ✅ | Community mirror used by downloader |
| kugelaudio-0-open | [kugelaudio/kugelaudio-0-open](https://huggingface.co/kugelaudio/kugelaudio-0-open) | ~18GB | ✅ | KugelAudio multilingual 7B variant |
| kugel-2 | [kugelaudio/kugel-2](https://huggingface.co/kugelaudio/kugel-2) | ~18.7GB | ✅ | KugelAudio v2 merged 7B variant |
## Higgs Audio 2
@@ -57,11 +59,18 @@ Use this as the canonical list of model repositories/links for offline setup.
| higgs-audio-v2-3B | [bosonai/higgs-audio-v2-generation-3B-base](https://huggingface.co/bosonai/higgs-audio-v2-generation-3B-base) | ~9GB | ✅ | Generation model |
| Audio tokenizer | [bosonai/higgs-audio-v2-tokenizer](https://huggingface.co/bosonai/higgs-audio-v2-tokenizer) | ~200MB | ✅ | Tokenizer model |
## IndexTTS-2
## Higgs Audio v3
| Component | Source | Size | Auto-Download | Notes |
|---|---|---|---|---|
| higgs-audio-v3-tts-4b | [bosonai/higgs-audio-v3-tts-4b](https://huggingface.co/bosonai/higgs-audio-v3-tts-4b) | ~8GB | ✅ | Official 4B multilingual controllable TTS model |
## IndexTTS 2 / 2.5
| Component | Source | Size | Auto-Download | Notes |
|---|---|---|---|---|
| IndexTTS-2 | [IndexTeam/IndexTTS-2](https://huggingface.co/IndexTeam/IndexTTS-2) | Multiple files | ✅ | Main TTS engine |
| IndexTTS-2.5 | [IndexTeam/IndexTTS-2.5](https://huggingface.co/IndexTeam/IndexTTS-2.5) | ~5.49GB | ✅ | Multilingual backend with bundled codec and official feature-duration scaling |
| w2v-bert-2.0 | [facebook/w2v-bert-2.0](https://huggingface.co/facebook/w2v-bert-2.0) | ~2GB | ✅ | Semantic feature extractor |
| qwen0.6bemo4-merge | Included with IndexTTS-2 | Included | ✅ | Text emotion model bundle |
@@ -88,13 +97,16 @@ Use this as the canonical list of model repositories/links for offline setup.
| Component | Source | Size | Auto-Download | Notes |
|---|---|---|---|---|
| granite-4.0-1b-speech | [ibm-granite/granite-4.0-1b-speech](https://huggingface.co/ibm-granite/granite-4.0-1b-speech) | ~4.6GB | ✅ | Main Granite ASR / AST model |
| granite-speech-4.1-2b | [ibm-granite/granite-speech-4.1-2b](https://huggingface.co/ibm-granite/granite-speech-4.1-2b) | ~4.6GB | ✅ | Granite Speech 4.1 2B multilingual model (supports Japanese) |
| granite-speech-4.1-2b-plus | [ibm-granite/granite-speech-4.1-2b-plus](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-plus) | ~4.6GB | ✅ | Plus variant supporting native speaker diarization and word-level timestamps |
| Qwen3-ForcedAligner-0.6B | [Qwen/Qwen3-ForcedAligner-0.6B](https://huggingface.co/Qwen/Qwen3-ForcedAligner-0.6B) | N/A | ✅ | Optional custom word-level timestamps/SRT path; reused from Qwen folder |
## Step Audio EditX
| Component | Source | Size | Auto-Download | Notes |
|---|---|---|---|---|
| Step-Audio-EditX | [stepfun-ai/Step-Audio-EditX](https://huggingface.co/stepfun-ai/Step-Audio-EditX) | ~7GB | ✅ | Main 3B audio editing model |
| Step-Audio-EditX-2026-01-23 | [stepfun-ai/Step-Audio-EditX](https://huggingface.co/stepfun-ai/Step-Audio-EditX) | ~7GB | ✅ | Default 3B checkpoint; pinned to 5fe2f8a05c2353301ad47d3c1747b262115da138 |
| Step-Audio-EditX-2025-11-28 | [stepfun-ai/Step-Audio-EditX](https://huggingface.co/stepfun-ai/Step-Audio-EditX/tree/7f3de603ae46c96dff6f06f47b1d5a45aabd34fe) | ~7GB | ✅ | Legacy checkpoint; pinned November weights in a separate folder |
| Step-Audio-Tokenizer | [stepfun-ai/Step-Audio-Tokenizer](https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer) | Included | ✅ | Tokenizer bundle used by Step EditX |
## Echo-TTS
@@ -104,6 +116,53 @@ Use this as the canonical list of model repositories/links for offline setup.
| echo-tts-base (model + PCA state) | [jordand/echo-tts-base](https://huggingface.co/jordand/echo-tts-base) | ~5.3GB | ✅ | pytorch_model.safetensors + pca_state.safetensors |
| fish-s1-dac-min (audio codec) | [jordand/fish-s1-dac-min](https://huggingface.co/jordand/fish-s1-dac-min) | ~1.8GB | ✅ | pytorch_model.safetensors — audio codec required by Echo-TTS |
## Fish Audio S2 Pro
| Component | Source | Size | Auto-Download | Notes |
|---|---|---|---|---|
| S2 Pro | [fishaudio/s2-pro](https://huggingface.co/fishaudio/s2-pro) | ~10.3GB | ✅ | Official 4B model and codec; BNB INT8/NF4 are optional load-time quantization modes that reuse these files; non-commercial license |
| S2 Pro FP8 | [drbaph/s2-pro-fp8](https://huggingface.co/drbaph/s2-pro-fp8) | ~8.0GB | ✅ | Community weight-only FP8 checkpoint; BF16 activations; RTX 4090/5090-class CUDA GPU required |
## Dots TTS
| Component | Source | Size | Auto-Download | Notes |
|---|---|---|---|---|
| dots.tts-base | [rednote-hilab/dots.tts-base](https://huggingface.co/rednote-hilab/dots.tts-base) | ~6GB | ✅ | Official base checkpoint with tokenizer, vocoder, speaker encoder, and latent stats |
| dots.tts-soar | [rednote-hilab/dots.tts-soar](https://huggingface.co/rednote-hilab/dots.tts-soar) | ~6GB | ✅ | Official SOAR checkpoint for higher-quality zero-shot cloning |
| dots.tts-mf | [rednote-hilab/dots.tts-mf](https://huggingface.co/rednote-hilab/dots.tts-mf) | ~6GB | ✅ | Official MeanFlow-distilled checkpoint for faster inference |
## DramaBox
| Component | Source | Size | Auto-Download | Notes |
|---|---|---|---|---|
| DramaBox DiT + audio components | [ResembleAI/Dramabox](https://huggingface.co/ResembleAI/Dramabox) | ~8.5GB | ✅ | Official merged DramaBox transformer and LTX audio VAE/vocoder components |
| Gemma 3 12B 4-bit text encoder | [unsloth/gemma-3-12b-it-bnb-4bit](https://huggingface.co/unsloth/gemma-3-12b-it-bnb-4bit) | ~7.8GB | ✅ | Official pre-quantized text encoder; loaded locally with no HF cache fallback |
## OmniVoice
| Component | Source | Size | Auto-Download | Notes |
|---|---|---|---|---|
| OmniVoice | [k2-fsa/OmniVoice](https://huggingface.co/k2-fsa/OmniVoice) | ~3.7GB | ✅ | Official main model including bundled audio_tokenizer assets |
## MOSS-TTS
| Component | Source | Size | Auto-Download | Notes |
|---|---|---|---|---|
| MOSS-TTS-Local-Transformer | [OpenMOSS-Team/MOSS-TTS-Local-Transformer](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Local-Transformer) | ~6.1GB | ✅ | Official 1.7B local-transformer model |
| MOSS-TTS | [OpenMOSS-Team/MOSS-TTS](https://huggingface.co/OpenMOSS-Team/MOSS-TTS) | ~17GB | ✅ | Official 8B delay model |
| MOSS-TTS-v1.5 | [OpenMOSS-Team/MOSS-TTS-v1.5](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-v1.5) | ~17GB | ✅ | Current official 8B delay model with 31 languages and more stable voice cloning |
| MOSS-TTS v1.5 Voice Acting 8B (Community - LAION) | [laion/moss-tts-v1.5-8b-voice-acting](https://huggingface.co/laion/moss-tts-v1.5-8b-voice-acting) | ~17GB | ✅ | Third-party full MOSS-TTS v1.5 fine-tune for expressive voice acting; not an official OpenMOSS model |
| MOSS-VoiceGenerator | [OpenMOSS-Team/MOSS-VoiceGenerator](https://huggingface.co/OpenMOSS-Team/MOSS-VoiceGenerator) | ~4.2GB | ✅ | Official 1.7B reference-free voice-design model |
| MOSS-TTSD-v1.0 | [OpenMOSS-Team/MOSS-TTSD-v1.0](https://huggingface.co/OpenMOSS-Team/MOSS-TTSD-v1.0) | ~18GB | ✅ | Official 8B native multi-speaker dialogue model |
| MOSS-SoundEffect | [OpenMOSS-Team/MOSS-SoundEffect](https://huggingface.co/OpenMOSS-Team/MOSS-SoundEffect) | ~17GB | ✅ | Official MOSS v1 prompt-only sound-effect checkpoint; uses the shared MOSS audio tokenizer |
| MOSS-Audio-Tokenizer | [OpenMOSS-Team/MOSS-Audio-Tokenizer](https://huggingface.co/OpenMOSS-Team/MOSS-Audio-Tokenizer) | ~8.5GB | ✅ | Shared official codec required by MOSS-TTS |
## MOSS-SoundEffect v2
| Component | Source | Size | Auto-Download | Notes |
|---|---|---|---|---|
| MOSS-SoundEffect-v2.0 | [OpenMOSS-Team/MOSS-SoundEffect-v2.0](https://huggingface.co/OpenMOSS-Team/MOSS-SoundEffect-v2.0) | ~11.2GB | ✅ | Official DiT + DAC VAE + Qwen3 text-encoder sound-effect pipeline |
## RVC
| Component | Source | Size | Auto-Download | Notes |
@@ -112,5 +171,6 @@ Use this as the canonical list of model repositories/links for offline setup.
| RVC index pack (.index) | [SayanoAI/RVC-Studio (.index folder)](https://huggingface.co/datasets/SayanoAI/RVC-Studio/tree/main/RVC/.index) | Varies | ✅ | Optional FAISS indexes for improved voice similarity |
| content-vec-best.safetensors | [lengyue233/content-vec-best](https://huggingface.co/lengyue233/content-vec-best) | ~300MB | ✅ | Voice feature model |
| rmvpe.pt | [lj1995/VoiceConversionWebUI](https://huggingface.co/lj1995/VoiceConversionWebUI) | ~55MB | ✅ | Pitch extraction model |
| pretrained_v2 (f0 G/D pairs) | [lj1995/VoiceConversionWebUI](https://huggingface.co/lj1995/VoiceConversionWebUI/tree/main/pretrained_v2) | ~300MB total | ✅ | Training init checkpoints for 32k/40k/48k RVC runs; downloaded on first training use |
*Generated from [tts_audio_suite_engines.yaml](Dev%20reports/tts_audio_suite_engines.yaml).*
+212 -10
View File
@@ -104,17 +104,11 @@ ComfyUI/models/voices/
Requirements:
- WAV, clean speech, 5-30s (24kHz recommended).
- Clean speech, shorter than 12 seconds, with a little trailing silence.
- Text must match spoken audio.
- Naming: `name.wav` + `name.reference.txt` (preferred) or `name.txt`.
Inference guidelines:
1. Keep reference audio under ~12s with a little trailing silence.
2. Use uppercase letter-by-letter only when desired.
3. Use spaces/punctuation for explicit pauses.
4. Keep a space after sentence-ending punctuation.
5. For Chinese output, preprocess numbers if needed.
See [F5-TTS Reference and Inference Guidelines](F5_TTS_REFERENCE_GUIDE.md) for sample rates, reference clipping, pacing, and text preparation.
## Higgs Audio 2
@@ -154,6 +148,13 @@ ComfyUI/models/TTS/RVC/
├── content-vec-best.safetensors
├── rmvpe.pt
├── hubert/
├── pretrained_v2/
│ ├── f0G32k.pth
│ ├── f0D32k.pth
│ ├── f0G40k.pth
│ ├── f0D40k.pth
│ ├── f0G48k.pth
│ └── f0D48k.pth
└── .index/
```
@@ -167,6 +168,8 @@ Notes:
- Base models auto-download.
- Character `.pth` models can be auto-downloaded defaults or user-provided.
- `pretrained_v2/` is used by the integrated RVC trainer and auto-downloads on first training run.
- Training datasets, logs, progress snapshots, and resumable checkpoints are stored under `ComfyUI/output/tts_audio_suite_training/rvc/`, not inside the custom node repo.
- UVR models are downloaded under `ComfyUI/models/TTS/UVR/` (or legacy `ComfyUI/models/UVR/`).
## IndexTTS-2
@@ -185,14 +188,71 @@ Notes:
```text
ComfyUI/models/TTS/step_audio_editx/
├── Step-Audio-EditX/
├── Step-Audio-EditX-2026-01-23/
│ └── CosyVoice-300M-25Hz/
├── Step-Audio-EditX-2025-11-28/ # downloaded only when selected
│ └── CosyVoice-300M-25Hz/
├── Step-Audio-EditX/ # existing unversioned installations
│ └── CosyVoice-300M-25Hz/
└── FunASR-Paraformer/
```
Notes:
- Main model, tokenizer assets, and speech stack auto-download.
- Selected checkpoints and tokenizer assets auto-download at pinned revisions.
- Existing unversioned/local weights are retained and folders are not relabeled; known config compatibility fields may be normalized.
- See [Step Audio EditX checkpoints](STEP_AUDIO_EDITX_CHECKPOINTS.md) for selection and expanded tags.
## Higgs Audio v3
```text
ComfyUI/models/TTS/higgs_audio_v3/
└── higgs-audio-v3-tts-4b/
├── config.json
├── tokenizer.json
├── tokenizer_config.json
├── chat_template.jinja
├── model.safetensors
├── model.safetensors.index.json
└── LICENSE
```
Notes:
- Downloads into its own `higgs_audio_v3` folder.
- Requires the main Transformers 5 environment.
- Reference transcript `.txt` files are optional but improve cloning quality.
## DramaBox
```text
ComfyUI/models/TTS/dramabox/
├── DramaBox/
├── dramabox-dit-v1.safetensors
├── dramabox-audio-components.safetensors
├── assets/
│ └── silence_latent_frame.pt
└── gemma-3-12b-it-bnb-4bit/
├── config.json
├── model-00001-of-00002.safetensors
├── model-00002-of-00002.safetensors
├── model.safetensors.index.json
└── tokenizer and processor files...
└── loras/
└── <adapter_name>/
├── adapter_config.json
└── adapter_model.safetensors
```
Notes:
- Both repositories download directly into the organized suite folder.
- Transformers is forced into local-only loading after download.
- Requires NVIDIA CUDA. Fast mode targets approximately 24GB VRAM; experimental staged/sequential modes can run with less.
- Integrated training exports LoRA adapters into `dramabox/loras/<adapter_name>/`.
- Training jobs, normalized datasets, preprocessing caches, checkpoints, and logs are stored under `ComfyUI/output/tts_audio_suite_training/dramabox/`.
- The LTX-2 Community License requires a paid license for entities with at
least USD 10 million in annual revenue.
## CosyVoice3
@@ -229,6 +289,57 @@ Notes:
- Only selected variants download.
- Shared tokenizer assets are reused.
## MOSS-TTS
```text
ComfyUI/models/TTS/moss_tts/
├── MOSS-TTS-Local-Transformer/
├── MOSS-TTS-v1.5/
├── moss-tts-v1.5-8b-voice-acting/ # Community - LAION
├── MOSS-TTS/
├── MOSS-VoiceGenerator/
├── MOSS-SoundEffect/
├── MOSS-TTSD-v1.0/
├── MOSS-Audio-Tokenizer/
└── loras/
└── <adapter_name>/
├── adapter_config.json
└── adapter_model.safetensors
```
Notes:
- `MOSS-Audio-Tokenizer` is required by the official TTS and TTSD variants.
- `MOSS-TTS-Local-Transformer` is the smaller 1.7B model.
- `MOSS-TTS-v1.5` is the current 8B delay model with 31-language support.
- `moss-tts-v1.5-8b-voice-acting` is an optional third-party LAION full fine-tune for expressive speech, not an official OpenMOSS model.
- Other compatible full checkpoints placed here are discovered from their `config.json`; unsupported MOSS architectures are rejected explicitly.
- `MOSS-TTS` is the legacy official 8B delay model.
- `MOSS-VoiceGenerator` is the 1.7B voice-design provider used by Voice Designer.
- `MOSS-SoundEffect` is the v1 sound-effect checkpoint used through the MOSS-TTS engine and 🌩️ Sound Effects.
- `MOSS-TTSD-v1.0` is the official 8B native multi-speaker dialogue model.
- Integrated training currently exports LoRA adapters into `moss_tts/loras/<adapter_name>/`.
- Training jobs, temporary manifests, and checkpoints are stored under `ComfyUI/output/tts_audio_suite_training/moss_tts/`.
## MOSS-SoundEffect v2
```text
ComfyUI/models/TTS/moss_soundeffect_v2/
└── MOSS-SoundEffect-v2.0/
├── model_index.json
├── scheduler/
├── text_encoder/
├── tokenizer/
├── transformer/
└── vae/
```
Notes:
- This is a separate v2 diffusion family, not a MOSS-TTS checkpoint variant.
- It runs in the configured ComfyUI environment; the official Apache-2.0 inference package is bundled without modifying its dependencies.
- The 🌩️ Sound Effects node limits generation to the official 30-second maximum.
## Granite ASR
```text
@@ -245,6 +356,7 @@ ComfyUI/models/TTS/granite_asr/
Notes:
- Granite downloads into its own `granite_asr` folder.
- `granite-speech-4.1-2b-plus` adds native speaker diarization and native word-level timestamps, but drops Japanese.
- If Granite word timestamps are enabled, it lazily reuses `Qwen3-ForcedAligner-0.6B` from the Qwen ASR folder instead of duplicating that model.
## Echo-TTS
@@ -262,3 +374,93 @@ Notes:
- Both components are required and auto-downloaded on first use.
- License: CC-BY-NC-SA (non-commercial).
## Fish Audio S2 Pro
```text
ComfyUI/models/TTS/fish_audio_s2_pro/
├── codec.pth
├── config.json
├── model-00001-of-00002.safetensors
├── model-00002-of-00002.safetensors
├── model.safetensors.index.json
└── tokenizer.json
```
Optional FP8 variant:
```text
ComfyUI/models/TTS/fish_audio_s2_pro_fp8/
├── codec.pth
├── config.json
├── model.safetensors
├── quantization_info.json
└── tokenizer.json
```
The complete official repository metadata and tokenizer files are downloaded alongside these files. License: Fish Audio Research License (non-commercial without a separate commercial license).
The `s2-pro-bnb-int8` and `s2-pro-bnb-nf4` options reuse `fish_audio_s2_pro/` and quantize its official checkpoint while loading. They do not download another model copy and require `bitsandbytes`.
## Dots TTS
```text
ComfyUI/models/TTS/dots_tts/
├── dots.tts-base/
├── dots.tts-soar/
└── dots.tts-mf/
```
Typical checkpoint contents:
```text
ComfyUI/models/TTS/dots_tts/dots.tts-soar/
├── added_tokens.json
├── chat_template.jinja
├── config.json
├── latent_stats.pt
├── llm_config.json
├── merges.txt
├── model.safetensors
├── speaker_encoder.safetensors
├── special_tokens_map.json
├── tokenizer.json
├── tokenizer_config.json
├── vocab.json
└── vocoder.safetensors
```
Notes:
- Downloads into a dedicated `dots_tts/` folder.
- Native sample rate is 48kHz.
- Main-environment support works on Transformers 5; on Windows, `normalize_text` falls back to no-op if `WeTextProcessing` is unavailable.
## OmniVoice
```text
ComfyUI/models/TTS/omnivoice/
└── OmniVoice/
```
Main model contents:
```text
ComfyUI/models/TTS/omnivoice/OmniVoice/
├── chat_template.jinja
├── config.json
├── model.safetensors
├── tokenizer.json
├── tokenizer_config.json
└── audio_tokenizer/
├── config.json
├── model.safetensors
└── preprocessor_config.json
```
Notes:
- Downloads into a dedicated `omnivoice/` folder.
- Native sample rate is 24kHz.
- The main OmniVoice model stays in the main Transformers 5 environment.
- Voice cloning requires reference audio plus explicit reference text in this suite.
+119
View File
@@ -0,0 +1,119 @@
# MOSS LoRA Guide
This guide defines the local adapter layout and loading behavior for **MOSS-TTS LoRA adapters** in TTS Audio Suite.
## Local Folder Layout
MOSS LoRA adapters should live under:
```text
ComfyUI/models/TTS/moss_tts/loras/
```
Each adapter must have its **own folder**:
```text
ComfyUI/models/TTS/moss_tts/loras/
MyNorwegianLoRA/
adapter_config.json
adapter_model.safetensors
```
Recommended structure:
```text
<adapter_name>/
adapter_config.json
adapter_model.safetensors
```
Also accepted:
```text
<adapter_name>/
adapter_config.json
adapter_model.bin
```
## Required Files
Each adapter folder must contain:
- `adapter_config.json`
- one adapter weights file:
- `adapter_model.safetensors` preferred
- or `adapter_model.bin`
Extra files are allowed, for example:
- `README.md`
- training metadata
- tokenizer notes
## What Is Not Supported
Do not place loose adapter files directly inside:
```text
ComfyUI/models/TTS/moss_tts/loras/
```
This is **not** a supported layout:
```text
loras/
adapter_config.json
some_weights.safetensors
```
This is also **not** supported:
```text
loras/
MyLoRA.safetensors
```
One adapter must always be one folder.
## Engine Node Behavior
The MOSS engine node supports two adapter paths:
1. **Local LoRA dropdown**
- lists valid adapter folders discovered under `models/TTS/moss_tts/loras`
2. **Advanced override field**
- accepts:
- a local adapter folder path
- a Hugging Face repo id
If you enter a Hugging Face repo id, TTS Audio Suite installs it into:
```text
ComfyUI/models/TTS/moss_tts/loras/<owner__repo>/
```
Then it loads the adapter from that local folder.
## Why This Layout Exists
TTS Audio Suite does **not** use Hugging Face cache as the final managed storage location for MOSS LoRAs.
This layout keeps adapters:
- visible in the MOSS dropdown
- stored inside the normal TTS model tree
- ready for future training output
- separate from ComfyUI diffusion LoRA folders
## Training Output Standard
Future MOSS training should write adapters in exactly this format:
```text
ComfyUI/models/TTS/moss_tts/loras/<run_name>/
adapter_config.json
adapter_model.safetensors
```
That keeps training, manual installs, and inference all using the same contract.
+258
View File
@@ -0,0 +1,258 @@
# MOSS Training Guide
This guide explains the **practical dataset meaning** of the current MOSS training nodes in TTS Audio Suite.
Use this if `🧾 MOSS Dataset Rows` feels unclear.
## Current Training Scope
Current first training slice supports:
- **MOSS-TTS 8B v1.0 and v1.5 (Delay)**
- **LAION MOSS-TTS v1.5 Voice Acting 8B community full checkpoint (Delay, compatibility path; training results not yet validated by the suite maintainers)**
- **LoRA adapter training**
The model selected on the connected MOSS engine is used for dataset preparation and training. Prepare the dataset again after switching between v1.0 and v1.5.
The LAION Voice Acting checkpoint uses the same Delay architecture and can use this LoRA training path, but the suite maintainers have not completed an inference or training run with its full weights. Treat it as community-tested support and report results or incompatibilities.
It does **not** currently support:
- Local 1.7B training
- TTSD training
- full-parameter training from the node path
## Normal Training Flow
Current ComfyUI flow:
1. `🎞️ Training Clip Staging`
2. `🧾 MOSS Dataset Rows`
3. `📦 MOSS Dataset Prep`
4. `🎛️ MOSS Training Config`
5. `🎓 Model Training`
If clips and transcripts are already prepared on disk, you can skip the first two
nodes. Set `dataset_source` on `📦 MOSS Dataset Prep` to a folder containing
same-name audio and text pairs:
```text
my_dataset/
├── clip001.wav
├── clip001.txt
├── clip002.flac
└── clip002.txt
```
Each `.txt` file must contain the transcript spoken in its matching audio file.
Folder scanning supports WAV, FLAC, MP3, OGG, and M4A. Subfolders are ignored
unless `recursive_folder_scan` is enabled. Existing JSONL manifest paths continue
to work unchanged.
## The Important Fields
### `text_lines`
This is the easiest one:
- one line per staged clip
- each line is the **transcript/text of that clip**
Example:
- `clip001` audio says: `Hello there.`
- `clip002` audio says: `How are you?`
Then:
```text
Hello there.
How are you?
```
That is the normal default dataset shape:
- `audio + transcript`
For many voice LoRAs, this is enough.
### `reference_clip_lines`
This is **advanced**. Most users should leave it blank at first.
It means:
- which other staged clip should be used as the **reference audio** for that row
That teaches the model a different pattern:
- `text` says **what to say**
- `reference audio` says **whose voice/style to follow**
- `target audio` is the **correct output**
Example:
- `clip001` = Bob saying: `My name is Bob.`
- `clip002` = Bob saying: `Welcome to the show.`
Row 2 can be:
- target audio = `clip002`
- text = `Welcome to the show.`
- reference = `clip001`
That teaches:
- “use `clip001` as the reference voice, then produce the target line from `clip002`”
## When To Ignore `reference_clip_lines`
Ignore it for:
- first MOSS LoRA tests
- single-speaker voice LoRAs
- fixed-voice adaptation
- plain language/domain adaptation
That means:
- just use `audio + transcript`
- maybe also use `instruction`
This is the best default.
## When To Use `reference_clip_lines`
Use it only if you deliberately want **reference-conditioned training**.
That means teaching the model:
- how to use a separate reference clip for voice cloning
- how to follow reference style/prosody
- how to pick speaker identity from reference audio
This makes more sense for:
- reference-conditioned voice cloning behavior
- multi-speaker conditioning datasets
- style-following datasets
## How A Model Learns Voice Cloning
Think of it this way:
- if a TTS model only sees `audio + transcript`, it learns how to do TTS for the voices/styles present in the data
- if a TTS model also sees `ref_audio`, it can learn a different pattern:
- text says **what to say**
- reference audio says **whose voice/style to imitate**
- target audio is the correct output
That is how a model can learn **reference-conditioned voice cloning**, not just plain TTS.
So if your goal is to fine-tune that specific ability:
- “give the model a reference clip and make it speak in that voice/style”
then `ref_audio` is when this field makes sense.
If your goal is only:
- “make the model speak in this one fixed voice better”
then `ref_audio` usually does not matter much, and plain `audio + transcript` is the better starting point.
## `instruction` vs `ref_audio`
These are different tools.
### Use `instruction` for explicit text labels
Examples:
- `speaking angrily`
- `whispering`
- `calm narration`
This is usually clearer when you want to label the desired behavior directly.
### Use `quality` for recording/presentation quality
Examples:
- `telephone call quality`
- `studio recording`
- `noisy room recording`
This is better when the label is about how the audio sounds as a recording or presentation format, not about acting/performance.
### Use `ref_audio` for example audio conditioning
This is for:
- “sound like this other clip”
- “follow this speaker/style from audio”
So for many datasets:
- `instruction` is the simpler control
- `ref_audio` is the more advanced control
## Practical Recommendations
### Best first test
Use:
- staged clips
- correct transcripts
- no reference clips
- optional `instruction` only if clearly useful
In other words:
- `text_lines`: yes
- `reference_clip_lines`: leave blank
### If training a single person's voice
Start with:
- `audio + transcript`
Do **not** add `ref_audio` unless you specifically want to teach reference-conditioned cloning behavior.
### If training language adaptation like the Norwegian LoRA
Start with:
- `audio + transcript`
That is the normal base case.
You can still use reference audio later at inference time, because MOSS supports optional reference-based cloning, but the training dataset does not need to be built around `ref_audio`.
## Validation In Plain Terms
`📦 MOSS Dataset Prep` also mentions validation.
That means:
- a small holdout part of the dataset used only to check training progress
- not used to update weights
If you do not have a separate validation manifest:
- leave `validation_source` blank
- let the node auto-split the main dataset
## Short Version
If you want the least confusing starting point:
- use `🎞️ Training Clip Staging`
- use `🧾 MOSS Dataset Rows`
- fill only `text_lines`
- leave `reference_clip_lines` blank
- train the LoRA
That is the correct first path for most users.
+152
View File
@@ -0,0 +1,152 @@
# MOSS-TTS Prompt Fields Guide
## Overview
MOSS-TTS exposes official whole-segment prompt fields through the `⚙️ MOSS-TTS Engine` node.
These are not Step Audio EditX effects and they are not positional insertions. They are passed to the official MOSS prompt structure for the entire generated segment.
Supported official fields:
- `instruction`
- `quality`
- `sound_event`
- `ambient_sound`
- `language`
- `duration_tokens`
## Stability Note
`ambient_sound` should currently be treated as experimental on base `MOSS-TTS` inside this suite.
Reason:
- the official prompt schema exposes `ambient_sound`
- but the clearest public OpenMOSS example we found is in the `MOSS-SoundEffect` documentation path, not as a validated short-utterance base-TTS control
So on `MOSS-TTS` you may get:
- weak or missing ambience
- longer-than-expected generation
- unstable tails if the model is pushed to the token cap
## Important Limitation
MOSS does **not** treat these like exact inline insertions inside the sentence.
For example:
```text
[Alice|sound_event:Laughter] Hello there.
```
That means the whole segment is conditioned with the event. It is **not** positional the way Step Audio EditX paralinguistic tags are.
## Engine Fields
You can set official fields directly on the engine node:
- `instruction`: whole-segment speaking instruction
- `quality`: whole-segment quality/style hint
- `sound_event`: whole-segment event hint such as `Laughter`, `Sigh`, `Breathing`
- `ambient_sound`: whole-segment ambience hint such as `Rain`, `Crowd`, `Forest`
These apply to every generated segment that uses that engine config.
## Per-Segment Override Syntax
For per-segment overrides, use `[]` parameter switching.
Supported forms:
```text
[Narrator|instruction:Speak softly and calmly] Hello there.
[Narrator|quality:Studio recording] Hello there.
[Narrator|sound_event:Laughter] Hello there.
[Narrator|ambient_sound:Rain on window] Hello there.
```
Inside normal character tags, combine them like this:
```text
[Alice|instruction:Speak softly and calmly] Hello there.
[Bob|quality:Telephone call quality|ambient_sound:Office room tone] Hi.
[Alice|sound_event:Laughter] That's funny.
```
These are whole-segment overrides, not exact insertion points.
## Why Not `<>`
MOSS does not have true native inline positioning for these controls.
So `<>` should stay free for real inline post-processing tags such as Step Audio EditX effects. Using `<>` for MOSS prompt fields would blur two different systems and create confusion.
## Examples
### Direct engine fields
Set on the engine node:
- `sound_event = Laughter`
- `quality = Studio recording`
Text:
```text
[Alice] Hello there!
```
Result: the entire Alice segment is conditioned with `Laughter` and `Studio recording`.
### Per-segment `[]` syntax
```text
[Alice|sound_event:Laughter] Hello there!
[Bob|ambient_sound:Rain] Nice to meet you.
```
Result:
- Alice segment uses `sound_event = Laughter`
- Bob segment uses `ambient_sound = Rain`
## Conflicts and Precedence
- If you set an engine field and also override it in `[]`, the `[]` value wins for that segment.
- These MOSS prompt fields are not supposed to use `<>`.
## Native TTSD Dialogue Note
`MOSS-TTSD-v1.0` generates one native dialogue request across the formatted `[S1]...[S5]` conversation.
So these official fields still apply to the whole TTSD request, not to one precise speaker turn inside that request.
## Native TTSD Compatibility Rules
Native TTSD mode is strict in this suite.
If any of these are detected, generation **fails with an explicit error** instead of auto-switching models:
- `pause tags`
- inline Step Audio EditX tags
- per-segment `[]` parameter changes
- more than 5 speakers
This applies to both normal TTS Text and SRT workflows.
Required action when that happens:
- switch to `Custom Character Switching`
- choose a standard MOSS model (`MOSS-TTS-Local-Transformer` or `MOSS-TTS`)
## When To Use Something Else
If you need exact placement of laughter/breathing/cough inside the sentence, use:
- Step Audio EditX inline tags and audio editor workflow
- an engine with true native positional tag support, such as CosyVoice3
Use MOSS prompt fields when you want official MOSS conditioning, not precise insertion control.
If you specifically want generated environmental audio like rain, traffic, or scene beds, `MOSS-SoundEffect` is the more correct OpenMOSS model family target than base `MOSS-TTS`.
+204
View File
@@ -0,0 +1,204 @@
# 🏷️ Multiline TTS Tag Editor Guide
This guide covers the practical user-facing workflow for the `🏷️ Multiline TTS Tag Editor` node.
For related topics, also see:
- [Character Switching Guide](CHARACTER_SWITCHING_GUIDE.md)
- [Parameter Switching Guide](PARAMETER_SWITCHING_GUIDE.md)
- [Step Audio EditX Inline Tags User Guide](INLINE_EDIT_TAGS_USER_GUIDE.md)
- [Higgs Audio v3 Inline Tags](HIGGS_AUDIO_V3_INLINE_TAGS.md)
- [CosyVoice3 Tags Guide](COSYVOICE3_TAGS_GUIDE.md)
- [OmniVoice Native Tags Guide](OMNIVOICE_TAGS_GUIDE.md)
## What This Editor Is For
The editor is meant to make tag-heavy TTS text and SRT subtitle editing faster inside ComfyUI without manually rebuilding bracket syntax all the time.
It supports:
- Character switching tags
- Language switching tags
- Per-segment parameter overrides
- Engine-aware inline tags for Step Audio EditX, Higgs Audio v3, CosyVoice3, and OmniVoice
- Presets and edit history
- SRT-aware highlighting and timing editing
## Main Views
The top bar includes four views:
- `Editor` for direct text editing
- `History` for restoring earlier snapshots
- `Presets` for quick saved snippets or character setups
- `Library` for built-in reference material
## Basic Tag Editing
Use the left panel to insert or modify tags without writing every bracket manually.
Common examples:
```text
[Alice]
[en:Alice]
[Alice|seed:42|temperature:0.7]
[pause:1s]
```
Useful behavior:
- Character names are inserted at the caret or wrapped around the current selection
- Language and speaker can be combined in one tag
- Parameters can be stacked with `|`
- Keep pauses separate and before the tag they precede: `[pause:1s] [Alice|temperature:0.7]`
- `Format` moves a pause nested with character or parameter parts into that standalone form
- Presets can store either quick snippets or reusable speaker setups
## Inline Tags
The editor has an engine-aware `Inline Tags` panel.
Step Audio EditX examples:
```text
<Laughter>
<giggle>
<clears throat:2>
<Question-oh>
<emotion:happy>
<style:whisper:2>
<restore:1@2>
```
Higgs Audio v3 examples:
```text
<|emotion:amusement|>
<|style:whispering|>
<|prosody:pause|>
```
CosyVoice3 examples:
```text
<breath>
<laughing>that was funny</laughing>
```
OmniVoice examples:
```text
<laughter>
<sigh>
<question-ei>
```
Use the dedicated inline tag controls in the sidebar when you do not want to type these by hand.
The Step sound menu includes the original sounds plus the expanded sounds for the `2026-01-23` checkpoint: `inhale`, `exhale`, `laugh`, `chuckle`, `clears throat`, `snort`, `giggle`, `cough`, `breath`, `Surprise-yo`, `Question-ah`, `Question-en`, `Question-yi`, and `Question-oh`. Select `Step-Audio-EditX-2026-01-23` in the engine node to use these sounds. The `2025-11-28` checkpoint supports the original vocabulary; see the [checkpoint guide](STEP_AUDIO_EDITX_CHECKPOINTS.md).
Validation, highlighting, and the tag-swap menu recognize these sounds. Names are case-insensitive, and `<clears_throat>` is accepted as an alias for `<clears throat>`. The insertion controls preserve the existing iteration syntax, such as `<giggle:2>`, and pipe combinations, such as `<giggle:2|emotion:happy>`. The emotion menu also includes `fear` and `humour`.
The editor checks syntax for the chosen tag engine but does not inspect the engine node's selected model checkpoint. A valid new sound still requires the January weights.
Important differences:
- `Step Audio EditX` tags are post-process controls
- `Higgs Audio v3`, `CosyVoice3`, and `OmniVoice` tags are native generation controls
- `OmniVoice` editor insertion uses suite-default angle-tag aliases and the processor converts them internally to official native tags during generation
## SRT Editing
When the text looks like valid SRT, the editor highlights subtitle numbers and timings differently and enables subtitle-specific editing tools.
Example:
```srt
1
00:00:01,000 --> 00:00:04,000
[Alice] Hello there.
2
00:00:04,500 --> 00:00:08,000
[Bob] This is the next subtitle.
```
### Drag Timing Controls
Timing lines are directly draggable:
- Drag the left timestamp to change cue start
- Drag the right timestamp to change cue end
- Drag the arrow to move the whole cue while keeping its duration
Modifiers:
- Hold `Shift` while dragging start or end to move the neighboring cue boundary by the same delta and preserve the existing gap
- Hold `Alt` while dragging for finer timing adjustment
### Merge Cues
Sometimes TTS sounds better when two short subtitles are spoken as one longer chunk.
Use:
- `Alt+click` on a cue number to merge that cue with the next cue
- `Alt+Shift+click` on a cue number to merge that cue with the previous cue
Merge behavior:
- Keeps the first cue start time
- Keeps the second cue end time
- Combines the text into one subtitle body
- Renumbers the remaining cues
### Split Cues
Sometimes a subtitle is too long and you want to split it into two timed parts.
Use:
- Place the caret inside subtitle text
- Press `Ctrl+Shift+Enter`
Split behavior:
- Splits the cue at the caret position
- Estimates the new timing boundary from the text proportion on each side
- Gives a small bias to the left side when the split happens after punctuation
- Enforces a minimum duration for both resulting cues
- Renumbers following cues
## Shortcuts
Core shortcuts:
- `Ctrl+Z` or `Cmd+Z` = Undo
- `Ctrl+Shift+Z` / `Cmd+Shift+Z` or `Ctrl+Y` = Redo
- `Alt+Z` = Undo
- `Alt+Shift+Z` = Redo
- `Ctrl+Shift+Enter` = Split current SRT cue at caret
- `Alt+click` on cue number = Merge with next cue
- `Alt+Shift+click` on cue number = Merge with previous cue
Sidebar insertion shortcuts:
- `Alt+C` = Add character tag
- `Alt+L` = Add language tag
- `Alt+P` = Add parameter tag
- `Alt+1`, `Alt+2`, `Alt+3` = Load preset slot
## Editing Tips
- Use merge when subtitles are too short for natural speech flow
- Use split when a single subtitle is too dense or needs better pacing
- Keep SRT text semantically clean before generation so later timing edits remain predictable
- Use presets for repeated speaker or parameter combinations
- Use the `Library` tab as a quick reminder, not as the full manual
## Known Limits
- SRT merge and split are designed for standard subtitle blocks, not malformed subtitle text
- Split timing is estimated from text distribution, so it is a good first pass, not perfect forced alignment
- If you make large structural SRT edits, review the final timings before generation
@@ -0,0 +1,120 @@
# 01 - First Ask The LLM To Research The Official Model
Do this before implementation.
The official model implementation is the source of truth. Not this suite, not another ComfyUI node pack, not a random demo script. The LLM must understand what the model actually supports before it writes code.
## What The User Should Ask
Copy this to the LLM:
```text
Do not write code yet.
Research the official implementation for this model/engine:
[PASTE OFFICIAL REPO, MODEL CARD, PAPER, OR PACKAGE LINK HERE]
Produce a capability report for TTS Audio Suite. Read the official repo, model card, examples, docs, paper if useful, license, dependency files, and inference scripts.
The report must answer:
1. What task does this model actually support?
- TTS (in this suite, TTS means both Unified TTS Text and Unified SRT TTS)
- Voice Changer / voice conversion
- ASR
- Audio editing
- Other special feature
2. What native parameters does the official implementation expose?
3. What features are real native model capabilities?
4. What features are not supported natively?
5. What model files are required, and where do they come from?
6. What voice/reference input mode does it require?
- Reference audio only
- Reference audio plus exact transcript/reference text
- Either reference-audio-only or reference-audio-plus-text modes
- No reference audio
7. What sample rate, channel layout, and audio tensor/file format does it expect?
8. What dependencies and versions does it require?
9. Does it conflict with common packages already used by TTS Audio Suite?
10. What license applies to the code and weights?
11. Does it need only unified nodes, or does it need a special node?
12. Is this a short-form or long-form generation model?
- If long-form, what is the native segmentation strategy?
- Should chunking be text-length based, duration/minute based, or model-internal?
- Which controls should be ignored or overridden (for example, max chars per chunk) to match native behavior?
Do not start implementation until this report is complete.
```
## What The Capability Report Must Decide
The report must decide the actual scope:
- TTS integration, which includes both Unified TTS Text and Unified SRT TTS.
- Voice Changer / voice conversion.
- ASR / transcription.
- Special node for a native unique feature.
- No integration, if the model is not suitable.
Do not ask whether SRT is natively supported by the model. For TTS engines, SRT support is implemented by TTS Audio Suite by generating subtitle text segments and assembling timing with the suite timing systems.
If the model has a native feature that does not fit existing unified nodes, say so. Do not hide it inside a generic node if it needs a special UI.
Also decide the reference archetype early:
- Use Qwen3-TTS as default reference for standard TTS/SRT architecture.
- Use VibeVoice as reference when the target engine is long-form and needs duration-aware/minute-aware segmentation behavior.
## Native Features Come First
Implement native official features first.
Good native examples:
- Official speaker/reference audio support.
- Official language parameter.
- Official emotion/style controls.
- Official voice conversion mode.
- Official ASR timestamps.
Bad unsupported examples unless discussed first:
- Fake speed control by time-stretching output.
- Fake emotion by post-processing audio.
- Fake language dropdown when the model is English-only.
- Fake SRT timing controls that bypass the suite timing engine or imply unsupported model behavior.
Extra suite-added features can be useful, but they need maintainer discussion first.
## Minimum Research Checklist
The LLM should inspect:
- Official README.
- Model card.
- Inference examples.
- CLI or demo scripts.
- Python package metadata.
- Requirements files.
- Download instructions.
- License files.
- Hugging Face or ModelScope file layout.
- Open issues if the model has known broken behavior.
If any of these cannot be found, the report must say that clearly instead of guessing.
---
Navigation: [Guide Hub](README.md) | Next: [02 - Check Existing ComfyUI Implementations](02_CHECK_EXISTING_COMFYUI_IMPLEMENTATIONS.md)
@@ -0,0 +1,81 @@
# 02 - Check Existing ComfyUI Implementations
After official model research, check whether someone already made a ComfyUI implementation.
This is useful, but dangerous if followed blindly.
## What The User Should Ask
Copy this to the LLM:
```text
Search for existing ComfyUI implementations of this model.
If you find any, clone them into:
IgnoredForGitHubDocs/For_reference/[ENGINE_NAME]/
Study them as references only. Do not treat them as the source of truth. Compare every feature and parameter against the official model implementation.
Produce notes that answer:
1. Which ComfyUI implementations exist?
2. What files or patterns are useful as reference?
3. What dependencies or install problems do they reveal?
4. What UI controls do they expose?
5. Which controls are native official model features?
6. Which controls are artificial additions?
7. Which parts should not be copied into TTS Audio Suite?
```
## Where To Put References
Clone references here:
```text
IgnoredForGitHubDocs/For_reference/[ENGINE_NAME]/
```
Do not place reference repos in the main package folders. They are study material, not suite code.
## How To Use Them
Use existing ComfyUI implementations to learn:
- How the model loads in a ComfyUI environment.
- Which dependencies break easily.
- Which model files are expected.
- Which audio formats are accepted.
- Which UI patterns users may expect.
- Which edge cases others already hit.
But always compare back to the official implementation.
## What Not To Copy Blindly
Do not copy:
- Fake parameters that are not official model features.
- Random post-processing presented as native model behavior.
- Bad model download paths.
- Silent downloads into cache directories.
- Large engine logic stuffed directly into ComfyUI node UI code.
- Dependency pins that break other engines without analysis.
- UI controls that imply unsupported capabilities.
## Maintainer Discussion Required
If a ComfyUI implementation adds something useful but non-native, discuss it first in an issue or PR.
Examples:
- Artificial speed control.
- Post-processing emotion controls.
- Non-native denoise controls.
- Any special workflow that changes model behavior beyond official inference.
The default implementation should expose native model capabilities first.
---
Navigation: [Guide Hub](README.md) | Previous: [01 - Research The Official Model](01_FIRST_ASK_THE_LLM_TO_RESEARCH_THE_OFFICIAL_MODEL.md) | Next: [03 - Decide Engine Scope](03_DECIDE_ENGINE_SCOPE.md)
@@ -0,0 +1,103 @@
# 03 - Decide Engine Scope
Before coding, decide exactly what the integration should include.
This prevents half-integrations where basic TTS works but the required SRT path, cache, tags, VRAM, or docs are missing.
## What The User Should Ask
Copy this to the LLM:
```text
Using the official capability report and ComfyUI reference notes, decide the TTS Audio Suite scope for this engine.
Do not write code yet.
Produce a short scope document with:
1. Which node family should be supported?
- If this is TTS, include both Unified TTS Text and Unified SRT TTS.
2. Which native features will be implemented now?
3. Which native features will be intentionally skipped, and why?
4. Which non-native suite-added features are proposed, if any?
5. Which existing engine should be used as implementation reference?
6. What files/modules will likely need changes?
7. Does this engine stay in the Main Environment, or does it need runtime isolation?
8. What manual tests will prove the integration works?
```
## Runtime Decision Order
Test the official package with `--no-deps` in the main T5 environment first. Prefer a small compatibility patch when practical; use the existing shared T4 SHRED runtime only if T5 is genuinely incompatible. Never create another environment or download/reinstall Torch or Transformers without explicit maintainer approval.
## Node Types To Choose From
Decide whether the engine needs:
- TTS integration: Unified TTS Text plus Unified SRT TTS. These are a mandatory pair for TTS engines.
- Voice Changer / voice conversion.
- ASR Transcribe.
- A special feature node.
- No integration.
Do not add a special node just because it looks interesting. Add one only when the official model has a capability that does not fit the unified nodes.
## Native Features vs Suite Enhancements
Separate these clearly.
Native model features:
- Supported by official model code or docs.
- Should be implemented first when practical.
- Can be exposed in the engine node UI.
Suite-added enhancements:
- Added by TTS Audio Suite around the model.
- May be useful, but must be explicit.
- Need maintainer discussion if they change behavior or user expectations.
## Written Scope Required
The LLM should write a scope before implementation.
A good scope looks like this:
```text
Engine scope:
- Supports TTS: yes, with both Unified TTS Text and Unified SRT TTS
- SRT approach: generate each subtitle independently and assemble timing with AudioAssemblyEngine
- Supports Voice Changer: no, official model has no voice conversion mode
- Supports ASR: no
- Needs special node: yes, official model has native voice design from text description
Native parameters to expose:
- seed
- temperature
- top_p
- language
Do not expose:
- speed, because official model has no speed parameter
Primary reference:
- Qwen3-TTS for TTS/SRT architecture
Manual tests:
- TTS Text basic generation
- TTS Text with character tags
- TTS Text with pause tags
- SRT generation with interrupt test
- Clear VRAM then regenerate
Runtime plan:
- Main Environment only
- or Shared Runtime if dependency conflicts are expected
```
If the scope is vague, do not code yet.
---
Navigation: [Guide Hub](README.md) | Previous: [02 - Check Existing ComfyUI Implementations](02_CHECK_EXISTING_COMFYUI_IMPLEMENTATIONS.md) | Next: [04 - Implementation Order For LLM](04_IMPLEMENTATION_ORDER_FOR_LLM.md)
@@ -0,0 +1,186 @@
# 04 - Implementation Order For LLM
Use this order when the capability report and scope are complete.
Do not start from random files. Follow the architecture.
## What The User Should Ask
Copy this to the LLM:
```text
Implement this engine using the TTS Audio Suite architecture.
Before editing files, read:
- PROJECT_INDEX.md
- docs/New Engines Guides/NEW_ENGINE_IMPLEMENTATION_GUIDE.md
- docs/New Engines Guides/fails_to_avoid_TTS_Engine_Implementation.md
Use Qwen3-TTS as the primary modern reference for full TTS/SRT architecture and suite parity.
Use Step Audio EditX as a secondary reference for wrapper/model lifecycle and special post-processing patterns.
If another existing engine is closer to this model, inspect that too.
Do not copy any reference blindly. The official model implementation decides native capabilities.
```
## Reference Engines
Primary reference:
- `engines/adapters/qwen3_tts_adapter.py`
- `nodes/qwen3_tts/qwen3_tts_processor.py`
- `nodes/qwen3_tts/qwen3_tts_srt_processor.py`
- `nodes/engines/qwen3_tts_engine_node.py`
Use Qwen3-TTS for:
- Adapter/processor separation.
- Character handling.
- Pause tags.
- Segment parameters.
- Cache integration.
- SRT processor structure that reuses the main TTS processor for generation.
- Interrupt handling.
- Progress feedback.
- Console observability parity, such as input-text echo and useful live generation progress.
- Unified model loading.
Secondary reference:
- `engines/step_audio_editx/`
- `engines/adapters/step_audio_editx_adapter.py`
- `nodes/step_audio_editx/`
- `nodes/engines/step_audio_editx_engine_node.py`
Use Step Audio EditX for:
- Wrapper/model lifecycle patterns.
- Special post-processing behavior.
- Engine-specific extra node patterns.
- Model loading edge cases.
Long-form reference:
- `engines/vibevoice_engine/`
- `engines/adapters/vibevoice_adapter.py`
- `nodes/vibevoice/vibevoice_processor.py`
Use VibeVoice when the target engine is long-form and needs duration-aware segmentation behavior:
- Prefer model-appropriate duration/minute-based segmentation when required.
- Do not force short-form `max_chars_per_chunk` behavior if it conflicts with native long-form generation strategy.
- If the suite already provides chunk splitting / chunk combination controls in the unified node, reuse those. Do not add engine-local chunk-silence or chunk-combination controls unless the official model has a clearly separate native long-form mechanism.
## Implementation Order
Follow this order:
1. Confirm scope.
2. Verify model download layout.
3. Add or update downloader/model layout logic.
4. Build the engine wrapper around official inference.
5. Register the unified model factory.
6. Build the adapter.
7. Build the main processor.
8. For TTS engines, build the SRT processor too. The SRT processor should call the main TTS processor/generation path for each subtitle instead of duplicating generation logic. Build VC, ASR, or special processors only if scoped.
9. Add the engine configuration node UI.
10. Wire node and adapter registration.
11. Add segment parameter support if the engine has native parameters.
12. Add generated audio cache where applicable.
13. Add interrupt checks in long loops.
14. Add progress feedback for long generation.
15. Update docs/YAML metadata.
16. Run automated and live ComfyUI validation. Use FL-MCP-assisted validation when it is installed and connected; otherwise perform the same checks manually. Follow `tests/FL_MCP_VALIDATION.md`.
17. Run the required parity checklist.
## Live ComfyUI Validation Rule
Passing imports or pytest is not enough for a new engine. Validate it in the canonical Windows ComfyUI installation after implementation.
If [ComfyUI_FL-MCP](https://github.com/filliptm/ComfyUI_FL-MCP) is available, the LLM should use it to inspect and operate the live ComfyUI instance. Treat it as an optional test driver, not a project dependency and not a substitute for the existing test suite.
The LLM should:
- After changing Python code, restart the canonical Windows ComfyUI process before testing. An already-running process still has the old modules loaded.
- Use PowerShell to identify the process listening on port `8188`, verify its command line belongs to the canonical ComfyUI `main.py`, stop only that process, and relaunch it with the canonical Windows Python.
- Wait for `http://127.0.0.1:8188/system_stats` to respond before using FL-MCP.
- Refresh the existing ComfyUI browser tab after restart and confirm the FL-MCP browser bridge has reconnected before calling canvas-only tools.
- Confirm the engine node and the relevant unified node are registered.
- Load or construct the smallest useful workflow.
- Inspect workflow JSON for the expected node types, links, and widget values.
- Queue the workflow and wait for completion.
- Inspect execution history and report the full actionable error if execution fails.
- Confirm the expected audio output artifact exists.
- Capture a canvas screenshot for UI and broken-node inspection.
- Exercise TTS Text and SRT for every TTS engine, plus any other scoped capability.
- Record what was actually tested, what was skipped, and why.
Screenshots prove only visible workflow state. They do not prove that generation succeeded or that audio is correct. Execution history and output artifacts are required evidence, and the user must still judge subjective audio quality.
Do not install FL-MCP, alter its safety settings, or enable destructive tools unless the user authorizes it. If FL-MCP is unavailable, report that fact and follow the manual fallback in `tests/FL_MCP_VALIDATION.md`.
Repeat the edit, restart, reconnect, and validation cycle after every implementation fix that changes imported Python code. Frontend-only changes may require a hard browser refresh as well. Do not claim that a fix was tested against a ComfyUI process started before the fix was written.
## Architecture Rule
Unified nodes should stay thin.
Reference engines are examples only. Every engine must have dedicated processors and adapters; share only engine-neutral utilities.
Do not put hundreds of lines of engine-specific orchestration into:
- `nodes/unified/tts_text_node.py`
- `nodes/unified/tts_srt_node.py`
- `nodes/unified/voice_changer_node.py`
- `nodes/unified/asr_transcribe_node.py`
Engine-specific orchestration belongs in processors and adapters.
For TTS engines, SRT support is mandatory. It is the suite's subtitle-driven TTS flow, not a native capability the model has to expose.
## SRT Reuses TTS Generation
Follow the Qwen3-TTS pattern: the SRT processor is the timing/orchestration layer, not a second implementation of the engine.
The SRT processor should:
- Import or instantiate the engine's main TTS processor.
- Parse SRT subtitles and loop through subtitle entries.
- Build the subtitle-specific voice mapping and seed/parameter context.
- Call the main TTS processor, usually `processor.process_text(...)`, for each subtitle's text.
- Reuse the same adapter, cache keys, character handling, pause tags, parameter switching, progress hooks, and model lifecycle used by normal TTS Text generation.
- Assemble subtitle audio with the suite timing systems, such as `AudioAssemblyEngine`, after generation.
The SRT processor should not:
- Call the raw model directly through a separate generation path.
- Reimplement character switching, pause tags, parameter switching, cache, or reference-audio handling differently from the TTS processor.
- Add separate fake SRT-only parameters that do not exist in the normal TTS path or the suite timing layer.
## Chunking Rule
If an engine needs long-text chunking, prefer the suite chunking and shared chunk combiner unless the official model exposes a real native long-form/chunking system that is materially different.
Do not:
- Add duplicate engine-node controls for silence between chunks, chunk combination method, or similar suite-owned chunk-assembly behavior.
- Bypass the shared chunk combiner with a custom engine-local join path unless there is a documented official reason.
## Registration Areas To Check
The LLM should inspect and update the relevant registration files for the chosen scope:
- `nodes.py`
- `engines/adapters/__init__.py`
- `utils/models/unified_model_interface.py`
- `utils/models/engine_registry.py`
- `utils/text/segment_parameters.py`
- `docs/Dev reports/tts_audio_suite_engines.yaml`
- `docs/Dev reports/tts_audio_suite_aux_models.yaml` if helper models are added
Only update files that are actually needed for the scoped integration.
---
Navigation: [Guide Hub](README.md) | Previous: [03 - Decide Engine Scope](03_DECIDE_ENGINE_SCOPE.md) | Next: [05 - Required Parity Checklist](05_REQUIRED_PARITY_CHECKLIST.md)
@@ -0,0 +1,120 @@
# 05 - Required Parity Checklist
Paste this checklist to the LLM before review.
If the LLM cannot answer these questions, the implementation is not ready.
## Copy-Paste Checklist
```text
Review the new engine implementation against this TTS Audio Suite parity checklist.
Architecture:
- Did you use a processor instead of putting engine logic in unified nodes?
- Are unified nodes only doing thin routing/delegation?
- Is engine-specific orchestration in processors/adapters?
Model loading and lifecycle:
- Did you use unified_model_interface.load_model() in all model-loading paths?
- Does Clear VRAM / unload / reload work?
- Did you avoid __del__ destructors that auto-unload models after generation?
- Are model variants unloaded or reused correctly instead of accumulating VRAM?
Downloads and dependencies:
- Did you verify the real Hugging Face / ModelScope / official file layout?
- Do models download into organized ComfyUI/models/TTS/ folders?
- Did you prevent silent downloads into random cache folders?
- Did you document dependency conflicts or install.py changes?
- Did you test `--no-deps` on Main/T5 and simple patching before falling back to the shared T4 runtime?
- Did you avoid creating another environment or downloading/reinstalling Torch or Transformers without explicit maintainer approval?
- If runtime isolation is needed, did you document the default mode and the reason in YAML/README?
Audio format:
- Did you verify the model native sample rate?
- Does output use valid ComfyUI audio dict/tensor shape?
- Did you resample only when needed?
- Did you avoid pitch/speed bugs from wrong sample-rate metadata?
Generated audio cache:
- Did you add generated audio cache where applicable?
- Does the cache key include text, voice/reference audio identity, model variant, seed, and all behavior-affecting params?
- Did you test generating the same input twice and confirm cache behavior?
Text features:
- Did you wire character tags if this node supports text generation?
- Did you handle narrator fallback?
- Did you wire pause tags?
- Did you wire parameter switching for real native parameters?
- Did you avoid adding fake unsupported parameters?
- Did you remove misleading controls like language selection if unsupported?
- If the engine supports language control, did you normalize the user-facing language UI/logs honestly and map them back to official backend codes without breaking segment/alias language switching?
- Did you reuse the suite chunking/chunk-combination controls instead of inventing duplicate engine-local chunk UI or chunk-silence parameters?
SRT features:
- If this is a TTS engine, did you implement the SRT path?
- Did you add interrupt checks in long SRT loops?
- Does SRT use the shared timing/assembly/reporting systems correctly, using the real existing APIs instead of guessed helper names?
- Did you avoid unnecessary timing-mode restrictions?
- Does character switching work in SRT if text character tags are supported?
UX:
- Did you add progress feedback for long generation?
- Did you preserve useful console observability parity instead of making the engine run silently (for example input-text echo and live it/s/progress)?
- Are error messages clear when required inputs are missing?
- Are tooltips honest about limitations, license, VRAM, and expected text length?
Docs and metadata:
- Did you update YAML-backed engine metadata?
- Did you regenerate docs/tables if metadata changed?
- Did you document model paths and download sources?
- Did you add examples or notes for special native features?
- If the engine needs a secondary environment, does the engine table clearly show that?
Manual tests:
- Did TTS Text work?
- Did SRT TTS work for every TTS engine?
- Did Voice Changer work if scoped?
- Did ASR work if scoped?
- Did character tags work?
- Did pause tags work?
- Did parameter switching work?
- Did Clear VRAM then regenerate work, and did unload actually tear down runtime/cache state instead of only moving weights to CPU?
- Did interrupt/cancel work in long generation?
Live ComfyUI integration evidence:
- Was validation run in the canonical Windows ComfyUI environment?
- Was ComfyUI restarted after the final Python changes, and can you show that the tested process started after those edits?
- After restart, was the browser refreshed and the FL-MCP browser bridge confirmed connected before canvas checks?
- If FL-MCP was available, did you follow tests/FL_MCP_VALIDATION.md and identify which MCP checks were run?
- If FL-MCP was unavailable, did you perform and document the equivalent manual checks instead of claiming MCP validation?
- Are the engine node and relevant unified nodes present without import or registration errors?
- Does the saved workflow JSON contain the expected node types, links, and widget values?
- Does the workflow load without missing/broken node state?
- Was the workflow queued, and does execution history show successful completion or the full actionable error?
- Does each successful run produce the expected audio output artifact?
- Was a canvas screenshot captured for UI inspection without treating the screenshot as execution proof?
- Did you record tested, failed, and skipped cases, including the reason for every skip?
- Did a human assess subjective audio quality separately?
```
## Important Failures This Prevents
This checklist exists because working audio is not enough.
Common failures:
- Engine works once, then fails after ComfyUI unloads models.
- SRT generation cannot be cancelled.
- Cache is missing, so repeated generations waste time and VRAM.
- UI exposes fake language/speed/emotion controls.
- Reference audio uses the wrong sample rate and changes pitch.
- Character tags work in TTS Text but not SRT.
- Pause tags break because segments are handled incorrectly.
- Model files download to a hidden cache instead of `ComfyUI/models/TTS/`.
- Unified node gets filled with engine-specific code.
If any item fails, fix it before PR review.
---
Navigation: [Guide Hub](README.md) | Previous: [04 - Implementation Order For LLM](04_IMPLEMENTATION_ORDER_FOR_LLM.md) | Next: [06 - User Prompts To Copy Paste](06_USER_PROMPTS_TO_COPY_PASTE.md)
@@ -0,0 +1,162 @@
# 06 - User Prompts To Copy Paste
These prompts are for a user guiding an LLM.
Use them in order. Do not skip the research prompts.
## Prompt 1 - Official Model Research
```text
Do not write code yet.
I want to add this model/engine to TTS Audio Suite:
[PASTE OFFICIAL LINK]
Read the official repo, model card, docs, examples, license, dependency files, model download instructions, and inference scripts.
Produce a capability report for TTS Audio Suite. It must explain:
- What task the model supports: TTS, Voice Changer, ASR, audio editing, or special feature. If it supports TTS, TTS Audio Suite requires both Unified TTS Text and Unified SRT TTS.
- Native official parameters.
- Native official features.
- Unsupported features that should not be exposed as UI controls.
- Required model files and expected folder layout.
- Sample rate and audio format.
- Dependencies and likely conflicts.
- License restrictions.
- Whether this needs a special node beyond unified nodes.
Do not implement anything until this report is complete.
```
## Prompt 2 - Existing ComfyUI Implementations
```text
Search for existing ComfyUI implementations of this model.
If you find any, clone them into:
IgnoredForGitHubDocs/For_reference/[ENGINE_NAME]/
Study them only as references. Compare every feature against the official implementation.
Produce notes explaining:
- Which implementations exist.
- Which files/patterns are useful.
- Which dependencies or install issues they reveal.
- Which UI controls are native model features.
- Which UI controls are artificial or should not be copied.
- What should be discussed with the maintainer before adding.
Do not treat third-party ComfyUI code as the source of truth.
```
## Prompt 3 - Decide Scope
```text
Using the official capability report and ComfyUI reference notes, decide the integration scope.
Do not write code yet.
Create a scope summary with:
- Whether this is a TTS integration. If yes, include both Unified TTS Text and Unified SRT TTS.
- Whether to support Voice Changer.
- Whether to support ASR.
- Whether a special node is needed.
- Native parameters to expose.
- Native features to implement now.
- Native features intentionally skipped, with reasons.
- Non-native suite-added features proposed, if any.
- Existing engine references to inspect.
- Manual tests required.
```
## Prompt 4 - Implementation Plan
```text
Create an implementation plan for this engine in TTS Audio Suite.
Read first:
- PROJECT_INDEX.md
- docs/New Engines Guides/NEW_ENGINE_IMPLEMENTATION_GUIDE.md
- docs/New Engines Guides/fails_to_avoid_TTS_Engine_Implementation.md
Use Qwen3-TTS as the primary modern reference for full TTS/SRT architecture and suite parity.
Use Step Audio EditX as a secondary reference for wrapper/model lifecycle and special post-processing patterns.
Do not copy references blindly. The official implementation decides capabilities.
The plan must include files/modules to change, model download approach, model lifecycle approach, processor/adapter design, tag support, cache support, SRT support for TTS engines, VC/ASR support if scoped, docs updates, and manual tests.
```
## Prompt 5 - Implement First Phase
```text
Implement the first phase only.
Focus on:
- Model download/layout.
- Engine wrapper.
- Unified model factory registration.
- Adapter skeleton with real generation path.
- Engine node UI for native official parameters only.
Do not add fake parameters. Do not implement special non-native features unless they were explicitly approved.
After editing, summarize changed files and what still needs implementation.
```
## Prompt 6 - Review For Suite Parity
```text
Review this implementation against:
- docs/New Engines Guides/05_REQUIRED_PARITY_CHECKLIST.md
- docs/New Engines Guides/fails_to_avoid_TTS_Engine_Implementation.md
Be strict. Find missing architecture, cache, VRAM, interrupt, audio shape, sample rate, tag, SRT, docs, and fake-parameter issues.
Return findings ordered by severity with file references.
```
## Prompt 7 - Check Fake Or Non-Native Parameters
```text
Compare the engine UI and adapter parameters against the official model implementation.
List every exposed parameter and mark it as:
- Native official model parameter.
- Suite infrastructure parameter.
- Non-native enhancement.
- Unsupported/fake and should be removed.
If any non-native enhancement exists, explain whether it was approved by the maintainer.
```
## Prompt 8 - Prepare PR Summary
```text
Prepare a PR summary for this engine integration.
Include:
- Official model capability summary.
- Existing ComfyUI references studied.
- Supported node types.
- Native features implemented.
- Native features skipped, with reasons.
- Any non-native suite-added features, with approval note.
- Model download paths.
- Manual ComfyUI test results.
- Known limitations.
```
---
Navigation: [Guide Hub](README.md) | Previous: [05 - Required Parity Checklist](05_REQUIRED_PARITY_CHECKLIST.md) | Next: [07 - PR Review Checklist](07_PR_REVIEW_CHECKLIST.md)
@@ -0,0 +1,142 @@
# 07 - PR Review Checklist
Use this before merging a new engine PR.
A PR is not ready just because it generates audio.
## Required PR Evidence
The PR author or LLM should provide:
- Official model capability report.
- Existing ComfyUI reference notes, if any.
- Selected engine scope.
- Native parameters implemented.
- Native features intentionally skipped, with reasons.
- Non-native suite-added features, if any, with maintainer approval note.
- Model download path and file layout.
- Manual ComfyUI test results.
- Known limitations.
If this evidence is missing, ask for it before deep review.
## Architecture Review
Check:
- Unified nodes are thin.
- Engine-specific orchestration lives in processors/adapters.
- TTS Text processor exists for every TTS engine.
- SRT processor exists for every TTS engine. Missing SRT support requires an explicit maintainer-approved exception.
- VC or ASR processors follow existing suite patterns when scoped.
- Similar existing engines were used as references, not blindly copied.
Primary reference for modern full TTS/SRT integration:
- Qwen3-TTS.
Secondary reference for wrapper/lifecycle and special post-processing:
- Step Audio EditX.
## Model Lifecycle Review
Check:
- All model loading paths use `unified_model_interface.load_model()` unless there is a documented exception.
- ComfyUI unload/reload/Clear VRAM works.
- No `__del__` destructor unloads the model after generation.
- Device movement is handled safely.
- Quantized models are not moved with unsupported `.to()` calls.
- Multiple model variants do not accumulate in VRAM unnecessarily.
## Downloads And Dependencies Review
Check:
- Official model file layout was verified.
- Downloads go to organized `ComfyUI/models/TTS/` paths.
- No hidden or automatic downloads to random cache directories.
- `install.py` or requirements changes are justified.
- Dependency pins do not break existing engines.
- License restrictions are documented.
## Feature Review
Check:
- UI exposes native model parameters only, plus clearly documented suite infrastructure options.
- Unsupported controls are not present.
- Language controls are only present when official model supports language control.
- Character tags work if text generation is supported.
- Narrator fallback works.
- Pause tags work.
- Segment parameter switching works for real parameters.
- Inline edit/post-processing behavior respects segment boundaries if used.
- Special nodes are justified by native model capability.
## SRT Review
For every TTS engine, check the SRT path:
- Interrupt/cancel works in long loops.
- Timing modes use the shared timing/assembly systems correctly.
- There are no unnecessary restrictions on timing modes.
- Character switching works in SRT if character tags are supported.
- Pause tags and generated segment durations do not corrupt timing.
- Timing report/generation info is consistent with other engines.
## Cache And Progress Review
Check:
- Generated audio cache exists where applicable.
- Cache key includes all behavior-affecting parameters.
- Reference audio identity is included when relevant.
- Progress bars or progress messages exist for long generation.
- Interrupt checks are present inside long generation loops, not only before the loop.
## Audio Review
Check:
- Output tensor shape is valid for ComfyUI.
- Sample rate is correct and documented.
- Reference audio is resampled to the native rate where required.
- Audio is not accidentally squeezed from 3D to invalid 2D output.
- Pitch/speed are not changed by wrong sample-rate metadata.
## Documentation Review
Check:
- YAML-backed engine metadata is updated if engine docs changed.
- Generated docs/tables are regenerated when required.
- Model download sources are documented.
- Model layout docs are updated if needed.
- README or user docs mention important limitations.
- Tooltips are honest about license, VRAM, speed, and model limits.
## Manual Test Matrix
At minimum, test scoped features:
- Basic TTS Text generation.
- TTS Text with character tags.
- TTS Text with pause tags.
- TTS Text with parameter switching.
- For TTS engines, SRT generation.
- For TTS engines, SRT interrupt/cancel.
- Voice Changer if supported.
- ASR if supported.
- Clear VRAM / unload models, then generate again.
- Repeat identical generation to check cache behavior.
- Missing input error paths.
## Final Merge Rule
Do not merge until the implementation passes the required parity checklist or the remaining gaps are explicitly accepted by the maintainer.
---
Navigation: [Guide Hub](README.md) | Previous: [06 - User Prompts To Copy Paste](06_USER_PROMPTS_TO_COPY_PASTE.md)
@@ -308,11 +308,13 @@ from utils.downloads.unified_downloader import UnifiedDownloader
**File:** `engines/adapters/[engine_name]_adapter.py`
#### Step 4: Create Engine Configuration Node
**File:** `nodes/engines/[engine_name]_engine_node.py`
### Phase 2: Unified Systems Integration
#### Step 4: Create Engine Configuration Node
**File:** `nodes/engines/[engine_name]_engine_node.py`
**Model dropdown rule:** Always keep canonical/downloadable model choices visible and add detected installations as separate `local:ModelName` choices; selecting a canonical choice should reuse its organized local installation when available, not replace or hide either choice.
### Phase 2: Unified Systems Integration
#### Step 5: Integrate with Unified Model Loading
@@ -462,11 +464,12 @@ Also to test, requirements and dependencies need to be added.
- [ ] Character switching works with `[CharacterName] text`
- [ ] Language switching works (if applicable)
- [ ] Pause tags work with `[pause:1.5s]`
- [ ] Caching works (same input = cached output)
- [ ] Model auto-download works
- [ ] VRAM management works (model unloads)
- [ ] Different parameter combinations work
- [ ] **Interrupt handling works** - User can stop SRT generation and it stops within ~1 segment
- [ ] Caching works (same input = cached output)
- [ ] Model auto-download works
- [ ] VRAM management works (model unloads)
- [ ] Different parameter combinations work
- [ ] Engine prints a standard `Settings:` summary with the active generation/load parameters
- [ ] **Interrupt handling works** - User can stop SRT generation and it stops within ~1 segment
### Phase 4: SRT Implementation
@@ -645,4 +648,4 @@ Update the engines comparison table.
- Model management logic
- Audio format conversion utilities
---
---
+79
View File
@@ -0,0 +1,79 @@
# So You Want To Add A New Engine?
This folder is the start-here guide for adding a new engine to TTS Audio Suite with help from an LLM.
The goal is simple: do not let the LLM jump straight into coding. A new engine is not ready just because it can generate one audio clip. It must fit the suite architecture, model loading, cache, tags, SRT timing, docs, and ComfyUI lifecycle.
## Start Here
If you want support for a new TTS engine, Voice Changer, ASR, or special audio model, follow these guides in order.
For this suite, TTS support means both **Unified TTS Text** and **Unified SRT TTS**. SRT is our own text-to-subtitle generation flow around a TTS engine, not an optional native model feature.
1. [First Ask The LLM To Research The Official Model](01_FIRST_ASK_THE_LLM_TO_RESEARCH_THE_OFFICIAL_MODEL.md)
2. [Check Existing ComfyUI Implementations](02_CHECK_EXISTING_COMFYUI_IMPLEMENTATIONS.md)
3. [Decide Engine Scope](03_DECIDE_ENGINE_SCOPE.md)
4. [Implementation Order For LLM](04_IMPLEMENTATION_ORDER_FOR_LLM.md)
5. [Required Parity Checklist](05_REQUIRED_PARITY_CHECKLIST.md)
6. [User Prompts To Copy Paste](06_USER_PROMPTS_TO_COPY_PASTE.md)
7. [PR Review Checklist](07_PR_REVIEW_CHECKLIST.md)
Keep these technical references nearby:
- [Deep technical implementation guide](NEW_ENGINE_IMPLEMENTATION_GUIDE.md)
- [Known traps and failures](fails_to_avoid_TTS_Engine_Implementation.md)
## The Rule
The official model implementation is the source of truth.
Existing ComfyUI implementations are useful references, but they are not authorities. They may add fake or artificial features that the official model does not support. Do not copy those by default.
Example: if another implementation adds a `speed` parameter by stretching audio after generation, that is not a native model feature. It may still be useful, but it must be discussed with the maintainer first in an issue or PR.
## What To Tell Your LLM First
Copy this before asking for implementation:
```text
Do not write code yet.
I want to add a new engine to TTS Audio Suite. First, read these files:
- PROJECT_INDEX.md
- docs/New Engines Guides/README.md
- docs/New Engines Guides/01_FIRST_ASK_THE_LLM_TO_RESEARCH_THE_OFFICIAL_MODEL.md
- docs/New Engines Guides/02_CHECK_EXISTING_COMFYUI_IMPLEMENTATIONS.md
- docs/New Engines Guides/03_DECIDE_ENGINE_SCOPE.md
- docs/New Engines Guides/NEW_ENGINE_IMPLEMENTATION_GUIDE.md
- docs/New Engines Guides/fails_to_avoid_TTS_Engine_Implementation.md
Then research the official model implementation and produce a capability report. Do not start coding until the report explains the native model features, required nodes, model files, sample rate, dependencies, license, and supported parameters.
```
## Reference Engines
Use these references after the official model research is done:
- **Qwen3-TTS**: primary modern reference for full TTS/SRT architecture and suite parity.
- **Step Audio EditX**: secondary reference for wrapper/model lifecycle patterns and special post-processing cases.
Do not copy either blindly. The new engine must be scoped from the official implementation first.
## Definition Of Done
A new engine is not done until the answer to these questions is yes:
- Does it use the suite architecture instead of stuffing engine logic into unified nodes?
- Does it load models through the unified model interface?
- Does model unload/reload/Clear VRAM work?
- Did someone explicitly decide whether this engine belongs in the Main Environment or needs runtime isolation?
- Does it support generated audio cache where applicable?
- Does it handle ComfyUI audio shape and sample rate correctly?
- Does it wire character tags, pause tags, and parameter switching where applicable?
- If it is a TTS engine, does its SRT path support interrupt handling and correct timing modes?
- Does the UI avoid fake unsupported controls?
- Are docs and engine metadata updated?
- Did someone test it manually in ComfyUI?
If the answer is no, the PR is not ready.
@@ -56,11 +56,22 @@
- **Cache key pattern**: `audio_cache.generate_cache_key(engine_type, text=..., audio_component=..., **all_params)`
- **Duration calculation**: Update `_calculate_duration()` with engine sample rate (e.g., 24000 for Step Audio EditX, F5-TTS)
### Engine Settings Logging
- **Missing standard print**: New engines should print the usual `Settings:` summary once per run with the active generation/load parameters so validation can confirm what actually executed
- **Resolved prompt preview**: Reuse `utils.voice.character_logging.format_resolved_character_block()` for boxed text previews so logs show the voice that will actually speak, not only the parser alias
### Model Lifecycle - __del__ Destructor
- **CRITICAL**: Remove `__del__` from engine classes - causes automatic unload after generation ends (when object goes out of scope)
- **Pattern**: F5-TTS and ChatterBox don't have `__del__`, only IndexTTS and StepAudio did (wrong)
- **Correct behavior**: Models stay in VRAM for reuse; only unload on explicit Clear VRAM button, engine switch, or parameter change
### Clear VRAM - Runtime Teardown vs CPU Offload
- **Wrong**: Assuming every engine is safe to unload with only `.to('cpu')`
- **Symptom**: Clear VRAM frees only part of VRAM; most memory disappears only after closing ComfyUI
- **Cause**: Runtime-heavy engines can keep compiled graphs, prompt caches, workspaces, or runtime objects alive after CPU offload
- **Correct**: Add an engine-specific unload handler that clears internal caches, drops runtime/model references, then runs `gc.collect()`, `torch.cuda.empty_cache()`, and `torch.cuda.ipc_collect()` when available
- **Prevention**: If the engine uses warmup, `torch.compile`, streaming caches, or a custom runtime object, explicitly test Clear VRAM and reload instead of assuming generic offload parity
### Generation Info Reports - Modular Pattern
- **CRITICAL - Consistency required**: ALL engines must use `ChunkCombiner` + `ChunkTimingHelper` for generation reports
- **Correct pattern** (from ChatterBox 23-Lang, F5-TTS):
@@ -74,10 +85,27 @@
### Segment Parameter Registration
- **Forgot to add**: New engines to `PARAMETER_ENGINES` in `utils/text/segment_parameters.py` for `[seed:X]` tag support
### Language UI Normalization
- **Wrong**: Exposing raw backend language codes in the engine node just because the official runtime accepts them
- **Symptom**: New engine UI looks inconsistent with the suite, logs show backend codes instead of readable names, and character-tag / alias languages do not map cleanly to node config values
- **Correct**: Keep user-facing language controls normalized to honest display names when the suite already does that, then map them back to the official backend codes in the adapter/runtime layer
- **Prevention**: Reuse or add a small engine-local language helper so node UI, processor logs, segment language overrides, and adapter normalization all agree on the same mapping
### SRT Processor Import Pattern
- **Wrong**: Using `from nodes.xxx import` - fails with `'nodes' is not a package`
- **Correct**: Use `importlib.util.spec_from_file_location()` for cross-module imports in dynamically loaded processors
### SRT Shared API Guessing
- **Wrong**: Inventing shared helper names like `AudioAssemblyEngine.assemble(...)`, `TimingReportGenerator`, or `AdjustedSRTGenerator`
- **Correct**: Open a working SRT processor first and match the exact API that already exists in `utils/timing/assembly.py`, `utils/timing/engine.py`, and `utils/timing/reporting.py`
- **Real pattern**:
- `stretch_to_fit` → `TimedAudioAssembler.assemble_timed_audio(...)`
- `pad_with_silence` → `AudioAssemblyEngine(...).assemble_with_overlaps(...)`
- `concatenate` → `TimingEngine.calculate_concatenation_adjustments(...)` + `assemble_concatenation(...)`
- `smart_natural` → `TimingEngine.calculate_smart_timing_adjustments(...)` + `assemble_smart_natural(...)`
- reporting / adjusted SRT → `SRTReportGenerator().generate_timing_report(...)` and `generate_adjusted_srt_string(...)`
- **Prevention**: Never assume shared helper names. Grep existing SRT processors and copy the real call pattern exactly.
### ComfyUI Audio Dict Format
- **Wrong**: Passing `audio_tensor` directly to functions expecting raw tensor
- **Correct**: Extract waveform with `audio_tensor.get('waveform')` - ComfyUI audio is `{"waveform": tensor, "sample_rate": int}`
+73
View File
@@ -0,0 +1,73 @@
# OmniVoice Native Tags Guide
OmniVoice has its own official inline square-bracket control tokens upstream, but **this suite does not expose `[]` for OmniVoice tags**.
Inside TTS Audio Suite, you can write the suite-default angle-tag aliases in text:
```text
<laughter>
<sigh>
<question-ei>
```
The OmniVoice processor converts those aliases internally to the official OmniVoice native form before generation:
```text
[laughter]
[sigh]
[question-ei]
```
Do **not** type OmniVoice non-verbal tags in `[]` form in suite text. In this suite, `[]` belongs to character, language, parameter, and pause syntax.
## Supported Non-Verbal Tags
Official OmniVoice non-verbal meanings exposed by this suite through `<>` aliases:
```text
laughter
sigh
confirmation-en
question-en
question-ah
question-oh
question-ei
question-yi
surprise-ah
surprise-oh
surprise-wa
surprise-yo
dissatisfaction-hnn
```
Examples:
```text
[Alice] <laughter> You really got me there.
[Bob] <sigh> Fine, let's try again.
[Narrator] <question-ei> Really?
[Narrator] <surprise-oh> I didn't expect that.
```
## Important Behavior
- OmniVoice uses native generation tags here, not Step Audio EditX inline post-processing.
- User-facing suite syntax stays in `<>` form for OmniVoice non-verbal tags.
- `[]` is reserved for suite structural syntax like `[Alice]`, `[en:Alice]`, `[pause:1s]`, and parameter switching.
- Do not rely on Step-style `<Laughter:2>` or `<emotion:happy>` semantics in the OmniVoice text path.
- If you want Step Audio EditX as a second pass on OmniVoice output, use the separate `🎨 Audio Editor` node manually after generation.
## Multiline Tag Editor
The `🏷️ Multiline TTS Tag Editor` has a dedicated `OmniVoice` mode in the `Inline Tags` panel.
- The editor inserts suite-default angle-tag aliases like `<laughter>`
- The processor converts them internally to official OmniVoice square tags during generation
- The editor does not encourage raw OmniVoice `[]` input because `[]` is suite syntax
## Sources
This behavior follows the official OmniVoice documentation for non-verbal symbols and pronunciation control:
- [OmniVoice GitHub README](https://github.com/k2-fsa/OmniVoice)
- [OmniVoice Hugging Face model card](https://huggingface.co/k2-fsa/OmniVoice)
+131 -15
View File
@@ -2,7 +2,7 @@
## Overview
The TTS Audio Suite supports **per-segment parameter control** through inline tags. This allows you to override TTS generation parameters (seed, temperature, cfg, speed, etc.) on a per-segment basis, providing fine-grained control over individual audio segments without modifying node-level defaults.
The TTS Audio Suite supports **per-segment parameter control** through bracket tags like `[Alice|seed:42|temp:0.5]`. This allows you to override TTS generation parameters (seed, temperature, cfg, speed, etc.) on a per-segment basis, providing fine-grained control over individual audio segments without modifying node-level defaults.
Parameters are applied **only to the current segment** and automatically revert to node defaults for subsequent segments.
@@ -75,13 +75,45 @@ Parameters are applied **only to the current segment** and automatically revert
| `seed` | — | int | 0-4294967295 | Random seed for reproducible generation |
| `temperature` | `temp` | float | 0.1-2.0 | Randomness/creativity (lower=more deterministic) |
### Engine-Specific Parameters
#### ChatterBox & ChatterBox Official 23-Lang
| Parameter | Alias | Type | Range | Description |
|-----------|-------|------|-------|-------------|
| `cfg` | `cfg_weight` | float | 0.0-1.0 | CFG weight for speaker guidance |
| `exaggeration` | `exag` | float | 0.0-2.0 | Voice emotion exaggeration |
### Engine-Specific Parameters
#### MOSS-TTS
| Parameter | Alias | Type | Range | Description |
|-----------|-------|------|-------|-------------|
| `top_p` | `topp` | float | 0.0-1.0 | Nucleus sampling probability |
| `top_k` | `topk` | int | 1-100 | Top-k sampling |
| `audio_temperature` | `audio_temp` | float | 0.1-2.5 | MOSS audio sampling temperature |
| `audio_top_p` | — | float | 0.0-1.0 | MOSS audio nucleus sampling |
| `audio_top_k` | — | int | 1-200 | MOSS audio top-k sampling |
| `repetition_penalty` | `rep_penalty` | float | 0.5-3.0 | MOSS repetition penalty |
| `audio_repetition_penalty` | `audio_rep_penalty` | float | 0.5-3.0 | MOSS audio repetition penalty |
| `duration_tokens` | `tokens` | int | 0-8192 | MOSS target output length hint in audio tokens |
| `max_new_tokens` | — | int | 64-16384 | Hard generation token limit |
| `n_vq_for_inference` | `n_vq` | int | 0-32 | Local 1.7B only: RVQ layers/codebooks for inference |
| `instruction` | — | string | text | Whole-segment speaking instruction |
| `quality` | — | string | text | Whole-segment recording/presentation quality hint |
| `sound_event` | — | string | text | Whole-segment sound event hint |
| `ambient_sound` | — | string | text | Whole-segment ambient sound hint |
#### MOSS Sound Effects
MOSS-SoundEffect v1 uses the applicable MOSS-TTS parameters above. Both sound-effect engines also support a duration override:
| Parameter | Alias | Engines | Type | Range | Description |
|-----------|-------|---------|------|-------|-------------|
| `duration_seconds` | `seconds` | v1, v2 | float | 0.5-300 | Duration of the sound segment |
| `inference_steps` | `steps` | v2 | int | 1-150 | Diffusion steps |
| `cfg` | — | v2 | float | 0.0-20.0 | Prompt guidance strength |
| `sigma_shift` | — | v2 | float | 0.0-10.0 | Flow-matching schedule shift |
| `negative_prompt` | `negative`, `neg` | v2 | string | text | Sounds or qualities to discourage |
See the [Sound Effects Guide](SOUND_EFFECTS_GUIDE.md) for pauses, crossfades, long-duration chunking, and complete examples.
#### ChatterBox & ChatterBox Official 23-Lang
| Parameter | Alias | Type | Range | Description |
|-----------|-------|------|-------|-------------|
| `cfg` | `cfg_weight` | float | 0.0-1.0 | CFG weight for speaker guidance |
| `exaggeration` | `exag` | float | 0.0-2.0 | Voice emotion exaggeration |
#### F5-TTS
| Parameter | Alias | Type | Range | Description |
@@ -103,27 +135,111 @@ Parameters are applied **only to the current segment** and automatically revert
| `top_p` | `topp` | float | 0.0-1.0 | Nucleus sampling probability |
| `inference_steps` | `steps` | int | 1-100 | Number of inference steps |
#### IndexTTS-2
#### IndexTTS 2 / 2.5
| Parameter | Alias | Type | Range | Description |
|-----------|-------|------|-------|-------------|
| `cfg` | — | float | 0.0-20.0 | CFG strength |
| `top_p` | `topp` | float | 0.0-1.0 | Nucleus sampling probability |
| `top_k` | `topk` | int | 1-100 | Top-k sampling |
| `emotion_alpha` | — | float | 0.0-1.0 | Emotion control strength |
| `emotion_alpha` | — | float | 0.0-1.0 | Shared audio/vector/text emotion intensity |
| `duration_factor` | `dur_factor` | float | 0.5-2.0 | Official IndexTTS-2.5 internal feature-duration scaling; 0.5 shorter/faster, 2.0 longer/slower |
`duration_factor` is a 2.5-only upstream parameter. It uses nearest-neighbor scaling inside the semantic length regulator after speech codes are generated. It is not natural prosody planning, exact-seconds targeting, waveform playback-speed control, or an inference-performance control. IndexTTS continues to use the suite's ordinary final timing modes in TTS SRT.
Switching the engine node between IndexTTS-2 and IndexTTS-2.5 invalidates the cached Text/SRT processor and model identity. `language`, `duration_factor`, and `text_normalization` also participate in the generated-audio cache identity, so changing a supported 2.5 generation parameter cannot return audio produced with the previous setting.
IndexTTS-2 also supports inline emotion controls. Named unsigned values replace
that dimension; explicitly signed values adjust the connected vector:
```text
[sad:0.7|calm:0.2] Absolute values for this segment.
[sad:+0.3|calm:-0.2] Adjust the connected vector for this segment.
```
All eight values can be supplied in the official order `happy, angry, sad,
afraid, disgusted, melancholic, surprised, calm`:
```text
[vector:0,0,0.7,0,0,0.4,0,0.2] Absolute replacement.
[vector:+0,+0,+0.3,+0,+0,+0,+0,-0.2] Relative adjustment.
```
Full relative vectors require an explicit sign on every value. Results are
clamped to IndexTTS-2's supported range and revert to the connected vector at
the next segment.
Text emotion can use a saved preset or quoted text. `{seg}` is expanded with
the current segment before QwenEmotion analysis:
```text
[emotion:restrained_anger] A saved preset.
[emotion:"Quiet grief masking frustration"] A direct description.
[emotion:"Infer nervous anticipation from this line: {seg}"] Dynamic analysis.
```
Click a numeric emotion tag in the TTS Tag Editor to open a contextual radar
directly beside that tag. The editor also creates and manages text
presets in `models/TTS/IndexTTS/emotion_presets.json`.
IndexTTS-2 has separate engine inputs for these sources: connect vector or text
emotion to `emotion_control` and audio emotion references to `emotion_audio`.
Both may be connected simultaneously; IndexTTS-2 blends them during emotion
conditioning. Inline vector/text controls override the connected vector/text
values for their segment, while `[Character:emotion_ref]` selects a
segment-local audio reference that can still blend with vector/text emotion.
When an emotion control is inserted with the caret inside a character/audio tag,
the editor appends or updates it as another pipe parameter, for example
`[Bob:br_ivan_raiva3|sad:+0.25]`.
The tag editor's quick-swap palette is engine-aware: parameter choices are
filtered to the selected inline engine, while each engine's supported native
emotion/style/prosody/sound tags use their own replacement choices. Named
emotion presets can be swapped from the text; quoted `[emotion:"..."]` text is
intentionally left as direct editable content rather than treated as a preset.
---
## 🎯 Advanced Usage
### Combining with Character Switching
Parameters work seamlessly with character and language tags:
### Combining with Character Switching
Parameters work seamlessly with character and language tags:
```
[de:Alice|seed:42] German Alice with specific seed.
[fr:Bob|temperature:0.7] French Bob with higher temperature.
[seed:123|de:Alice] Order-independent syntax.
```
[seed:123|de:Alice] Order-independent syntax.
```
### MOSS Whole-Segment Prompt Fields
MOSS official prompt fields are better treated as per-segment parameters, not inline `<>` tags:
```
[Alice|instruction:Speak softly and calmly] Hello there.
[Bob|quality:Telephone call quality|ambient_sound:Office room tone] Can you hear me?
[Alice|sound_event:Laughter] That's actually funny.
```
Important:
- These are whole-segment controls
- They are not positional inline effects
- Keep `<>` free for true inline post-processing tags like Step Audio EditX
### DramaBox Prompt Templates
`prompt_template` (alias `template`) applies a `{seg}` wrapper and enables
templating for that segment automatically:
```text
[Narrator|template:A woman whispers, "{seg}"] This line is whispered.
[Narrator] This line returns to the DramaBox engine-node settings.
```
The template should include `{seg}`. If it is omitted, DramaBox warns once and
appends `"{seg}"` automatically. A separate inline enable parameter is not
required.
### Per-Segment Fine-Tuning in SRT
+113
View File
@@ -0,0 +1,113 @@
# 🌩️ Sound Effects Guide
The `🌩️ Sound Effects` node generates non-speech audio from a written description. It works with any connected engine that advertises sound-effect support.
## Engines
| Model | Engine node | Notes |
|---|---|---|
| MOSS-SoundEffect v1 | `⚙️ MOSS-TTS Engine` | Autoregressive MOSS 8B model |
| MOSS-SoundEffect v2 | `⚙️ MOSS SoundEffect v2 Engine` | 48 kHz diffusion model; up to 30 seconds per native generation |
Connecting a speech-only engine stops with a user-facing compatibility error.
## Basic workflow
1. Select a sound-effect model on its engine node.
2. Connect the engine to `🌩️ Sound Effects`.
3. Describe the sound rather than words to be spoken.
4. Choose the duration and seed, then queue the workflow.
Example:
```text
Heavy rain hitting a metal rooftop, distant rolling thunder, occasional wind gusts.
```
Descriptions generally work best when they state the source, environment, distance, texture, and progression of the sound.
## Timeline segments
Separate descriptions with parameter tags to generate multiple segments and concatenate them:
```text
[seconds:4|seed:42] Bright application startup chime. [seconds:2|cfg:5] Low, dark shutdown tone.
```
`duration_seconds` is the default duration for every generated segment. `[seconds:X]` overrides it for the following segment.
Newlines also create segments, as they do in TTS. They are optional because a parameter tag can start another segment on the same line.
## Pauses
Use a standalone pause tag to insert exact silence:
```text
[seconds:4] Startup chime. [pause:1.2] [seconds:2] Shutdown tone.
```
The aliases `[wait:X]` and `[stop:X]` are also accepted. Durations may use seconds or milliseconds:
```text
[wait:500ms]
```
Keep pauses separate from parameter tags:
```text
[pause:1.2] [cfg:7.5] Thunder crack.
```
Do not combine them as `[pause:1.2|cfg:7.5]`.
## Crossfade and long sounds
`crossfade_seconds` overlaps adjacent generated segments to soften their join. Set it to `0` for a hard join.
A pause creates an exact silent boundary, so crossfade is not applied across that pause.
MOSS-SoundEffect v2 has a native 30-second generation limit. Longer requested segments are generated as overlapping chunks, joined with the selected crossfade, and trimmed to the requested duration.
## Per-segment parameters
Common parameters:
| Tag | Engines | Purpose |
|---|---|---|
| `seed` | v1, v2 | Generated variation |
| `seconds` / `duration_seconds` | v1, v2 | Segment duration |
| `temperature` | v1 | Sampling randomness |
| `top_p`, `top_k` | v1 | Sampling limits |
| `repetition_penalty` | v1 | Discourage repetition |
| `duration_tokens` | v1 | Native duration-token control |
| `max_new_tokens` | v1 | Generation token limit |
| `steps` / `inference_steps` | v2 | Diffusion steps |
| `cfg` | v2 | Prompt guidance strength |
| `sigma_shift` | v2 | Flow-matching schedule shift |
| `negative_prompt`, `negative`, `neg` | v2 | Sounds or qualities to discourage |
Example using a negative prompt:
```text
[seconds:8|cfg:5|neg:speech, music] Dense forest ambience with insects and distant birds.
```
Unsupported parameters are ignored with a warning rather than being sent blindly to the engine.
## Seed and cache behavior
- `seed: 0` chooses a random seed.
- Reusing a positive seed with identical settings makes the request repeatable.
- With audio caching enabled, an identical segment and configuration can reuse its generated audio.
- Changing the description, seed, duration, engine configuration, or inline parameters invalidates that cached result.
## MOSS-SoundEffect v2 first-run compilation
The v2 DiT uses `torch.compile`. Its first generation may spend several minutes compiling before progress begins. Compatible compilation artifacts are cached and can be reused across later ComfyUI sessions.
This compile delay is separate from model downloading and normal generation time.
## Related guides
- [Per-Segment Parameter Switching](PARAMETER_SWITCHING_GUIDE.md)
- [Multiline TTS Tag Editor](MULTILINE_TTS_TAG_EDITOR_GUIDE.md)
+51
View File
@@ -0,0 +1,51 @@
# Step Audio EditX checkpoints
Choose the checkpoint in the existing **model_path** dropdown of **⚙️ Step Audio EditX Engine**. No widget was added or reordered, so saved device, precision, quantization, generation, and runtime values keep their positions.
| Model choice | Checkpoint | Download revision |
|---|---|---|
| `Step-Audio-EditX-2026-01-23` | Updated weights and expanded sound tags; default for new engine nodes | `5fe2f8a05c2353301ad47d3c1747b262115da138` |
| `Step-Audio-EditX-2025-11-28` | Legacy weights | `7f3de603ae46c96dff6f06f47b1d5a45aabd34fe` |
| `Step-Audio-EditX` | Compatibility choice for an existing unversioned installation | Existing files are retained; a missing installation downloads the January checkpoint |
| `local:...` | Explicitly selected local model | Existing files are retained |
Dates identify the weight uploads. The January release was announced on January 29; its pinned revision also includes subsequent configuration fixes. These are versions of the same 3B model architecture, not official “v1/v2” names.
Sources: [official model](https://huggingface.co/stepfun-ai/Step-Audio-EditX), [November snapshot](https://huggingface.co/stepfun-ai/Step-Audio-EditX/tree/7f3de603ae46c96dff6f06f47b1d5a45aabd34fe), [upstream release notes](https://github.com/stepfun-ai/Step-Audio-EditX#readme).
## Existing workflows and storage
- Existing `Step-Audio-EditX` and `local:...` selections remain valid. The suite does not replace their existing weights, rename their folders, or infer their checkpoint. The existing compatibility normalization of `config.json` still applies.
- Dated models download lazily into separate folders under `ComfyUI/models/TTS/step_audio_editx/`, respecting `extra_model_paths.yaml`.
- Selecting both checkpoints stores both sets of model files. The second LLM checkpoint requires approximately another 7 GB.
- Model and edit-result caches distinguish the selected checkpoint.
- The shared Transformers 4 runtime remains recommended. Selecting the updated standard checkpoint does not switch to vLLM or to the separate upstream AWQ checkpoint. The suite's `int4` option remains bitsandbytes NF4.
## Sound tags
Use the **2026-01-23** checkpoint for expanded sound tags. The November checkpoint is available to preserve older behavior; selecting it does not add the new learned sounds.
Existing spellings remain supported:
`<Breathing>`, `<Laughter>`, `<Sigh>`, `<Uhm>`, `<Surprise-oh>`, `<Surprise-ah>`, `<Surprise-wa>`, `<Confirmation-en>`, `<Question-ei>`, `<Dissatisfaction-hnn>`.
The January vocabulary also exposes:
`<inhale>`, `<exhale>`, `<laugh>`, `<chuckle>`, `<clears throat>`, `<snort>`, `<giggle>`, `<cough>`, `<breath>`, `<Surprise-yo>`, `<Question-ah>`, `<Question-en>`, `<Question-yi>`, `<Question-oh>`.
Input is case-insensitive. `<clears_throat>` is an alias for `<clears throat>`; both become `[clears throat]` in the model instruction. Expanded tags support the existing iteration and pipe syntax, for example:
```text
[Alice] That was funny <giggle:2>.
[Bob] Let me explain <clears_throat:1|style:serious>.
```
For ChatterBox v2/v3, use an explicit iteration to request Step editing: `<giggle:1>`. Bare `<giggle>`, `<inhale>`, `<exhale>`, and `<cough>` retain their ChatterBox-native meaning. CosyVoice's native sound tags remain native as well.
## Audio Editor and other engines
Connect the dated **Step Audio EditX Engine** output to **🎨 Step Audio EditX - Audio Editor** to choose a checkpoint explicitly. In the Audio Editor's transcript, use bare sound tags such as `<giggle>` or `<clears throat>`; its **n_edit_iterations** input controls the number of passes.
Without a connected engine, the Audio Editor prefers an installed January checkpoint, then an existing unversioned installation. If neither exists, it downloads the dated January checkpoint. This same fallback applies to automatic Step editing of other engines' audio. An existing unversioned installation may contain older weights; use the manual Audio Editor with a dated engine for explicit control.
See [the inline tag guide](INLINE_EDIT_TAGS_USER_GUIDE.md) for position, iteration, restoration, language, and duration behavior.
+47
View File
@@ -0,0 +1,47 @@
# Text Chunking Guide
TTS Audio Suite supports long text generation through the shared unified chunking system.
## What It Does
- splits long text into smaller generation chunks
- preserves sentence boundaries when possible
- can fall back to comma-based splitting for very long sentences
- combines generated audio using the suite chunk combiner
## Main Controls
- `enable_chunking`
- turns chunking on or off
- `max_chars_per_chunk`
- character limit per chunk
- `chunk_combination_method`
- `auto`
- `concatenate`
- `silence_padding`
- `crossfade`
- `silence_between_chunks_ms`
- silence inserted when the selected combination method uses silence padding
## Auto Combination Behavior
When `chunk_combination_method=auto`, the suite picks a join strategy based on text size:
- longer text tends toward `silence_padding`
- medium text tends toward `crossfade`
- shorter chunked text tends toward `concatenate`
The exact choice is made by the shared chunk combiner, not by engine-local custom logic.
## Important Architecture Rule
Chunking is a suite feature.
- new engines should reuse the shared chunking and chunk-combination controls
- engines should not invent duplicate local chunk-silence or chunk-join settings unless the official model has a real distinct native long-form system
## Related Docs
- [Example Workflows](../README.md#-example-workflows)
- [Character Switching Guide](./CHARACTER_SWITCHING_GUIDE.md)
- [Multiline TTS Tag Editor Guide](./MULTILINE_TTS_TAG_EDITOR_GUIDE.md)
@@ -11,6 +11,33 @@ This document tracks updates applied to our bundled IndexTTS-2 code from the ups
---
## 2026-08-11: IndexTTS-2.5 Version Integration
**Official sources:** `index-tts/index-tts` commit `b5ea881bec284b72f0b1cc04e0a724ff0c6b93e9`; model snapshot `ba2480d9f7f629eb18f6acaebb357679d9ba88a4`
### Changes applied
- Added IndexTTS-2.5 as a selectable version of the existing `index_tts` engine.
- Bundled the official 25 Hz semantic codec, multilingual tokenizer, Japanese G2P, and NeMo normalization bridge.
- Preserved suite dual-source audio plus vector/text emotion blending.
- Added Chinese, English, Japanese, Spanish, and Arabic conditioning.
- Added the official 2.5-only `duration_factor`, documented honestly as nearest-neighbor internal semantic-feature scaling rather than natural prosody or exact-duration planning.
- Deliberately excluded IndexTTS-2.5 from TTS SRT's native-duration option; the suite-owned exact-seconds extrapolation was removed after source and listening review.
- Kept legacy IndexTTS-2 checkpoints, FP16 loading, MaskGCT, workflows, and node identity intact.
- Pinned the audited Hugging Face model revision and retained the main Transformers 5 environment.
- Added model-aware Text/SRT processor and audio-cache identities so switching 2.0/2.5 or a 2.5 generation parameter cannot reuse stale output.
- Documented the suite's manual finding that 2.5 is not a universal cloning-quality upgrade: 2.0 may retain speaker resemblance better under strong different-speaker emotion transfer.
### Validation status
- [x] Python compilation
- [x] Bundled backend import under `TTS_SUITE_TEST_VENV_PYTHON`
- [x] Full checkpoint download and live ComfyUI generation
- [x] Manual audio-quality review of the official duration factor and 2.0/2.5 speaker resemblance
- [x] Live 2.5 → 2.0 model switching after processor-cache invalidation fix
---
## 2025-09-18: Major Update - Cache & Emotion Improvements
**Reference commit range:** `8336824..64cb31a` (September 11 → September 18, 2025)
@@ -207,6 +234,47 @@ This document tracks updates applied to our bundled IndexTTS-2 code from the ups
---
---
## 2026-07-11: Upstream Audit Before Release
**Upstream repository checked:** `index-tts/index-tts` (`main`)
**Upstream head observed:** `b5bd657` (2026-07-08)
**Check performed:** 2026-07-11
### Relevant upstream changes reviewed
| Commit | Upstream change | Bundled status | Release action |
|---|---|---|---|
| `843972e` | Coerce QwenEmotion JSON emotion scores to `float` and reject non-numeric values clearly | **Applied** in bundled `clamp_score()` | Run a text-emotion generation with numeric-string classifier output |
| `b154a1b` | WebUI text/vector preset save/load management | **Already covered differently** by the ComfyUI-native preset manager and `emotion_presets.json` integration | No direct port needed |
| `b5bd657` | Expose `--accel` and `--torch-compile`, add optional acceleration extras and WebUI settings | **Applied selectively**; suite now forwards `use_accel` into the bundled GPT path, while retaining ComfyUI-owned dependency handling | Validate acceleration fallback on compatible and non-accelerated setups |
| `7264ce2` | Improve IndexTTS-2 model resource checks and HF cache handling | **Suite-owned downloader differs** and needs a separate comparison if download failures are reported | No blind copy into bundled code |
### Findings
- No upstream change was found that invalidates the current eight-emotion vector order,
Qwen text-emotion syntax, audio-reference blending, or the suite's inline tag format.
- The upstream QwenEmotion string-score fix is directly relevant to the suite's text-emotion
path and should be applied before a release.
- The upstream acceleration work is not a drop-in replacement because this repository
bundles and adapts IndexTTS-2. The suite already contains the acceleration modules, but
`utils/models/unified_model_interface.py` should be checked so `use_accel` reaches the
bundled `IndexTTS2` constructor.
- Upstream WebUI presets are not copied verbatim: the suite's ComfyUI editor has a richer
vector/radar, inline-tag, sidebar, and filesystem preset implementation.
### Release follow-up checklist
- [x] Apply the upstream `clamp_score()` numeric coercion.
- [x] Pass `use_accel` through the unified IndexTTS-2 factory; validate fallback behavior.
- [ ] Run a Qwen text-emotion generation using numeric-string classifier output.
- [ ] Verify acceleration flags on a compatible CUDA installation and on a setup without
optional acceleration dependencies.
- [ ] Recheck bundled model-resource validation against the current upstream `check` logic.
---
## Next Update Check: 2025-11-20
**Monitoring:** Watch for commits to `indextts/infer_v2.py` in https://github.com/index-tts/index-tts
@@ -222,4 +290,4 @@ git log --oneline 1d5d079..HEAD -- indextts/infer_v2.py
- Additional performance optimizations
- Bug fixes in new acceleration code
- Breaking API changes
- Model loading improvements
- Model loading improvements
+48 -24
View File
@@ -4,24 +4,32 @@ This document explains how to enable torch.compile optimizations for Qwen3-TTS t
## Requirements
**CRITICAL**: torch.compile optimizations require specific versions tested with the reference implementation:
- **PyTorch 2.10.0+** with CUDA 13.0 support
- **triton-windows 3.6.0+** (Windows) or **triton 3.6.0+** (Linux)
- Python 3.12 (recommended)
**CRITICAL**: torch.compile optimizations require specific versions tested with the reference implementation:
- **PyTorch 2.10.0+** with CUDA 13.0 support
- **triton-windows 3.6.0+** (Windows) or **triton 3.6.0+** (Linux)
- **Visual Studio C++ Build Tools** on Windows (a real `cl.exe` toolchain, not just stale PATH entries)
- Python 3.12 (recommended)
**Note**: PyTorch 2.8.x and 2.9.x have compatibility issues with triton on Windows. Only PyTorch 2.10+ is supported.
### Installation
#### Windows (Tested Configuration)
```bash
# Upgrade PyTorch to 2.10 with CUDA 13.0
pip install --upgrade torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130
# Install triton-windows 3.6.x
pip install -U "triton-windows<3.7"
```
#### Windows (Tested Configuration)
```bash
# Upgrade PyTorch to 2.10 with CUDA 13.0
pip install --upgrade torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130
# Install triton-windows 3.6.x
pip install -U "triton-windows<3.7"
```
Also install **Visual Studio Build Tools** with the C++ toolchain:
- Workload: `Desktop development with C++`
- Or at minimum the MSVC x64/x86 build tools component
If `cl.exe` is missing, `torch.compile` in the isolated Qwen runtime will fall back to the non-compiled path.
**Verification:**
```bash
@@ -46,7 +54,12 @@ python -c "import triton; print(triton.__version__)"
## Environment Configuration
**No special configuration needed** - PyTorch 2.10 works with the default cudaMallocAsync allocator when using `compile_mode="default"`.
**Main environment**: no special configuration beyond a working compiler toolchain.
**Shared Runtime isolation**:
- TTS Audio Suite now tries to detect Visual Studio automatically and inject a proper MSVC build environment into the isolated worker
- This is meant to preserve `torch.compile` parity with old direct Transformers 4 environments
- It still depends on the Build Tools actually being installed
**Note**: The `reduce-overhead` mode (which includes automatic CUDA graphs) may still have issues with cudaMallocAsync. Use `compile_mode="default"` for best compatibility.
@@ -88,10 +101,20 @@ Once the environment is configured:
pip install --upgrade torchvision --index-url https://download.pytorch.org/whl/cu130
```
### Error: "cudaMallocAsync does not support checkPoolLiveAllocations"
**Cause**: Using `compile_mode="reduce-overhead"` which enables automatic CUDA graphs.
**Solution**: Use `compile_mode="default"` instead (recommended for Windows).
### Error: "cudaMallocAsync does not support checkPoolLiveAllocations"
**Cause**: Using `compile_mode="reduce-overhead"` which enables automatic CUDA graphs.
**Solution**: Use `compile_mode="default"` instead (recommended for Windows).
### Error: "Failed to find C compiler. Please specify via CC environment variable."
**Cause**: Triton/Inductor could not find a usable Windows C++ toolchain.
**Solution**:
1. Install or repair Visual Studio Build Tools with the C++ workload
2. Restart ComfyUI so the new environment is visible
3. If using Shared Runtime, let TTS Audio Suite auto-detect the toolchain
If detection still fails, your machine state is broken, not just the Python package stack.
### Warning: "TensorFloat32 tensor cores... not enabled"
**Info**: This is just a performance hint, not an error. TF32 is automatically enabled by the extension for better performance.
@@ -110,12 +133,13 @@ pip install --upgrade torchvision --index-url https://download.pytorch.org/whl/c
### Key Findings
- **torch.compile alone provides ~1.7x speedup** with mode="default"
- **Manual CUDA graphs add almost nothing** on top of torch.compile
- **reduce-overhead and max-autotune fail on Windows** (work on Linux)
- **First generation is slower** (~30-60s compilation overhead)
- **Subsequent generations are fast** (compiled kernels cached)
- **Model unloading works correctly** even with CUDA graphs enabled
- **torch.compile alone provides ~1.7x speedup** with mode="default"
- **Manual CUDA graphs add almost nothing** on top of torch.compile
- **reduce-overhead and max-autotune fail on Windows** (work on Linux)
- **First generation is slower** (~30-60s compilation overhead)
- **Subsequent generations are fast** (compiled kernels cached)
- **Model unloading works correctly** even with CUDA graphs enabled
- **Shared Runtime should preserve these gains** if the Windows compiler toolchain is installed and detected correctly
## Notes
+66 -5
View File
@@ -4,7 +4,7 @@
try:
from .chatterbox_adapter import ChatterBoxEngineAdapter
CHATTERBOX_ADAPTER_AVAILABLE = True
except ImportError as e:
except Exception as e:
CHATTERBOX_ADAPTER_AVAILABLE = False
# Create dummy class for compatibility
class ChatterBoxEngineAdapter:
@@ -14,7 +14,7 @@ except ImportError as e:
try:
from .f5tts_adapter import F5TTSEngineAdapter
F5TTS_ADAPTER_AVAILABLE = True
except ImportError as e:
except Exception as e:
F5TTS_ADAPTER_AVAILABLE = False
# Create dummy class for compatibility
class F5TTSEngineAdapter:
@@ -24,7 +24,7 @@ except ImportError as e:
try:
from .cosyvoice_adapter import CosyVoiceAdapter
COSYVOICE_ADAPTER_AVAILABLE = True
except ImportError as e:
except Exception as e:
COSYVOICE_ADAPTER_AVAILABLE = False
# Create dummy class for compatibility
class CosyVoiceAdapter:
@@ -34,14 +34,75 @@ except ImportError as e:
try:
from .echo_tts_adapter import EchoTTSEngineAdapter
ECHO_TTS_ADAPTER_AVAILABLE = True
except ImportError as e:
except Exception as e:
ECHO_TTS_ADAPTER_AVAILABLE = False
class EchoTTSEngineAdapter:
def __init__(self, *args, **kwargs):
raise ImportError(f"Echo-TTS adapter not available: {e}")
try:
from .dots_tts_adapter import DotsTTSEngineAdapter
DOTS_TTS_ADAPTER_AVAILABLE = True
except Exception as e:
DOTS_TTS_ADAPTER_AVAILABLE = False
class DotsTTSEngineAdapter:
def __init__(self, *args, **kwargs):
raise ImportError(f"Dots TTS adapter not available: {e}")
try:
from .dramabox_adapter import DramaBoxEngineAdapter
DRAMABOX_ADAPTER_AVAILABLE = True
except Exception as e:
DRAMABOX_ADAPTER_AVAILABLE = False
class DramaBoxEngineAdapter:
def __init__(self, *args, **kwargs):
raise ImportError(f"DramaBox adapter not available: {e}")
try:
from .fish_audio_s2_adapter import FishAudioS2Adapter
FISH_AUDIO_S2_ADAPTER_AVAILABLE = True
except Exception as e:
FISH_AUDIO_S2_ADAPTER_AVAILABLE = False
class FishAudioS2Adapter:
def __init__(self, *args, **kwargs):
raise ImportError(f"Fish Audio S2 adapter not available: {e}")
try:
from .omnivoice_adapter import OmniVoiceEngineAdapter
OMNIVOICE_ADAPTER_AVAILABLE = True
except Exception as e:
OMNIVOICE_ADAPTER_AVAILABLE = False
class OmniVoiceEngineAdapter:
def __init__(self, *args, **kwargs):
raise ImportError(f"OmniVoice adapter not available: {e}")
try:
from .moss_tts_adapter import MossTTSEngineAdapter
MOSS_TTS_ADAPTER_AVAILABLE = True
except Exception as e:
MOSS_TTS_ADAPTER_AVAILABLE = False
class MossTTSEngineAdapter:
def __init__(self, *args, **kwargs):
raise ImportError(f"MOSS-TTS adapter not available: {e}")
try:
from .higgs_audio_v3_adapter import HiggsAudioV3EngineAdapter
HIGGS_AUDIO_V3_ADAPTER_AVAILABLE = True
except Exception as e:
HIGGS_AUDIO_V3_ADAPTER_AVAILABLE = False
class HiggsAudioV3EngineAdapter:
def __init__(self, *args, **kwargs):
raise ImportError(f"Higgs Audio v3 adapter not available: {e}")
__all__ = [
'ChatterBoxEngineAdapter', 'F5TTSEngineAdapter', 'CosyVoiceAdapter', 'EchoTTSEngineAdapter',
'DotsTTSEngineAdapter', 'DramaBoxEngineAdapter', 'OmniVoiceEngineAdapter',
'MossTTSEngineAdapter', 'HiggsAudioV3EngineAdapter',
'CHATTERBOX_ADAPTER_AVAILABLE', 'F5TTS_ADAPTER_AVAILABLE', 'COSYVOICE_ADAPTER_AVAILABLE',
'ECHO_TTS_ADAPTER_AVAILABLE'
'ECHO_TTS_ADAPTER_AVAILABLE', 'DOTS_TTS_ADAPTER_AVAILABLE',
'DRAMABOX_ADAPTER_AVAILABLE', 'OMNIVOICE_ADAPTER_AVAILABLE',
'MOSS_TTS_ADAPTER_AVAILABLE', 'HIGGS_AUDIO_V3_ADAPTER_AVAILABLE',
'MossSoundEffectV2Adapter'
]
from .moss_soundeffect_v2_adapter import MossSoundEffectV2Adapter
+350 -28
View File
@@ -10,11 +10,211 @@ import torch.nn.functional as F
import comfy.model_management as model_management
import comfy.utils
import time
import re
from engines.granite_asr.prompting import DEFAULT_TRANSLATE_INSTRUCTION_TEMPLATE
from utils.asr.types import ASRRequest, ASRResult, ASRSegment, ASRWord
def parse_granite_native_timestamps(ts_text: str, chunk_offset: float) -> Tuple[str, List[ASRSegment]]:
"""
Parses native timestamped text generated by ibm-granite/granite-speech-4.1-2b-plus.
Reconstructs absolute timestamps by unwrapping modulo-1000 rollovers.
"""
if not ts_text or not ts_text.strip():
return "", []
ts_words = re.split(r"\[T:([^\]]+)\]", ts_text)
last_word_end_time = 0.0
offset_time = 0.0
segments: List[ASRSegment] = []
plain_words: List[str] = []
for raw_word, ts_str in zip(ts_words[::2], ts_words[1::2]):
word_text = raw_word.strip()
if not ts_str:
continue
try:
word_end_time = float(ts_str) / 100.0
except ValueError:
continue
while word_end_time + offset_time < last_word_end_time:
offset_time += 10.0
absolute_end_time = word_end_time + offset_time
absolute_start_time = last_word_end_time
last_word_end_time = absolute_end_time
if word_text == "_":
continue
if word_text:
plain_words.append(word_text)
word = ASRWord(
start=absolute_start_time + chunk_offset,
end=absolute_end_time + chunk_offset,
text=word_text,
)
segments.append(
ASRSegment(
start=word.start,
end=word.end,
text=word.text,
words=[word],
)
)
plain_text = " ".join(plain_words)
return plain_text, segments
def parse_granite_diarization(text: str, chunk_offset: float, chunk_duration: float) -> Tuple[str, List[ASRSegment]]:
"""
Parses speaker-attributed ASR text generated by ibm-granite/granite-speech-4.1-2b-plus.
Returns:
plain_text: Cleaned text without speaker tags.
segments: List of ASRSegment instances with speaker property populated.
"""
if not text or not text.strip():
return "", []
# Find speaker tags like [Speaker 1]:, [Speaker 2]:, etc.
# Keep the delimiter matching group so we can reconstruct who is speaking.
pattern = r"\[(Speaker \d+)\]:?"
parts = re.split(pattern, text)
segments: List[ASRSegment] = []
# Handle text before the first speaker tag, if any
first_text = parts[0].strip()
if first_text:
segments.append(
ASRSegment(
start=chunk_offset,
end=chunk_offset + chunk_duration,
text=first_text,
speaker=None,
words=[],
)
)
for speaker, content in zip(parts[1::2], parts[2::2]):
segment_text = content.strip()
if not segment_text:
continue
segments.append(
ASRSegment(
start=chunk_offset,
end=chunk_offset + chunk_duration,
text=segment_text,
speaker=speaker,
words=[],
)
)
plain_text = " ".join(seg.text for seg in segments if seg.text).strip()
plain_text = re.sub(r"\s+", " ", plain_text).strip()
# Distribute the chunk_duration proportionally across segments based on character length
total_len = sum(len(seg.text) for seg in segments if seg.text)
if total_len > 0:
current_time = chunk_offset
for seg in segments:
seg_len = len(seg.text) if seg.text else 0
duration = chunk_duration * (seg_len / total_len)
seg.start = current_time
seg.end = current_time + duration
current_time += duration
return plain_text, segments
def format_granite_diarization_segments(segments: List[ASRSegment]) -> str:
"""
Normalize Granite speaker attribution to the suite's canonical bracket format:
[Speaker 1] text
"""
parts: List[str] = []
for seg in segments:
text = (seg.text or "").strip()
if not text:
continue
if seg.speaker:
parts.append(f"[{seg.speaker}] {text}")
else:
parts.append(text)
return " ".join(parts).strip()
def remap_words_to_speaker_segments(speaker_segments: List[ASRSegment], aligned_words: List[ASRWord]) -> None:
"""
Distributes aligned words sequentially to speaker_segments based on text matching.
"""
if not speaker_segments or not aligned_words:
return
word_idx = 0
num_words = len(aligned_words)
for segment in speaker_segments:
# Tokenize segment text to find expected words
expected_tokens = [w.lower() for w in re.findall(r"\b\w+\b", segment.text) if w]
if not expected_tokens:
continue
segment_words = []
match_count = 0
limit = len(expected_tokens)
while word_idx < num_words and match_count < limit:
aligned_word = aligned_words[word_idx]
aligned_token_clean = "".join(ch for ch in aligned_word.text.lower() if ch.isalnum())
expected_token_clean = "".join(ch for ch in expected_tokens[match_count] if ch.isalnum())
if not aligned_token_clean:
segment_words.append(aligned_word)
word_idx += 1
continue
if (aligned_token_clean == expected_token_clean or
expected_token_clean in aligned_token_clean or
aligned_token_clean in expected_token_clean):
segment_words.append(aligned_word)
word_idx += 1
match_count += 1
elif (match_count + 1 < limit and
("".join(ch for ch in expected_tokens[match_count + 1] if ch.isalnum()) in aligned_token_clean or
aligned_token_clean in "".join(ch for ch in expected_tokens[match_count + 1] if ch.isalnum()))):
# Missed an expected word in alignment, skip the expected word and match current aligned to next expected
match_count += 2
segment_words.append(aligned_word)
word_idx += 1
else:
# Mismatch, consume to avoid getting stuck
segment_words.append(aligned_word)
word_idx += 1
match_count += 1
if segment_words:
segment.words = segment_words
segment.start = segment_words[0].start
segment.end = segment_words[-1].end
# If there are leftover words, append them to the last segment that had some words
if word_idx < num_words and speaker_segments:
last_seg = speaker_segments[-1]
if last_seg.words:
last_seg.words.extend(aligned_words[word_idx:])
last_seg.end = aligned_words[-1].end
else:
last_seg.words = aligned_words[word_idx:]
last_seg.start = aligned_words[word_idx].start
last_seg.end = aligned_words[-1].end
class GraniteASREngineAdapter:
GRANITE_LANGUAGES = {
"english": "English",
@@ -48,7 +248,11 @@ class GraniteASREngineAdapter:
def _normalize_language(self, language: Optional[str]) -> Optional[str]:
if not language or str(language).lower() == "auto":
return None
return self.GRANITE_LANGUAGES.get(str(language).strip().lower())
normalized = self.GRANITE_LANGUAGES.get(str(language).strip().lower())
model_name = self.config.get("model_name", "")
if normalized == "Japanese" and "plus" in str(model_name).lower():
return None
return normalized
def _get_model(self):
from utils.models.unified_model_interface import unified_model_interface, ModelLoadConfig
@@ -63,8 +267,8 @@ class GraniteASREngineAdapter:
config = ModelLoadConfig(
engine_name="granite_asr",
model_type="asr",
model_name=self.config.get("model_name", "granite-4.0-1b-speech"),
model_path=self.config.get("model_name", "granite-4.0-1b-speech"),
model_name=self.config.get("model_name", "granite-speech-4.1-2b"),
model_path=self.config.get("model_name", "granite-speech-4.1-2b"),
device=device,
additional_params={
"precision": self.config.get("dtype", "auto"),
@@ -75,6 +279,7 @@ class GraniteASREngineAdapter:
def _get_forced_aligner(self):
from utils.models.unified_model_interface import unified_model_interface, ModelLoadConfig
from utils.models.factory_config import RUNTIME_MODE_SHARED
from utils.device import resolve_torch_device
device = self.config.get("device", "auto")
@@ -89,6 +294,8 @@ class GraniteASREngineAdapter:
model_name="Qwen3-ForcedAligner-0.6B",
model_path="Qwen3-ForcedAligner-0.6B",
device=device,
runtime_mode=self.config.get("forced_aligner_runtime_mode", RUNTIME_MODE_SHARED),
runtime_profile=self.config.get("forced_aligner_runtime_profile"),
additional_params={
"precision": self.config.get("dtype", "auto"),
"attn_implementation": self.config.get("attn_implementation", "auto"),
@@ -108,13 +315,12 @@ class GraniteASREngineAdapter:
waveform = waveform[0]
if sample_rate != 16000:
w = waveform.view(1, 1, -1)
w = F.interpolate(w, size=int(w.shape[-1] * 16000 / sample_rate), mode="linear", align_corners=False)
waveform = w.view(-1)
import torchaudio.functional as TA_F
waveform = TA_F.resample(waveform, orig_freq=sample_rate, new_freq=16000)
return waveform
def _alignment_requested(self, req: ASRRequest) -> bool:
return req.timestamps == "word" or req.use_forced_aligner
return req.timestamps == "word"
def _can_use_aligner(self, req: ASRRequest, normalized_language: Optional[str], warnings: List[str]) -> bool:
if not self._alignment_requested(req):
@@ -216,6 +422,32 @@ class GraniteASREngineAdapter:
warnings.append(f"Granite timestamps skipped: Qwen forced aligner failed ({e}).")
return []
def _resolve_max_new_tokens(
self,
req: ASRRequest,
*,
chunk_num_samples: int,
sample_rate: int,
timestamps: bool = False,
diarization: bool = False,
) -> int:
base_max_new_tokens = int(self.config.get("max_new_tokens", req.max_new_tokens or 200))
chunk_seconds = max(1.0, float(chunk_num_samples) / float(sample_rate))
# Native timestamp mode expands output heavily because each spoken word is
# followed by a [T:NNN] tag. Speaker attribution also adds extra label tokens.
# Keep the user-configured cap as the floor, but auto-raise low budgets so
# longer chunks do not get truncated mid-transcript.
if timestamps:
recommended_floor = int(max(256, min(2048, round(chunk_seconds * 16.0))))
return max(base_max_new_tokens, recommended_floor)
if diarization:
recommended_floor = int(max(256, min(2048, round(chunk_seconds * 10.0))))
return max(base_max_new_tokens, recommended_floor)
return base_max_new_tokens
def _run_chunk(
self,
runtime,
@@ -224,9 +456,17 @@ class GraniteASREngineAdapter:
normalized_language: Optional[str],
target_language: Optional[str],
translate_instruction_override: Optional[str],
timestamps: bool = False,
diarization: bool = False,
) -> Tuple[str, Optional[str]]:
generation_kwargs = {
"max_new_tokens": int(self.config.get("max_new_tokens", req.max_new_tokens or 200)),
"max_new_tokens": self._resolve_max_new_tokens(
req,
chunk_num_samples=len(chunk_np),
sample_rate=16000,
timestamps=timestamps,
diarization=diarization,
),
"do_sample": bool(self.config.get("do_sample", False)),
"num_beams": int(self.config.get("num_beams", 1)),
"temperature": float(self.config.get("temperature", 1.0)),
@@ -244,6 +484,8 @@ class GraniteASREngineAdapter:
target_language=target_language,
translate_instruction_override=translate_instruction_override,
generation_kwargs=generation_kwargs,
timestamps=timestamps,
diarization=diarization,
)
return result.get("text", ""), result.get("language")
@@ -258,15 +500,30 @@ class GraniteASREngineAdapter:
warnings: List[str] = []
notes: List[str] = []
normalized_language = self._normalize_language(req.language)
model_name = self.config.get("model_name", "granite-speech-4.1-2b")
is_plus = "plus" in str(model_name).lower()
diarization_requested = is_plus and req.diarization and req.task == "transcribe"
if req.diarization:
if not is_plus:
warnings.append("Granite diarization skipped: Only 'plus' variants support speaker attribution natively.")
elif req.task != "transcribe":
warnings.append("Granite diarization skipped: Speaker attribution is only supported for transcription, not translation.")
target_language = self.config.get("asr_translate_target_language", "English")
translate_instruction_override = str(
self.config.get("asr_translate_instruction_override", DEFAULT_TRANSLATE_INSTRUCTION_TEMPLATE)
or DEFAULT_TRANSLATE_INSTRUCTION_TEMPLATE
).strip()
if req.language and str(req.language).lower() != "auto" and not normalized_language:
warnings.append(
f"Granite received unsupported language hint '{req.language}'. Falling back to Granite auto language handling."
)
if str(req.language).lower() == "japanese" and "plus" in str(model_name).lower():
warnings.append(
"Granite Speech Plus variant does not support Japanese. Falling back to Granite auto language handling."
)
else:
warnings.append(
f"Granite received unsupported language hint '{req.language}'. Falling back to Granite auto language handling."
)
if req.task == "translate":
self._append_once(
notes,
@@ -278,7 +535,8 @@ class GraniteASREngineAdapter:
"Granite custom translation instruction override is active on the engine node. This is still wrapped in Granite's chat format, but malformed or weak instructions can fall back to plain transcription.",
)
use_aligner = self._can_use_aligner(req, normalized_language, warnings)
use_native_timestamps = is_plus and req.timestamps == "word" and req.task == "transcribe" and not diarization_requested
use_aligner = not use_native_timestamps and self._can_use_aligner(req, normalized_language, warnings)
aligner = self._get_forced_aligner() if use_aligner else None
chunk_size = int(req.chunk_size)
@@ -311,25 +569,58 @@ class GraniteASREngineAdapter:
normalized_language,
target_language,
translate_instruction_override,
timestamps=use_native_timestamps,
diarization=diarization_requested,
)
if text:
full_text_parts.append(text)
if detected_language and not output_language:
output_language = detected_language
if aligner is not None and text:
aligner_language = self._resolve_aligner_language(normalized_language, text, notes)
if diarization_requested:
chunk_offset = start / sample_rate
segments.extend(
self._align_words(
parsed_text, chunk_segments = parse_granite_diarization(text, chunk_offset, len(chunk_np) / sample_rate)
if parsed_text:
full_text_parts.append(format_granite_diarization_segments(chunk_segments))
if aligner is not None and parsed_text:
aligner_language = self._resolve_aligner_language(normalized_language, parsed_text, notes)
word_segments = self._align_words(
aligner=aligner,
chunk_audio=chunk_np,
transcript=text,
transcript=parsed_text,
language=aligner_language,
chunk_offset=chunk_offset,
warnings=warnings,
)
)
aligned_words = [seg.words[0] for seg in word_segments if seg.words]
remap_words_to_speaker_segments(chunk_segments, aligned_words)
for seg in chunk_segments:
if seg.speaker:
seg.text = f"[{seg.speaker}] {seg.text}"
segments.extend(chunk_segments)
elif use_native_timestamps:
chunk_offset = start / sample_rate
parsed_text, chunk_segments = parse_granite_native_timestamps(text, chunk_offset)
if parsed_text:
full_text_parts.append(parsed_text)
segments.extend(chunk_segments)
else:
if text:
full_text_parts.append(text)
if aligner is not None and text:
aligner_language = self._resolve_aligner_language(normalized_language, text, notes)
chunk_offset = start / sample_rate
segments.extend(
self._align_words(
aligner=aligner,
chunk_audio=chunk_np,
transcript=text,
language=aligner_language,
chunk_offset=chunk_offset,
warnings=warnings,
)
)
try:
chunk_progress_bar.update(1)
@@ -360,23 +651,54 @@ class GraniteASREngineAdapter:
normalized_language,
target_language,
translate_instruction_override,
timestamps=use_native_timestamps,
diarization=diarization_requested,
)
if text:
full_text_parts.append(text)
output_language = detected_language or output_language
if aligner is not None and text:
aligner_language = self._resolve_aligner_language(normalized_language, text, notes)
segments.extend(
self._align_words(
if diarization_requested:
parsed_text, chunk_segments = parse_granite_diarization(text, 0.0, len(wav_np) / sample_rate)
if parsed_text:
full_text_parts.append(format_granite_diarization_segments(chunk_segments))
if aligner is not None and parsed_text:
aligner_language = self._resolve_aligner_language(normalized_language, parsed_text, notes)
word_segments = self._align_words(
aligner=aligner,
chunk_audio=wav_np,
transcript=text,
transcript=parsed_text,
language=aligner_language,
chunk_offset=0.0,
warnings=warnings,
)
)
aligned_words = [seg.words[0] for seg in word_segments if seg.words]
remap_words_to_speaker_segments(chunk_segments, aligned_words)
for seg in chunk_segments:
if seg.speaker:
seg.text = f"[{seg.speaker}] {seg.text}"
segments.extend(chunk_segments)
elif use_native_timestamps:
parsed_text, chunk_segments = parse_granite_native_timestamps(text, 0.0)
if parsed_text:
full_text_parts.append(parsed_text)
segments.extend(chunk_segments)
else:
if text:
full_text_parts.append(text)
if aligner is not None and text:
aligner_language = self._resolve_aligner_language(normalized_language, text, notes)
segments.extend(
self._align_words(
aligner=aligner,
chunk_audio=wav_np,
transcript=text,
language=aligner_language,
chunk_offset=0.0,
warnings=warnings,
)
)
text = " ".join(part for part in full_text_parts if part).strip()
raw = None
+2
View File
@@ -76,6 +76,8 @@ class Qwen3ASREngineAdapter:
model_name=model_name,
model_path=model_id,
device=device,
runtime_mode=self.config.get("runtime_mode", "main_environment"),
runtime_profile=self.config.get("runtime_profile"),
additional_params=additional_params,
)
+40 -33
View File
@@ -67,7 +67,43 @@ class ChatterBoxEngineAdapter:
"""
self.node.load_tts_model(device, language)
def generate_segment_audio(self, text: str, char_audio: str,
def create_segment_cache(self, character="narrator", *, cache_model_name=None,
cache_probe=False, **params):
"""Share cache identity between generation and pre-load probes."""
if not params.get("enable_audio_cache", True):
return None
from utils.audio.cache import create_cache_function
language = cache_model_name or params.get("current_language", params.get("model", "English"))
device = params.get("device", "auto")
model_source = params.get("model_source")
if not model_source:
# Do not borrow the source of a different currently loaded language.
sources = getattr(self, "_cache_model_sources", {})
if cache_probe:
model_source = sources.get((language, device))
if model_source is None:
return None
else:
if hasattr(self.node, "model_manager"):
model_source = self.node.model_manager.get_model_source("tts")
model_source = model_source or "unknown"
sources[(language, device)] = model_source
self._cache_model_sources = sources
audio_component = params.get("stable_audio_component", "main_reference")
if character != "narrator":
audio_component = f"char_file_{character}"
return create_cache_function(
engine_type="chatterbox", character=character,
exaggeration=params.get("exaggeration", 1.0),
temperature=params.get("temperature", 0.8),
cfg_weight=params.get("cfg_weight", 1.0),
seed=params.get("seed", 0), audio_component=audio_component,
model_source=model_source, device=device, language=language,
)
def generate_segment_audio(self, text: str, char_audio: str,
character: str = "narrator", **params) -> torch.Tensor:
"""
Generate ChatterBox audio for a text segment.
@@ -88,34 +124,7 @@ class ChatterBoxEngineAdapter:
seed = params.get("seed", 0)
enable_cache = params.get("enable_audio_cache", True)
# Create cache function if caching is enabled
cache_fn = None
if enable_cache:
from utils.audio.cache import create_cache_function
# Get current language/model for cache key
current_language = params.get("current_language", params.get("model", "English"))
audio_component = params.get("stable_audio_component", "main_reference")
if character != "narrator":
audio_component = f"char_file_{character}"
# Get model source
model_source = params.get("model_source")
if not model_source and hasattr(self.node, 'model_manager'):
model_source = self.node.model_manager.get_model_source("tts")
cache_fn = create_cache_function(
engine_type="chatterbox",
character=character,
exaggeration=exaggeration,
temperature=temperature,
cfg_weight=cfg_weight,
seed=seed,
audio_component=audio_component,
model_source=model_source or "unknown",
device=params.get("device", "auto"),
language=current_language
)
cache_fn = self.create_segment_cache(character=character, **params)
# Handle caching externally for consistency with F5-TTS
if cache_fn:
@@ -142,8 +151,7 @@ class ChatterBoxEngineAdapter:
enable_pause_tags=True,
character=character,
seed=seed,
enable_cache=True, # Enable internal caching for pause tag processing
crash_protection_template=params.get("crash_protection_template", "hmm ,, {seg} hmm ,,"),
enable_cache=enable_cache,
stable_audio_component=params.get("stable_audio_component", "main_reference")
)
# Don't use external cache for pause tag segments
@@ -160,7 +168,6 @@ class ChatterBoxEngineAdapter:
character=character,
seed=seed,
enable_cache=False, # Disable internal caching since we handle it externally
crash_protection_template=params.get("crash_protection_template", "hmm ,, {seg} hmm ,,"),
stable_audio_component=params.get("stable_audio_component", "main_reference")
)
@@ -400,4 +407,4 @@ class ChatterBoxEngineAdapter:
else:
num_samples = audio_tensor.numel()
return num_samples / 44100 # ChatterBox sample rate
return num_samples / 44100 # ChatterBox sample rate
@@ -110,7 +110,6 @@ class ChatterBoxStreamingAdapter(StreamingEngineAdapter):
"seed": kwargs.get("seed", 42),
"enable_chunking": False, # Don't chunk in streaming - already handled
"enable_audio_cache": kwargs.get("enable_audio_cache", True),
"crash_protection_template": kwargs.get("crash_protection_template", "hmm ,, {seg} hmm ,,"),
"device": kwargs.get("device", "auto"),
"reference_audio": kwargs.get("reference_audio", None)
}
@@ -300,7 +299,6 @@ class ChatterBoxStreamingAdapter(StreamingEngineAdapter):
cfg_weight = kwargs.get('cfg_weight', 0.5)
seed = kwargs.get('seed', 42)
enable_cache = kwargs.get('enable_audio_cache', True)
crash_protection = kwargs.get('crash_protection_template', 'hmm ,, {seg} hmm ,,')
# Check if node has generation method
if hasattr(self.node, '_generate_tts_with_pause_tags'):
@@ -321,7 +319,6 @@ class ChatterBoxStreamingAdapter(StreamingEngineAdapter):
text, voice_path, exaggeration, temperature, cfg_weight,
language, True, character=character, seed=seed,
enable_cache=enable_cache,
crash_protection_template=crash_protection,
stable_audio_component=stable_audio_component
)
elif hasattr(self.node, 'tts_model') and self.node.tts_model:
+3 -2
View File
@@ -23,6 +23,7 @@ from engines.cosyvoice.cosyvoice import CosyVoiceEngine
from engines.cosyvoice.cosyvoice_downloader import cosyvoice_downloader
from utils.text.character_parser import character_parser
from utils.voice.discovery import get_character_mapping, get_available_characters
from utils.voice.character_logging import resolved_character_label
from utils.audio.cache import get_audio_cache
@@ -336,7 +337,7 @@ class CosyVoiceAdapter:
speaker_audio = char_audio
if char_text:
reference_text = char_text
print(f"📖 Using character voice '{character_name}'")
print(f"📖 Using character voice '{resolved_character_label(character_name, speaker_audio)}'")
# Generate cache key for this segment
segment_cache_key = self._generate_cache_key(
@@ -352,7 +353,7 @@ class CosyVoiceAdapter:
# Check cache first
cached_segment_audio = self.audio_cache.get_cached_audio(segment_cache_key)
if cached_segment_audio:
print(f"💾 Using cached CosyVoice3 segment for '{character_name}'")
print(f"💾 Using cached CosyVoice3 segment for '{resolved_character_label(character_name, speaker_audio)}'")
segment_audio = cached_segment_audio[0]
else:
# Convert CosyVoice paralinguistic tags from <tag> to [tag]
+221
View File
@@ -0,0 +1,221 @@
"""
Dots TTS engine adapter.
Translates TTS Audio Suite processor calls into the official dots.tts runtime.
"""
import os
import sys
from typing import Any, Dict, Optional, Tuple
import torch
current_dir = os.path.dirname(__file__)
engines_dir = os.path.dirname(current_dir)
project_root = os.path.dirname(engines_dir)
if project_root not in sys.path:
sys.path.insert(0, project_root)
from utils.audio.audio_hash import generate_stable_audio_component
from utils.audio.cache import get_audio_cache
from utils.audio.processing import AudioProcessingUtils
from utils.models.factory_config import ModelLoadConfig
from utils.models.language_mapper import resolve_language_alias
from utils.voice.reference import effective_voice_audio
from engines.dots_tts.languages import normalize_dots_language
class DotsTTSEngineAdapter:
"""Adapter for official Dots TTS inference."""
SAMPLE_RATE = 48000
def __init__(self, config: Optional[Dict[str, Any]] = None):
self.config = config.copy() if config else {}
self.audio_cache = get_audio_cache()
self._last_config: Optional[ModelLoadConfig] = None
self._load_signature = None
def update_config(self, new_config: Dict[str, Any]):
self.config = new_config.copy() if new_config else {}
def _build_load_signature(self) -> Tuple[Any, ...]:
return (
self.config.get("model_variant", "dots.tts-soar"),
self.config.get("device", "auto"),
self.config.get("precision", "auto"),
bool(self.config.get("optimize", False)),
int(self.config.get("max_generate_length", 500)),
)
def load_model(
self,
model_variant: str,
device: str = "auto",
precision: str = "auto",
optimize: bool = False,
max_generate_length: int = 500,
):
from utils.models.unified_model_interface import unified_model_interface
config = ModelLoadConfig(
engine_name="dots_tts",
model_type="tts",
model_name=model_variant,
model_path=model_variant,
device=device,
additional_params={
"precision": precision,
"optimize": bool(optimize),
"max_generate_length": int(max_generate_length),
},
)
self._last_config = config
unified_model_interface.load_model(config)
def _ensure_model_loaded(self):
signature = self._build_load_signature()
if signature == self._load_signature and self._last_config is not None:
return
self.load_model(
model_variant=self.config.get("model_variant", "dots.tts-soar"),
device=self.config.get("device", "auto"),
precision=self.config.get("precision", "auto"),
optimize=self.config.get("optimize", False),
max_generate_length=self.config.get("max_generate_length", 500),
)
self._load_signature = signature
def _get_engine(self):
if self._last_config is None:
self._ensure_model_loaded()
from utils.models.unified_model_interface import unified_model_interface
return unified_model_interface.load_model(self._last_config)
def _extract_voice_reference(self, voice_ref: Optional[Dict[str, Any]]) -> Tuple[Optional[str], str, str]:
if not voice_ref or not isinstance(voice_ref, dict):
return None, "", "default_voice"
prompt_text = (
voice_ref.get("reference_text")
or voice_ref.get("prompt_text")
or voice_ref.get("text")
or ""
).strip()
ref_audio = effective_voice_audio(voice_ref)
if ref_audio is None:
return None, prompt_text, "default_voice"
if isinstance(ref_audio, str):
return ref_audio, prompt_text, generate_stable_audio_component(audio_file_path=ref_audio)
if isinstance(ref_audio, dict) and "waveform" in ref_audio:
temp_path = AudioProcessingUtils.save_audio_to_temp_file(
ref_audio["waveform"],
ref_audio.get("sample_rate", self.SAMPLE_RATE),
)
return temp_path, prompt_text, generate_stable_audio_component(reference_audio=ref_audio)
if torch.is_tensor(ref_audio):
sample_rate = int(voice_ref.get("sample_rate", self.SAMPLE_RATE))
audio_dict = {"waveform": ref_audio, "sample_rate": sample_rate}
temp_path = AudioProcessingUtils.save_audio_to_temp_file(ref_audio, sample_rate)
return temp_path, prompt_text, generate_stable_audio_component(reference_audio=audio_dict)
raise TypeError(f"Unsupported Dots TTS voice reference type: {type(ref_audio)}")
@staticmethod
def _normalize_language(language: Optional[str]) -> Optional[str]:
if language is None:
return None
normalized = str(language).strip()
if not normalized:
return None
normalized_language = normalize_dots_language(normalized)
if normalized_language is not None:
return normalized_language
resolved = resolve_language_alias(normalized)
return normalize_dots_language(resolved)
def generate_single(
self,
text: str,
voice_ref: Optional[Dict[str, Any]],
seed: int = 0,
enable_audio_cache: bool = True,
character_name: Optional[str] = None,
) -> torch.Tensor:
stripped = (text or "").strip()
if not stripped:
return torch.zeros(1, 0, dtype=torch.float32)
self._ensure_model_loaded()
engine = self._get_engine()
prompt_audio_path, prompt_text, audio_component = self._extract_voice_reference(voice_ref)
language = self._normalize_language(self.config.get("language", "auto"))
model_variant = self.config.get("model_variant", "dots.tts-soar")
template_name = self.config.get("template_name", "tts")
num_steps = int(self.config.get("num_steps", 10))
guidance_scale = float(self.config.get("guidance_scale", 1.2))
speaker_scale = float(self.config.get("speaker_scale", 1.5))
normalize_text = bool(self.config.get("normalize_text", False))
precision = self.config.get("precision", "auto")
optimize = bool(self.config.get("optimize", False))
max_generate_length = int(self.config.get("max_generate_length", 500))
cache_key = None
if enable_audio_cache:
cache_key = self.audio_cache.generate_cache_key(
"dots_tts",
text=stripped,
audio_component=audio_component,
prompt_text=prompt_text,
model_variant=model_variant,
language=language or "none",
template_name=template_name,
num_steps=num_steps,
guidance_scale=guidance_scale,
speaker_scale=speaker_scale,
normalize_text=normalize_text,
max_generate_length=max_generate_length,
precision=precision,
optimize=optimize,
seed=int(seed or 0),
device=self.config.get("device", "auto"),
character=character_name or "narrator",
)
cached_audio = self.audio_cache.get_cached_audio(cache_key)
if cached_audio:
print(f"💾 Using cached Dots TTS audio for '{character_name or 'narrator'}': '{stripped[:30]}...'")
return cached_audio[0]
result = engine.generate(
text=stripped,
prompt_audio_path=prompt_audio_path,
prompt_text=prompt_text or None,
language=language,
num_steps=num_steps,
guidance_scale=guidance_scale,
speaker_scale=speaker_scale,
normalize_text=normalize_text,
template_name=template_name,
seed=int(seed or 0),
)
audio_tensor = result["audio"]
if not isinstance(audio_tensor, torch.Tensor):
audio_tensor = torch.tensor(audio_tensor, dtype=torch.float32)
audio_tensor = audio_tensor.detach().float().cpu()
if audio_tensor.dim() == 1:
audio_tensor = audio_tensor.unsqueeze(0)
if enable_audio_cache and cache_key:
duration = self.audio_cache._calculate_duration(audio_tensor, "dots_tts")
self.audio_cache.cache_audio(cache_key, audio_tensor, duration)
return audio_tensor
+300
View File
@@ -0,0 +1,300 @@
"""Adapter between unified TTS processing and official DramaBox inference."""
import math
import os
from typing import Any, Dict, Optional, Tuple
import torch
import torchaudio
from utils.audio.audio_hash import generate_stable_audio_component
from utils.audio.cache import get_audio_cache
from utils.audio.processing import AudioProcessingUtils
from utils.models.factory_config import ModelLoadConfig
from utils.voice.reference import effective_voice_audio
class DramaBoxEngineAdapter:
"""Translate suite voice/config/cache data into DramaBox calls."""
SAMPLE_RATE = 48000
SILENCE_RMS_THRESHOLD = 1e-3
SILENCE_PEAK_THRESHOLD = 2e-2
def __init__(self, config: Optional[Dict[str, Any]] = None):
self.config = config.copy() if config else {}
self.audio_cache = get_audio_cache()
self._last_config: Optional[ModelLoadConfig] = None
self._load_signature = None
self._lora_signature = None
self.last_generation_status: Dict[str, Any] = {"near_silent": False}
def update_config(self, new_config: Dict[str, Any]):
self.config = new_config.copy() if new_config else {}
@staticmethod
def _lora_revision(path: Any) -> str:
"""Return a cheap cache token that changes when a managed adapter is replaced."""
value = str(path or "").strip()
if not value:
return ""
try:
candidate = os.path.abspath(os.path.expanduser(value))
if os.path.isfile(candidate):
stat = os.stat(candidate)
return f"{candidate}:{stat.st_size}:{stat.st_mtime_ns}"
if os.path.isdir(candidate):
entries = []
for item in os.listdir(candidate):
if not item.endswith(".safetensors"):
continue
item_path = os.path.join(candidate, item)
stat = os.stat(item_path)
entries.append(f"{item}:{stat.st_size}:{stat.st_mtime_ns}")
return f"{candidate}|{'|'.join(sorted(entries))}"
except OSError:
pass
return value
@classmethod
def _warn_if_near_silent(
cls,
audio: torch.Tensor,
*,
character_name: Optional[str],
seed: int,
cached: bool = False,
) -> Optional[Dict[str, Any]]:
"""Warn about clearly near-silent model output without altering it."""
if not isinstance(audio, torch.Tensor) or audio.numel() == 0:
return None
samples = torch.nan_to_num(audio.detach().float().cpu())
rms = float(samples.square().mean().sqrt())
peak = float(samples.abs().max())
if rms >= cls.SILENCE_RMS_THRESHOLD or peak >= cls.SILENCE_PEAK_THRESHOLD:
return None
rms_db = 20.0 * math.log10(max(rms, 1e-12))
peak_db = 20.0 * math.log10(max(peak, 1e-12))
source = "cached " if cached else ""
print(
f"\n⚠️ DramaBox generated a near-silent {source}segment for "
f"'{character_name or 'narrator'}' "
f"(RMS {rms_db:.1f} dBFS, peak {peak_db:.1f} dBFS)."
)
print(
"⚠️ This can depend on generation duration, reference duration, "
"reference audio, guidance settings, and seed."
)
print(
"⚠️ Try changing those parameters for this segment; another seed "
"may help, but is not guaranteed to fix it.\n"
)
return {
"near_silent": True,
"character": character_name or "narrator",
"seed": int(seed),
"rms_dbfs": rms_db,
"peak_dbfs": peak_db,
"cached": bool(cached),
}
def _build_load_signature(self) -> Tuple[Any, ...]:
"""Identity of the expensive base runtime, excluding live LoRA state."""
return (
self.config.get("model_name", "DramaBox"),
self.config.get("device", "auto"),
self.config.get("precision", "auto"),
self.config.get("memory_mode", "fast"),
self.config.get("transformer_quantization", "none"),
bool(self.config.get("compile_model", False)),
)
def _build_lora_signature(self) -> Tuple[Any, ...]:
path = self.config.get("lora_path", "")
return (
str(path or "").strip(),
self._lora_revision(path),
float(self.config.get("lora_strength", 1.0)),
)
def _ensure_model_loaded(self):
signature = self._build_load_signature()
from utils.models.unified_model_interface import unified_model_interface
if signature != self._load_signature or self._last_config is None:
self._last_config = ModelLoadConfig(
engine_name="dramabox",
model_type="tts",
model_name=self.config.get("model_name", "DramaBox"),
device=self.config.get("device", "auto"),
additional_params={
"precision": self.config.get("precision", "auto"),
"memory_mode": self.config.get("memory_mode", "fast"),
"transformer_quantization": self.config.get(
"transformer_quantization", "none"
),
"compile_model": bool(self.config.get("compile_model", False)),
},
)
self._load_signature = signature
self._lora_signature = None
engine = unified_model_interface.load_model(self._last_config)
lora_signature = self._build_lora_signature()
if lora_signature != self._lora_signature:
lora_path, lora_revision, lora_strength = lora_signature
engine.set_lora(
lora_path=lora_path,
strength=lora_strength,
revision=lora_revision,
)
self._lora_signature = lora_signature
return engine
def _get_engine(self):
return self._ensure_model_loaded()
def _extract_voice_reference(
self, voice_ref: Optional[Dict[str, Any]]
) -> Tuple[Optional[str], str, bool]:
if not isinstance(voice_ref, dict):
return None, "default_voice", False
audio = effective_voice_audio(voice_ref)
if audio is None:
return None, "default_voice", False
if isinstance(audio, str):
return (
audio,
generate_stable_audio_component(audio_file_path=audio),
False,
)
if isinstance(audio, dict) and "waveform" in audio:
path = AudioProcessingUtils.save_audio_to_temp_file(
audio["waveform"], audio.get("sample_rate", self.SAMPLE_RATE)
)
return path, generate_stable_audio_component(reference_audio=audio), True
if torch.is_tensor(audio):
sample_rate = int(voice_ref.get("sample_rate", self.SAMPLE_RATE))
audio_dict = {"waveform": audio, "sample_rate": sample_rate}
path = AudioProcessingUtils.save_audio_to_temp_file(audio, sample_rate)
return (
path,
generate_stable_audio_component(reference_audio=audio_dict),
True,
)
raise TypeError(f"Unsupported DramaBox voice reference: {type(audio)}")
def generate_single(
self,
text: str,
voice_ref: Optional[Dict[str, Any]],
seed: int = 42,
enable_audio_cache: bool = True,
character_name: Optional[str] = None,
) -> torch.Tensor:
prompt = (text or "").strip()
if not prompt:
return torch.zeros(1, 0, dtype=torch.float32)
voice_path, audio_component, remove_voice_path = self._extract_voice_reference(
voice_ref
)
cfg_scale = float(self.config.get("cfg_scale", 2.5))
stg_scale = float(self.config.get("stg_scale", 1.5))
duration_multiplier = float(self.config.get("duration_multiplier", 1.1))
gen_duration = float(self.config.get("gen_duration", 0.0))
ref_duration = float(self.config.get("ref_duration", 10.0))
rescale_scale = self.config.get("rescale_scale", "auto")
watermark = bool(self.config.get("watermark", False))
negative_prompt = str(self.config.get("negative_prompt", ""))
model_name = self.config.get("model_name", "DramaBox")
cache_key = None
if enable_audio_cache:
cache_key = self.audio_cache.generate_cache_key(
"dramabox",
text=prompt,
audio_component=audio_component,
model_name=model_name,
cfg_scale=cfg_scale,
stg_scale=stg_scale,
duration_multiplier=duration_multiplier,
gen_duration=gen_duration,
ref_duration=ref_duration,
rescale_scale=rescale_scale,
watermark=watermark,
prompt_template=str(
self.config.get("prompt_template", '"{seg}"')
),
negative_prompt=negative_prompt,
precision=self.config.get("precision", "auto"),
transformer_quantization=self.config.get(
"transformer_quantization", "none"
),
memory_mode=self.config.get("memory_mode", "fast"),
compile_model=bool(self.config.get("compile_model", False)),
lora_path=self.config.get("lora_path", ""),
lora_strength=float(self.config.get("lora_strength", 1.0)),
lora_revision=self._lora_revision(self.config.get("lora_path", "")),
seed=int(seed),
character=character_name or "narrator",
)
cached = self.audio_cache.get_cached_audio(cache_key)
if cached:
print(
f"💾 Using cached DramaBox audio for "
f"'{character_name or 'narrator'}': '{prompt[:30]}...'"
)
self.last_generation_status = self._warn_if_near_silent(
cached[0],
character_name=character_name,
seed=int(seed),
cached=True,
) or {"near_silent": False}
return cached[0]
try:
result = self._get_engine().generate(
prompt=prompt,
voice_ref_path=voice_path,
cfg_scale=cfg_scale,
stg_scale=stg_scale,
duration_multiplier=duration_multiplier,
gen_duration=gen_duration,
ref_duration=ref_duration,
rescale_scale=rescale_scale,
watermark=watermark,
negative_prompt=negative_prompt,
seed=int(seed),
)
finally:
if remove_voice_path and voice_path:
try:
os.unlink(voice_path)
except OSError:
pass
audio = result["audio"]
if not isinstance(audio, torch.Tensor):
audio = torch.tensor(audio, dtype=torch.float32)
audio = audio.detach().float().cpu()
if audio.dim() == 1:
audio = audio.unsqueeze(0)
if int(result.get("sample_rate", self.SAMPLE_RATE)) != self.SAMPLE_RATE:
audio = torchaudio.functional.resample(
audio, int(result["sample_rate"]), self.SAMPLE_RATE
)
self.last_generation_status = self._warn_if_near_silent(
audio,
character_name=character_name,
seed=int(seed),
) or {"near_silent": False}
if enable_audio_cache and cache_key:
duration = audio.shape[-1] / self.SAMPLE_RATE
self.audio_cache.cache_audio(cache_key, audio, duration)
return audio
+26 -30
View File
@@ -73,7 +73,30 @@ class F5TTSEngineAdapter:
self.node.load_f5tts_model(model_name, device)
def generate_segment_audio(self, text: str, char_audio: str, char_text: str,
def create_segment_cache(self, character="narrator", char_text=None, *,
cache_model_name=None, cache_probe=False, **params):
"""Share cache identity between generation and pre-load probes."""
if not params.get("enable_audio_cache", True):
return None
from utils.audio.cache import create_cache_function
audio_component = params.get("stable_audio_component", "main_reference")
if character != "narrator":
audio_component = f"char_file_{character}"
model_name = cache_model_name or getattr(
self.node, "current_model_name", params.get("model", "F5TTS_Base")
)
return create_cache_function(
engine_type="f5tts", character=character, model_name=model_name,
device=params.get("device", "auto"), audio_component=audio_component,
ref_text=char_text, temperature=params.get("temperature", 0.8),
speed=params.get("speed", 1.0), target_rms=params.get("target_rms", 0.1),
cross_fade_duration=params.get("cross_fade_duration", 0.15),
nfe_step=max(1, min(params.get("nfe_step", 32), 71)),
cfg_strength=params.get("cfg_strength", 2.0), seed=params.get("seed", 0),
)
def generate_segment_audio(self, text: str, char_audio: str, char_text: str,
character: str = "narrator", **params) -> torch.Tensor:
"""
Generate F5-TTS audio for a text segment.
@@ -103,34 +126,7 @@ class F5TTSEngineAdapter:
if safe_nfe_step != nfe_step:
print(f"⚠️ F5-TTS: Clamped nfe_step from {nfe_step} to {safe_nfe_step} to prevent ODE solver issues")
# Create cache function if caching is enabled
cache_fn = None
if enable_cache:
from utils.audio.cache import create_cache_function
# Get audio component for cache key
audio_component = params.get("stable_audio_component", "main_reference")
if character != "narrator":
audio_component = f"char_file_{character}"
# Get current model name for cache key
current_model = getattr(self.node, 'current_model_name', params.get("model", "F5TTS_Base"))
cache_fn = create_cache_function(
engine_type="f5tts",
character=character,
model_name=current_model,
device=params.get("device", "auto"),
audio_component=audio_component,
ref_text=char_text,
temperature=temperature,
speed=speed,
target_rms=target_rms,
cross_fade_duration=cross_fade_duration,
nfe_step=safe_nfe_step,
cfg_strength=cfg_strength,
seed=seed
)
cache_fn = self.create_segment_cache(character=character, char_text=char_text, **params)
# Generate audio using F5-TTS with pause tag support
return self.node.generate_f5tts_with_pause_tags(
@@ -189,4 +185,4 @@ class F5TTSEngineAdapter:
else:
num_samples = audio_tensor.numel()
return num_samples / 24000 # F5-TTS sample rate
return num_samples / 24000 # F5-TTS sample rate
+133
View File
@@ -0,0 +1,133 @@
"""Adapter between suite processors and the isolated official Fish S2 runtime."""
from typing import Any, Dict, Optional
import torch
from engines.fish_audio_s2.downloader import FishAudioS2Downloader
from utils.audio.audio_hash import generate_stable_audio_component
from utils.audio.cache import get_audio_cache
from utils.audio.processing import AudioProcessingUtils
from utils.models.factory_config import ModelLoadConfig
from utils.text.fish_audio_s2_tags import translate_fish_s2_inline_tags
from utils.voice.character_logging import resolved_voice_name
from utils.voice.reference import effective_voice_audio
class FishAudioS2Adapter:
SAMPLE_RATE = 44100
def __init__(self, config: Optional[Dict[str, Any]] = None):
self.config = dict(config or {})
self.audio_cache = get_audio_cache()
self._model_config = None
def update_config(self, config):
self.config = dict(config or {})
def _engine(self):
from utils.models.unified_model_interface import unified_model_interface
model_selection = self.config.get("model_variant", "s2-pro")
quantization = self.config.get("quantization", "none")
model_path = FishAudioS2Downloader.resolve_model_path(model_selection)
model_variant = FishAudioS2Downloader.resolve_model_variant(model_selection, model_path)
self._model_config = ModelLoadConfig(
engine_name="fish_audio_s2", model_type="tts", model_name=model_variant,
model_path=model_path, device=self.config.get("device", "auto"),
runtime_mode="isolated",
additional_params={
"model_variant": model_variant,
"quantization": quantization,
"precision": self.config.get("precision", "bfloat16"),
"compile": bool(self.config.get("compile", False)),
"context_length": int(self.config.get("context_length", 8192)),
},
)
return unified_model_interface.load_model(self._model_config)
def _reference(self, voice_ref):
if not isinstance(voice_ref, dict):
return None, "", "default_voice"
text = (voice_ref.get("reference_text") or voice_ref.get("prompt_text") or "").strip()
effective_audio = effective_voice_audio(voice_ref)
path = effective_audio if isinstance(effective_audio, str) else None
if isinstance(effective_audio, dict):
path = AudioProcessingUtils.save_audio_to_temp_file(
effective_audio["waveform"], effective_audio.get("sample_rate", self.SAMPLE_RATE)
)
elif torch.is_tensor(effective_audio):
path = AudioProcessingUtils.save_audio_to_temp_file(
effective_audio, voice_ref.get("sample_rate", self.SAMPLE_RATE)
)
component = generate_stable_audio_component(audio_file_path=path) if path else "default_voice"
return path, text, component
def generate_single(self, text, voice_ref, seed=0, enable_audio_cache=True, character_name=None):
return self.generate_dialogue(
[(0, text)], [voice_ref], seed, enable_audio_cache,
cache_character=character_name or "narrator",
)
def generate_dialogue(self, turns, voice_refs, seed=0, enable_audio_cache=True,
cache_character="native_dialogue"):
formatted_turns = []
for speaker_index, turn_text in turns:
clean_text = translate_fish_s2_inline_tags((turn_text or "").strip())
if clean_text:
formatted_turns.append(f"<|speaker:{speaker_index}|>{clean_text}")
text = "\n".join(formatted_turns)
if not text:
return torch.zeros(1, 0)
references = []
reference_labels = []
components = []
reference_texts = []
for speaker_index, voice_ref in enumerate(voice_refs):
ref_path, ref_text, component = self._reference(voice_ref)
if ref_path:
if not ref_text:
raise ValueError("Fish S2 native speakers require exact reference transcripts")
references.append({
"audio_path": ref_path,
"text": f"<|speaker:{speaker_index}|>{ref_text}",
})
reference_labels.append(
f"local Speaker {speaker_index + 1}={resolved_voice_name(voice_ref)}"
)
components.append(component)
reference_texts.append(ref_text)
if references:
print(f"🎤 Fish local reference order: {', '.join(reference_labels)}")
params = {
"model_variant": self.config.get("model_variant", "s2-pro"),
"quantization": self.config.get("quantization", "none"),
"multi_speaker_mode": self.config.get("multi_speaker_mode", "Native Multi-Speaker"),
"seed": int(seed), "normalize": bool(self.config.get("normalize", True)),
"chunk_length": int(self.config.get("native_chunk_length", 200)),
"max_new_tokens": int(self.config.get("max_new_tokens", 1024)),
"top_p": float(self.config.get("top_p", 0.8)),
"repetition_penalty": float(self.config.get("repetition_penalty", 1.1)),
"temperature": float(self.config.get("temperature", 0.8)),
"cache_reference": bool(self.config.get("cache_reference", True)),
"context_length": int(self.config.get("context_length", 8192)),
}
cache_key = self.audio_cache.generate_cache_key(
"fish_audio_s2", text=text, audio_component="|".join(components),
reference_text="|".join(reference_texts), character=cache_character, **params,
) if enable_audio_cache else None
if cache_key:
cached = self.audio_cache.get_cached_audio(cache_key)
if cached:
print("💾 Fish Audio S2: Using cached audio")
return cached[0]
audio, sample_rate = self._engine().generate(text=text, references=references, **params)
if sample_rate != self.SAMPLE_RATE:
raise RuntimeError(f"Fish S2 returned unexpected sample rate {sample_rate}")
audio = audio.detach().float().cpu()
if audio.dim() == 1:
audio = audio.unsqueeze(0)
if cache_key:
self.audio_cache.cache_audio(cache_key, audio, audio.shape[-1] / self.SAMPLE_RATE)
return audio
+26 -4
View File
@@ -14,6 +14,8 @@ if parent_dir not in sys.path:
sys.path.insert(0, parent_dir)
from utils.models.language_mapper import get_model_for_language
from utils.models.engine_registry import get_default_runtime_profile
from utils.models.factory_config import normalize_runtime_mode, runtime_uses_isolation
from engines.higgs_audio.higgs_audio import HiggsAudioEngine
from engines.higgs_audio.higgs_audio_downloader import HIGGS_AUDIO_MODELS
@@ -31,6 +33,9 @@ class HiggsAudioEngineAdapter:
self.node = node_instance
self.engine_type = "higgs_audio"
self.higgs_engine = HiggsAudioEngine()
node_config = getattr(node_instance, "config", {}) or {}
self._runtime_mode = normalize_runtime_mode(node_config.get("runtime_mode"))
self._runtime_profile = node_config.get("runtime_profile")
def get_model_for_language(self, lang_code: str, default_model: str) -> str:
"""
@@ -85,6 +90,23 @@ class HiggsAudioEngineAdapter:
# Default fallback
generation_model = "bosonai/higgs-audio-v2-generation-3B-base"
tokenizer_model = "bosonai/higgs-audio-v2-tokenizer"
if runtime_uses_isolation(self._runtime_mode):
from utils.models.unified_model_interface import load_tts_model
self.higgs_engine = load_tts_model(
engine_name="higgs_audio",
model_name=model_name,
device=device,
model_path=generation_model,
tokenizer_path=tokenizer_model,
enable_cuda_graphs=enable_cuda_graphs,
runtime_mode=self._runtime_mode,
runtime_profile=self._runtime_profile or get_default_runtime_profile("higgs_audio"),
)
self.node.current_model_name = model_name
print(f"✅ Higgs Audio adapter: Loaded model '{model_name}' on {device} via shared runtime")
return
# Initialize the Higgs Audio engine with CUDA Graph setting
self.higgs_engine.initialize_engine(
@@ -138,7 +160,7 @@ class HiggsAudioEngineAdapter:
top_p = params.get("top_p", 0.95)
top_k = params.get("top_k", 50)
max_new_tokens = params.get("max_new_tokens", 2048)
force_audio_gen = params.get("force_audio_gen", False)
force_audio_gen = params.get("force_audio_gen", True)
ras_win_len = params.get("ras_win_len", 7) # Default from boson_multimodal
ras_max_num_repeat = params.get("ras_max_num_repeat", 2) # Default from boson_multimodal
enable_cuda_graphs = params.get("enable_cuda_graphs", True) # CUDA Graph toggle
@@ -161,7 +183,7 @@ class HiggsAudioEngineAdapter:
max_tokens_per_chunk = int(max_chars_per_chunk / 3.5)
# Initialize engine if not already done
if not self.higgs_engine.engine:
if not getattr(self.higgs_engine, "engine", None):
print(f"🚀 Initializing Higgs Audio engine with model: {model_name}")
self.load_base_model(model_name, device)
@@ -394,7 +416,7 @@ class HiggsAudioEngineAdapter:
validated["max_new_tokens"] = max(1, min(4096, int(max_tokens)))
# Force audio gen validation
force_audio_gen = params.get("force_audio_gen", False)
force_audio_gen = params.get("force_audio_gen", True)
validated["force_audio_gen"] = bool(force_audio_gen)
# RAS parameters validation
@@ -426,4 +448,4 @@ class HiggsAudioEngineAdapter:
# Cache settings
validated["enable_audio_cache"] = params.get("enable_audio_cache", True)
return validated
return validated
+206
View File
@@ -0,0 +1,206 @@
"""Higgs Audio v3 adapter for the unified TTS pipeline."""
from __future__ import annotations
import os
import sys
from typing import Any, Dict, Optional, Tuple
import torch
current_dir = os.path.dirname(__file__)
engines_dir = os.path.dirname(current_dir)
project_root = os.path.dirname(engines_dir)
if project_root not in sys.path:
sys.path.insert(0, project_root)
from utils.audio.audio_hash import generate_stable_audio_component
from utils.audio.cache import get_audio_cache
from utils.models.factory_config import ModelLoadConfig
from utils.text.pause_processor import PauseTagProcessor
class HiggsAudioV3EngineAdapter:
"""Adapter for native Higgs Audio v3 generation."""
SAMPLE_RATE = 24000
def __init__(self, node_instance=None):
self.node = node_instance
self.audio_cache = get_audio_cache()
self._last_config: Optional[ModelLoadConfig] = None
def load_model(
self,
model: str,
device: str = "auto",
dtype: str = "auto",
attention: str = "auto",
):
from utils.models.unified_model_interface import unified_model_interface
config = ModelLoadConfig(
engine_name="higgs_audio_v3",
model_type="tts",
model_name=model,
model_path=model,
device=device,
additional_params={
"dtype": dtype,
"attention": attention,
},
)
self._last_config = config
unified_model_interface.load_model(config)
def update_model_config(
self,
model: str,
device: str = "auto",
dtype: str = "auto",
attention: str = "auto",
):
self.load_model(model, device, dtype, attention)
def _get_engine(self):
if self._last_config is None:
raise RuntimeError("Higgs Audio v3 model has not been loaded")
from utils.models.unified_model_interface import unified_model_interface
return unified_model_interface.load_model(self._last_config)
def generate_with_pause_tags(
self,
text: str,
voice_ref: Optional[Dict[str, Any]],
params: Dict[str, Any],
process_pauses: bool = True,
character_name: Optional[str] = None,
) -> torch.Tensor:
engine = self._get_engine()
if process_pauses and PauseTagProcessor.has_pause_tags(text):
return self._generate_with_pauses(text, voice_ref, params, character_name, engine)
return self._generate_direct(text, voice_ref, params, character_name, engine)
def _generate_direct(
self,
text: str,
voice_ref: Optional[Dict[str, Any]],
params: Dict[str, Any],
character_name: Optional[str] = None,
engine=None,
) -> torch.Tensor:
if engine is None:
engine = self._get_engine()
reference_audio, reference_text, audio_component = self._extract_voice_reference(voice_ref)
model_name = params.get("model", "higgs-audio-v3-tts-4b")
seed = int(params.get("seed", 0) or 0)
max_new_tokens = int(params.get("max_new_tokens", 2048))
temperature = float(params.get("temperature", 1.0))
top_p = float(params.get("top_p", 0.95))
top_k = int(params.get("top_k", 50))
cache_key = self.audio_cache.generate_cache_key(
"higgs_audio_v3",
text=text,
model_variant=model_name,
audio_component=audio_component,
reference_text=reference_text,
temperature=temperature,
top_p=top_p,
top_k=top_k,
max_new_tokens=max_new_tokens,
seed=seed,
device=params.get("device", "auto"),
dtype=params.get("dtype", "auto"),
attention=params.get("attention", "auto"),
character=character_name or "narrator",
)
if params.get("enable_audio_cache", True):
cached_audio = self.audio_cache.get_cached_audio(cache_key)
if cached_audio:
print(f"💾 Using cached Higgs Audio v3 audio for '{character_name or 'narrator'}': '{text[:30]}...'")
return cached_audio[0]
audio_tensor, sample_rate = engine.generate(
text=text,
reference_audio=reference_audio,
reference_text=reference_text,
max_new_tokens=max_new_tokens,
temperature=temperature,
top_p=top_p,
top_k=top_k,
seed=seed,
)
if sample_rate != self.SAMPLE_RATE:
raise RuntimeError(f"Higgs Audio v3 returned {sample_rate} Hz; expected {self.SAMPLE_RATE} Hz")
if params.get("enable_audio_cache", True):
duration = self.audio_cache._calculate_duration(audio_tensor, "higgs_audio_v3")
self.audio_cache.cache_audio(cache_key, audio_tensor, duration)
return audio_tensor
def _generate_with_pauses(
self,
text: str,
voice_ref: Optional[Dict[str, Any]],
params: Dict[str, Any],
character_name: Optional[str] = None,
engine=None,
) -> torch.Tensor:
segments, _ = PauseTagProcessor.parse_pause_tags(text)
audio_parts = []
for segment_type, content in segments:
if segment_type == "text":
audio_tensor = self._generate_direct(content, voice_ref, params, character_name, engine)
if audio_tensor.dim() == 1:
audio_tensor = audio_tensor.unsqueeze(0)
elif audio_tensor.dim() == 3:
audio_tensor = audio_tensor.squeeze(0)
audio_parts.append(audio_tensor)
elif segment_type == "pause":
audio_parts.append(PauseTagProcessor.create_silence_segment(content, self.SAMPLE_RATE))
if not audio_parts:
return torch.zeros(1, 0)
return torch.cat(audio_parts, dim=-1)
def _extract_voice_reference(self, voice_ref: Optional[Dict[str, Any]]) -> Tuple[Optional[Dict[str, Any]], str, str]:
if not voice_ref or not isinstance(voice_ref, dict):
return None, "", "zero_shot"
ref_audio = None
for key in ("audio", "audio_dict", "reference_audio", "waveform"):
if key in voice_ref and voice_ref.get(key) is not None:
ref_audio = voice_ref.get(key)
break
reference_text = (
voice_ref.get("reference_text")
or voice_ref.get("text")
or voice_ref.get("prompt_text")
or ""
)
if ref_audio is None and "audio_path" in voice_ref:
audio_path = voice_ref.get("audio_path")
if audio_path and os.path.exists(audio_path):
try:
from utils.audio.processing import AudioProcessingUtils
waveform, sample_rate = AudioProcessingUtils.safe_load_audio(audio_path)
ref_audio = {"waveform": waveform, "sample_rate": sample_rate}
except Exception as e:
print(f"⚠️ Failed to load Higgs Audio v3 reference audio {audio_path}: {e}")
if isinstance(ref_audio, dict) and "waveform" in ref_audio:
return ref_audio, str(reference_text), generate_stable_audio_component(reference_audio=ref_audio)
if torch.is_tensor(ref_audio):
sample_rate = int(voice_ref.get("sample_rate", self.SAMPLE_RATE))
audio_dict = {"waveform": ref_audio, "sample_rate": sample_rate}
return audio_dict, str(reference_text), generate_stable_audio_component(reference_audio=audio_dict)
return None, str(reference_text), "zero_shot"
+126 -71
View File
@@ -12,7 +12,8 @@ from typing import Dict, Any, Optional, List, Union
from engines.index_tts.index_tts import IndexTTSEngine
from engines.index_tts.index_tts_downloader import index_tts_downloader
from utils.text.character_parser import character_parser
from utils.voice.discovery import get_character_mapping, get_available_characters
from utils.voice.discovery import get_character_mapping, get_available_characters
from utils.voice.character_logging import resolved_character_label
from utils.audio.cache import get_audio_cache
@@ -76,7 +77,26 @@ class IndexTTSAdapter:
)
def generate(self,
@staticmethod
def _normalize_emotion_audio(emotion_audio):
"""Return a path or waveform dict accepted by IndexTTS-2."""
if not isinstance(emotion_audio, dict):
return emotion_audio
if emotion_audio.get("audio_path"):
print(
"🎭 Using Character Voices emotion audio: "
f"{emotion_audio.get('character_name', 'unknown')} -> "
f"{emotion_audio['audio_path']}"
)
return emotion_audio["audio_path"]
if "waveform" in emotion_audio:
return emotion_audio
nested_audio = emotion_audio.get("audio")
if isinstance(nested_audio, dict) and "waveform" in nested_audio:
return nested_audio
return emotion_audio
def generate(self,
text: str,
speaker_audio: Optional[str] = None,
emotion_audio: Optional[str] = None,
@@ -93,9 +113,12 @@ class IndexTTSAdapter:
top_k: int = 30,
length_penalty: float = 0.0,
num_beams: int = 3,
repetition_penalty: float = 10.0,
max_mel_tokens: int = 1500,
# Streaming parameters
repetition_penalty: float = 10.0,
max_mel_tokens: int = 1500,
language: str = "English",
duration_factor: float = 1.0,
text_normalization: bool = True,
# Streaming parameters
stream_return: bool = False,
more_segment_before: int = 0,
**kwargs) -> torch.Tensor:
@@ -119,7 +142,10 @@ class IndexTTSAdapter:
length_penalty: Length penalty for beam search
num_beams: Number of beams for beam search
repetition_penalty: Repetition penalty
max_mel_tokens: Maximum mel tokens to generate
max_mel_tokens: Maximum mel tokens to generate
language: IndexTTS-2.5 language code/name
duration_factor: Official 2.5 internal feature-duration multiplier
text_normalization: Enable multilingual text normalization
**kwargs: Additional parameters
Returns:
@@ -135,9 +161,31 @@ class IndexTTSAdapter:
# Parse character switching tags with emotion support
processed_segments = self._process_character_tags_with_emotions(text)
if len(processed_segments) > 1:
# Multi-segment character switching - process each segment separately
return self._generate_multi_character_segments(processed_segments, speaker_audio, emotion_audio, **kwargs)
if len(processed_segments) > 1:
# Multi-segment character switching - process each segment separately
return self._generate_multi_character_segments(
processed_segments, speaker_audio, emotion_audio,
emotion_alpha=emotion_alpha,
emotion_vector=emotion_vector,
use_emotion_text=use_emotion_text,
emotion_text=emotion_text,
use_random=use_random,
interval_silence=interval_silence,
max_text_tokens_per_segment=max_text_tokens_per_segment,
temperature=temperature,
top_p=top_p,
top_k=top_k,
length_penalty=length_penalty,
num_beams=num_beams,
repetition_penalty=repetition_penalty,
max_mel_tokens=max_mel_tokens,
language=language,
duration_factor=duration_factor,
text_normalization=text_normalization,
stream_return=stream_return,
more_segment_before=more_segment_before,
**kwargs,
)
elif processed_segments:
# Single character segment
first_segment = processed_segments[0]
@@ -157,19 +205,10 @@ class IndexTTSAdapter:
# Determine final speaker and emotion audio
final_speaker_audio = speaker_audio
# Handle Character Voices emotion_audio format
if emotion_audio and isinstance(emotion_audio, dict):
if "audio_path" in emotion_audio:
# Character Voices format: {'audio': {...}, 'audio_path': 'path', ...}
final_emotion_audio = emotion_audio["audio_path"]
print(f"🎭 Using Character Voices emotion audio: {emotion_audio.get('character_name', 'unknown')} -> {final_emotion_audio}")
elif "waveform" in emotion_audio:
# Direct AUDIO format: {'waveform': tensor, 'sample_rate': rate}
final_emotion_audio = emotion_audio
else:
final_emotion_audio = emotion_audio
else:
final_emotion_audio = emotion_audio
# Normalize the two supported audio-reference shapes:
# Character Voices returns {audio: {waveform, sample_rate}, audio_path: ...},
# while ComfyUI AUDIO returns {waveform, sample_rate} directly.
final_emotion_audio = self._normalize_emotion_audio(emotion_audio)
# Only do character mapping if we actually have character tags
if has_character_tags:
@@ -221,9 +260,12 @@ class IndexTTSAdapter:
length_penalty=length_penalty,
repetition_penalty=repetition_penalty,
max_mel_tokens=max_mel_tokens,
max_text_tokens_per_segment=max_text_tokens_per_segment,
interval_silence=interval_silence,
stream_return=stream_return,
max_text_tokens_per_segment=max_text_tokens_per_segment,
interval_silence=interval_silence,
language=language,
duration_factor=duration_factor,
text_normalization=text_normalization,
stream_return=stream_return,
more_segment_before=more_segment_before,
**kwargs # Include seed and other kwargs in cache key
)
@@ -263,18 +305,11 @@ class IndexTTSAdapter:
engine_kwargs['stream_return'] = stream_return
engine_kwargs['more_segment_before'] = more_segment_before
# Apply consistent emotion priority: emotion_audio takes precedence over other emotion controls
# This ensures consistent behavior whether using character tags or direct engine inputs
if final_emotion_audio:
# emotion_audio connected - disable other emotion controls
final_emotion_vector = None
final_use_emotion_text = False
final_emotion_text = None
else:
# No emotion_audio - use provided emotion controls
final_emotion_vector = emotion_vector
final_use_emotion_text = use_emotion_text
final_emotion_text = emotion_text
# Audio emotion and vector/text emotion are independent conditioning
# sources. IndexTTS-2 blends them in its latent emotion space.
final_emotion_vector = emotion_vector
final_use_emotion_text = use_emotion_text
final_emotion_text = emotion_text
# Generate audio with OOM protection
try:
@@ -294,9 +329,12 @@ class IndexTTSAdapter:
top_k=top_k,
length_penalty=length_penalty,
num_beams=num_beams,
repetition_penalty=repetition_penalty,
max_mel_tokens=max_mel_tokens,
**engine_kwargs
repetition_penalty=repetition_penalty,
max_mel_tokens=max_mel_tokens,
language=language,
duration_factor=duration_factor,
text_normalization=text_normalization,
**engine_kwargs
)
except torch.OutOfMemoryError as e:
# Analyze audio after OOM to provide helpful feedback
@@ -376,7 +414,7 @@ class IndexTTSAdapter:
Returns:
Combined audio tensor [1, samples] at 22050 Hz
"""
audio_segments = []
audio_segments = []
# Get character mapping for all unique characters
unique_characters = set()
@@ -391,12 +429,22 @@ class IndexTTSAdapter:
if unique_characters:
character_mapping = get_character_mapping(list(unique_characters), engine_type="index_tts")
print(f"🎭 IndexTTS-2: Processing {len(segments)} character segment(s) - {', '.join([s.get('character', 'narrator') for s in segments])}")
resolved_names = [
resolved_character_label(
segment.get('character', 'narrator'),
character_mapping.get(segment.get('character', 'narrator'), (default_speaker_audio, None)),
)
for segment in segments
]
print(f"🎭 IndexTTS-2: Processing {len(segments)} character segment(s) - {', '.join(resolved_names)}")
for segment in segments:
character_name = segment.get('character', 'narrator')
segment_text = segment.get('text', '').strip()
emotion_ref = segment.get('emotion')
emotion_ref = segment.get('emotion')
segment_kwargs = dict(kwargs)
if segment.get('language'):
segment_kwargs['language'] = segment['language']
if not segment_text:
continue
@@ -407,7 +455,7 @@ class IndexTTSAdapter:
character_audio_path = character_mapping[character_name][0]
if character_audio_path:
speaker_audio = character_audio_path
print(f"📖 Using character voice '{character_name}' | Ref: '{speaker_audio}'")
print(f"📖 Using character voice '{resolved_character_label(character_name, speaker_audio)}' | Ref: '{speaker_audio}'")
else:
print(f"⚠️ Character '{character_name}' has no audio reference, using default")
@@ -422,24 +470,24 @@ class IndexTTSAdapter:
# Generate cache key for this segment
segment_cache_key = self._generate_cache_key(
text=segment_text,
speaker_audio=speaker_audio,
emotion_audio=emotion_audio,
**kwargs
speaker_audio=speaker_audio,
emotion_audio=emotion_audio,
**segment_kwargs
)
# Check cache first
cached_segment_audio = self.audio_cache.get_cached_audio(segment_cache_key)
if cached_segment_audio:
print(f"💾 Using cached IndexTTS-2 segment for '{character_name}': '{segment_text[:30]}...'")
print(f"💾 Using cached IndexTTS-2 segment for '{resolved_character_label(character_name, speaker_audio)}': '{segment_text[:30]}...'")
segment_audio = cached_segment_audio[0]
else:
# Generate audio for this segment with OOM protection
try:
segment_audio = self.engine.generate(
text=segment_text,
speaker_audio=speaker_audio,
emotion_audio=emotion_audio,
**kwargs
speaker_audio=speaker_audio,
emotion_audio=emotion_audio,
**segment_kwargs
)
except torch.OutOfMemoryError as e:
# Analyze audio after OOM in multi-character segments
@@ -463,9 +511,18 @@ class IndexTTSAdapter:
# Return silence if no segments generated
return torch.zeros(1, 22050, dtype=torch.float32)
def _generate_cache_key(self, **params) -> str:
"""Generate cache key for IndexTTS-2."""
return self.audio_cache.generate_cache_key('index_tts', **params)
def _generate_cache_key(self, **params) -> str:
"""Generate cache key for IndexTTS-2."""
model_identity = {}
if self.engine is not None:
model_identity = {
"model_name": getattr(self.engine, "model_name", None),
"model_version": getattr(self.engine, "model_version", None),
"model_path": getattr(self.engine, "model_dir", None),
}
return self.audio_cache.generate_cache_key(
'index_tts', **model_identity, **params
)
def _analyze_audio_after_oom(self, speaker_audio: str, emotion_audio: str, max_mel_tokens: int) -> str:
"""
@@ -534,16 +591,18 @@ class IndexTTSAdapter:
return "\n".join(analysis_parts)
def _get_stable_audio_identifier(self, audio_path: str) -> str:
"""
Get stable identifier for audio file using centralized audio hashing.
"""
if not audio_path:
return audio_path
# Use our centralized audio hashing utility
from utils.audio.audio_hash import generate_stable_audio_component
return generate_stable_audio_component(audio_file_path=audio_path)
def _get_stable_audio_identifier(self, audio_path: str) -> str:
"""
Get stable identifier for audio file using centralized audio hashing.
"""
if not audio_path:
return audio_path
# Use our centralized audio hashing utility
from utils.audio.audio_hash import generate_stable_audio_component
if isinstance(audio_path, dict):
return generate_stable_audio_component(reference_audio=audio_path)
return generate_stable_audio_component(audio_file_path=audio_path)
def get_supported_formats(self) -> List[str]:
"""Get supported audio formats."""
@@ -580,10 +639,6 @@ class IndexTTSAdapter:
def unload(self):
"""Unload the engine to free memory."""
if self.engine:
self.engine.unload()
self.engine = None
def __del__(self):
"""Cleanup on deletion."""
self.unload()
if self.engine:
self.engine.unload()
self.engine = None
@@ -0,0 +1,69 @@
"""Suite adapter for MOSS-SoundEffect v2."""
from typing import Any, Dict, Tuple
import torch
from engines.moss_soundeffect_v2.downloader import MossSoundEffectV2Downloader
from utils.audio.cache import get_audio_cache
from utils.models.factory_config import ModelLoadConfig
from utils.models.unified_model_interface import unified_model_interface
class MossSoundEffectV2Adapter:
def __init__(self, config: Dict[str, Any]):
self.config = dict(config)
self.audio_cache = get_audio_cache()
self._load_config = None
def _load(self):
model = self.config.get("model", MossSoundEffectV2Downloader.MODEL_NAME)
model_path = MossSoundEffectV2Downloader().resolve_model_path(model)
self._load_config = ModelLoadConfig(
engine_name="moss_soundeffect_v2",
model_type="tts",
model_name=str(model).removeprefix("local:"),
model_path=model_path,
device=self.config.get("device", "auto"),
runtime_mode="main_environment",
additional_params={"dtype": self.config.get("dtype", "auto")},
)
return unified_model_interface.load_model(self._load_config)
def generate(
self,
description: str,
duration_seconds: float,
seed: int,
enable_audio_cache: bool,
) -> Tuple[torch.Tensor, int, bool]:
duration_seconds = round(float(duration_seconds), 1)
if duration_seconds > 30.0:
raise ValueError(
"MOSS-SoundEffect v2 supports a maximum duration of 30 seconds. "
"Lower duration_seconds in the 🌩️ Sound Effects node."
)
params = {
"description": description,
"model": self.config.get("model"),
"duration_seconds": duration_seconds,
"inference_steps": self.config.get("inference_steps", 100),
"cfg_scale": self.config.get("cfg_scale", 4.0),
"sigma_shift": self.config.get("sigma_shift", 5.0),
"negative_prompt": self.config.get("negative_prompt", ""),
"seed": int(seed),
"dtype": self.config.get("dtype", "auto"),
"device": self.config.get("device", "auto"),
}
cache_key = self.audio_cache.generate_cache_key("moss_soundeffect_v2", **params)
if enable_audio_cache:
cached = self.audio_cache.get_cached_audio(cache_key)
if cached is not None:
return cached[0], 48000, True
engine = self._load()
waveform, sample_rate = engine.generate_sound_effect(**params)
waveform = waveform.detach().cpu().float()
if enable_audio_cache:
self.audio_cache.cache_audio(cache_key, waveform, waveform.shape[-1] / float(sample_rate))
return waveform, int(sample_rate), False
+392
View File
@@ -0,0 +1,392 @@
"""
MOSS-TTS engine adapter.
Translates TTS Audio Suite processor calls into official MOSS-TTS inference
without exposing non-native controls.
"""
import os
import sys
from typing import Any, Dict, List, Optional, Tuple
import torch
current_dir = os.path.dirname(__file__)
engines_dir = os.path.dirname(current_dir)
project_root = os.path.dirname(engines_dir)
if project_root not in sys.path:
sys.path.insert(0, project_root)
from utils.audio.audio_hash import generate_stable_audio_component
from utils.audio.cache import get_audio_cache
from utils.models.factory_config import ModelLoadConfig
from utils.text.pause_processor import PauseTagProcessor
from utils.voice.reference import effective_voice_audio
class MossTTSEngineAdapter:
"""Adapter for the official MOSS-TTS model API."""
SAMPLE_RATE = 24000
def __init__(self, node_instance=None):
self.node = node_instance
self.audio_cache = get_audio_cache()
self._last_config: Optional[ModelLoadConfig] = None
def load_model(
self,
model_variant: str,
device: str = "auto",
dtype: str = "auto",
attn_implementation: str = "auto",
codec_model: str = "MOSS-Audio-Tokenizer",
lora_adapter: Optional[str] = None,
defer_load: bool = False,
quantization: str = "none",
):
from utils.models.unified_model_interface import unified_model_interface
config = ModelLoadConfig(
engine_name="moss_tts",
model_type="tts",
model_name=model_variant,
model_path=model_variant,
device=device,
additional_params={
"dtype": dtype,
"attn_implementation": attn_implementation,
"codec_model": codec_model,
"lora_adapter": lora_adapter,
"quantization": quantization,
},
)
self._last_config = config
if not defer_load:
unified_model_interface.load_model(config)
def update_model_config(
self,
model_variant: str,
device: str = "auto",
dtype: str = "auto",
attn_implementation: str = "auto",
codec_model: str = "MOSS-Audio-Tokenizer",
lora_adapter: Optional[str] = None,
quantization: str = "none",
):
self.load_model(
model_variant, device, dtype, attn_implementation, codec_model, lora_adapter,
quantization=quantization,
)
def _get_engine(self):
if self._last_config is None:
raise RuntimeError("MOSS-TTS model has not been loaded")
from utils.models.unified_model_interface import unified_model_interface
return unified_model_interface.load_model(self._last_config)
def generate_with_pause_tags(
self,
text: str,
voice_ref: Optional[Dict[str, Any]],
params: Dict[str, Any],
process_pauses: bool = True,
character_name: Optional[str] = None,
) -> torch.Tensor:
engine = self._get_engine()
seed = int(params.get("seed", 0) or 0)
if seed > 0:
torch.manual_seed(seed)
if torch.cuda.is_available():
torch.cuda.manual_seed_all(seed)
if process_pauses and PauseTagProcessor.has_pause_tags(text):
return self._generate_with_pauses(text, voice_ref, params, character_name, engine)
return self._generate_direct(text, voice_ref, params, character_name, engine)
def _generate_direct(
self,
text: str,
voice_ref: Optional[Dict[str, Any]],
params: Dict[str, Any],
character_name: Optional[str] = None,
engine=None,
) -> torch.Tensor:
reference_audio, reference_sample_rate, audio_component = self._extract_voice_reference(voice_ref)
model_variant = params.get("model_variant", "MOSS-TTS-Local-Transformer")
language = params.get("language", "auto")
explicit_duration = params.get("duration_tokens")
duration_tokens = max(1, int(explicit_duration)) if explicit_duration else None
resolved_max_new_tokens = int(params.get("max_new_tokens", 4096))
n_vq_for_inference = params.get("n_vq_for_inference")
lora_adapter = params.get("lora_adapter")
instruction = params.get("instruction")
quality = params.get("quality")
sound_event = params.get("sound_event")
ambient_sound = params.get("ambient_sound")
cache_key = self.audio_cache.generate_cache_key(
"moss_tts",
text=text,
model_variant=model_variant,
audio_component=audio_component,
language=language,
instruction=instruction,
quality=quality,
sound_event=sound_event,
ambient_sound=ambient_sound,
duration_tokens=duration_tokens,
audio_temperature=params.get("audio_temperature", params.get("temperature", 1.0)),
audio_top_p=params.get("audio_top_p", params.get("top_p", 0.95)),
audio_top_k=params.get("audio_top_k", params.get("top_k", 50)),
audio_repetition_penalty=params.get("audio_repetition_penalty", params.get("repetition_penalty", 1.1)),
max_new_tokens=resolved_max_new_tokens,
n_vq_for_inference=n_vq_for_inference,
seed=params.get("seed", 0),
device=params.get("device", "auto"),
dtype=params.get("dtype", "auto"),
attn_implementation=params.get("attn_implementation", "auto"),
lora_adapter=lora_adapter,
quantization=params.get("quantization", "none"),
character=character_name or "narrator",
)
enable_audio_cache = bool(params.get("enable_audio_cache", True))
cached_audio = self.audio_cache.get_cached_audio(cache_key) if enable_audio_cache else None
if cached_audio:
print(f"💾 Using cached MOSS-TTS audio for '{character_name or 'narrator'}': '{text[:30]}...'")
return cached_audio[0]
if engine is None:
engine = self._get_engine()
audio_tensor, sample_rate = engine.generate(
text=text,
reference_audio=reference_audio,
reference_sample_rate=reference_sample_rate,
language=language,
duration_tokens=duration_tokens,
instruction=instruction,
quality=quality,
sound_event=sound_event,
ambient_sound=ambient_sound,
seed=int(params.get("seed", 0) or 0),
audio_temperature=float(params.get("audio_temperature", params.get("temperature", 1.0))),
audio_top_p=float(params.get("audio_top_p", params.get("top_p", 0.95))),
audio_top_k=int(params.get("audio_top_k", params.get("top_k", 50))),
audio_repetition_penalty=float(
params.get("audio_repetition_penalty", params.get("repetition_penalty", 1.1))
),
max_new_tokens=int(resolved_max_new_tokens),
n_vq_for_inference=n_vq_for_inference,
)
if sample_rate != self.SAMPLE_RATE:
raise RuntimeError(f"MOSS-TTS returned unexpected sample rate {sample_rate}; expected {self.SAMPLE_RATE}")
if enable_audio_cache:
duration = self.audio_cache._calculate_duration(audio_tensor, "moss_tts")
self.audio_cache.cache_audio(cache_key, audio_tensor, duration)
return audio_tensor
def generate_sound_effect(self, description: str, params: Dict[str, Any]) -> torch.Tensor:
"""Generate MOSS-SoundEffect v1 audio without a speech or narrator reference."""
sound_params = dict(params)
sound_params["ambient_sound"] = str(description or "").strip()
if not sound_params["ambient_sound"]:
raise ValueError("MOSS-SoundEffect requires a non-empty description")
return self._generate_direct(
text="",
voice_ref=None,
params=sound_params,
character_name="sound_effects",
)
def _generate_with_pauses(
self,
text: str,
voice_ref: Optional[Dict[str, Any]],
params: Dict[str, Any],
character_name: Optional[str] = None,
engine=None,
) -> torch.Tensor:
segments, _ = PauseTagProcessor.parse_pause_tags(text)
audio_parts = []
for segment_type, content in segments:
if segment_type == "text":
audio_tensor = self._generate_direct(content, voice_ref, params, character_name, engine)
if audio_tensor.dim() == 1:
audio_tensor = audio_tensor.unsqueeze(0)
elif audio_tensor.dim() == 3:
audio_tensor = audio_tensor.squeeze(0)
audio_parts.append(audio_tensor)
elif segment_type == "pause":
audio_parts.append(PauseTagProcessor.create_silence_segment(content, self.SAMPLE_RATE))
if not audio_parts:
return torch.zeros(1, 0)
return torch.cat(audio_parts, dim=-1)
def _extract_voice_reference(self, voice_ref: Optional[Dict[str, Any]]) -> Tuple[Any, Optional[int], str]:
if not voice_ref or not isinstance(voice_ref, dict):
return None, None, "default_voice"
ref_audio = effective_voice_audio(voice_ref)
if ref_audio is None:
return None, None, "default_voice"
if isinstance(ref_audio, str):
return ref_audio, None, generate_stable_audio_component(audio_file_path=ref_audio)
if isinstance(ref_audio, dict) and "waveform" in ref_audio:
return (
ref_audio,
int(ref_audio.get("sample_rate", self.SAMPLE_RATE)),
generate_stable_audio_component(reference_audio=ref_audio),
)
if torch.is_tensor(ref_audio):
sample_rate = int(voice_ref.get("sample_rate", self.SAMPLE_RATE))
audio_dict = {"waveform": ref_audio, "sample_rate": sample_rate}
return audio_dict, sample_rate, generate_stable_audio_component(reference_audio=audio_dict)
return ref_audio, voice_ref.get("sample_rate"), "custom_voice"
def _normalize_dialogue_voice_ref(self, voice_ref: Any, speaker_label: str) -> Tuple[Optional[Dict[str, Any]], str]:
if voice_ref is None:
return None, "no_voice"
if isinstance(voice_ref, str):
print(
f"❌ MOSS-TTSD {speaker_label}: received character name '{voice_ref}' instead of voice reference. "
"Connect Character Voices opt_narrator, not character_name."
)
return None, "invalid_string_voice"
if not isinstance(voice_ref, dict):
print(f"❌ MOSS-TTSD {speaker_label}: invalid voice reference type {type(voice_ref).__name__}")
return None, "invalid_voice"
ref_audio = effective_voice_audio(voice_ref)
reference_text = (
voice_ref.get("reference_text")
or voice_ref.get("text")
or voice_ref.get("prompt_text")
or ""
)
if ref_audio is None:
return None, "no_audio"
normalized = {
"audio": ref_audio,
"sample_rate": voice_ref.get("sample_rate", self.SAMPLE_RATE),
"reference_text": str(reference_text or "").strip(),
}
if isinstance(ref_audio, str):
audio_component = generate_stable_audio_component(audio_file_path=ref_audio)
elif isinstance(ref_audio, dict) and "waveform" in ref_audio:
audio_component = generate_stable_audio_component(reference_audio=ref_audio)
normalized["sample_rate"] = int(ref_audio.get("sample_rate", self.SAMPLE_RATE))
elif torch.is_tensor(ref_audio):
audio_component = generate_stable_audio_component(
reference_audio={
"waveform": ref_audio,
"sample_rate": normalized["sample_rate"],
}
)
else:
audio_component = "custom_voice"
if not normalized["reference_text"]:
print(f"⚠️ MOSS-TTSD {speaker_label}: reference audio has no transcript; cloning disabled for this speaker")
return None, audio_component
return normalized, audio_component
def generate_native_dialogue(
self,
dialogue_text: str,
speaker_voices: List[Any],
params: Dict[str, Any],
) -> Dict[str, Any]:
engine = self._get_engine()
normalized_refs: List[Optional[Dict[str, Any]]] = []
audio_components = []
for idx, voice_ref in enumerate(speaker_voices):
normalized, audio_component = self._normalize_dialogue_voice_ref(voice_ref, f"S{idx + 1}")
normalized_refs.append(normalized)
audio_components.append(audio_component)
model_variant = params.get("model_variant", "MOSS-TTSD-v1.0")
cache_key = self.audio_cache.generate_cache_key(
"moss_tts",
text=dialogue_text,
model_variant=model_variant,
audio_component="ttsd_" + "_".join(audio_components),
language=params.get("language", "auto"),
instruction=params.get("instruction"),
quality=params.get("quality"),
sound_event=params.get("sound_event"),
ambient_sound=params.get("ambient_sound"),
duration_tokens=params.get("duration_tokens"),
audio_temperature=params.get("audio_temperature", params.get("temperature", 1.1)),
audio_top_p=params.get("audio_top_p", params.get("top_p", 0.9)),
audio_top_k=params.get("audio_top_k", params.get("top_k", 50)),
audio_repetition_penalty=params.get("audio_repetition_penalty", params.get("repetition_penalty", 1.1)),
max_new_tokens=params.get("max_new_tokens", 4096),
seed=params.get("seed", 0),
device=params.get("device", "auto"),
dtype=params.get("dtype", "auto"),
attn_implementation=params.get("attn_implementation", "auto"),
character="native_multispeaker",
quantization=params.get("quantization", "none"),
)
cached_audio = self.audio_cache.get_cached_audio(cache_key)
if cached_audio:
print("💾 Using cached MOSS-TTSD native dialogue audio")
audio_tensor = cached_audio[0]
else:
audio_tensor, sample_rate = engine.generate_dialogue(
dialogue_text=dialogue_text,
speaker_references=normalized_refs,
language=params.get("language", "auto"),
duration_tokens=params.get("duration_tokens"),
instruction=params.get("instruction"),
quality=params.get("quality"),
sound_event=params.get("sound_event"),
ambient_sound=params.get("ambient_sound"),
seed=int(params.get("seed", 0) or 0),
audio_temperature=float(params.get("audio_temperature", params.get("temperature", 1.1))),
audio_top_p=float(params.get("audio_top_p", params.get("top_p", 0.9))),
audio_top_k=int(params.get("audio_top_k", params.get("top_k", 50))),
audio_repetition_penalty=float(
params.get("audio_repetition_penalty", params.get("repetition_penalty", 1.1))
),
max_new_tokens=int(params.get("max_new_tokens", 4096)),
)
if sample_rate != self.SAMPLE_RATE:
raise RuntimeError(f"MOSS-TTSD returned unexpected sample rate {sample_rate}; expected {self.SAMPLE_RATE}")
duration = self.audio_cache._calculate_duration(audio_tensor, "moss_tts")
self.audio_cache.cache_audio(cache_key, audio_tensor, duration)
if audio_tensor.dim() == 1:
audio_tensor = audio_tensor.unsqueeze(0)
elif audio_tensor.dim() == 3:
audio_tensor = audio_tensor.squeeze(0)
return {
"waveform": audio_tensor,
"sample_rate": self.SAMPLE_RATE,
"character": "native_multispeaker",
"text": dialogue_text,
}
def cleanup(self):
self._last_config = None
+299
View File
@@ -0,0 +1,299 @@
"""
OmniVoice engine adapter.
Translates TTS Audio Suite processor calls into the official OmniVoice runtime.
"""
import os
import sys
from typing import Any, Dict, Optional, Tuple, Union
import torch
current_dir = os.path.dirname(__file__)
engines_dir = os.path.dirname(current_dir)
project_root = os.path.dirname(engines_dir)
if project_root not in sys.path:
sys.path.insert(0, project_root)
from utils.audio.audio_hash import generate_stable_audio_component
from utils.audio.cache import get_audio_cache
from utils.models.factory_config import ModelLoadConfig
from utils.models.language_mapper import resolve_language_alias
from utils.voice.reference import effective_voice_audio
class OmniVoiceEngineAdapter:
"""Adapter for official OmniVoice inference."""
SAMPLE_RATE = 24000
def __init__(self, config: Optional[Dict[str, Any]] = None):
self.config = config.copy() if config else {}
self.audio_cache = get_audio_cache()
self._last_config: Optional[ModelLoadConfig] = None
self._load_signature = None
def update_config(self, new_config: Dict[str, Any]):
self.config = new_config.copy() if new_config else {}
def _build_load_signature(self) -> Tuple[Any, ...]:
return (
self.config.get("model_variant", "OmniVoice"),
self.config.get("device", "auto"),
self.config.get("dtype", "auto"),
)
def load_model(
self,
model_variant: str,
device: str = "auto",
dtype: str = "auto",
):
from utils.models.unified_model_interface import unified_model_interface
config = ModelLoadConfig(
engine_name="omnivoice",
model_type="tts",
model_name=model_variant,
model_path=model_variant,
device=device,
additional_params={
"dtype": dtype,
},
)
self._last_config = config
unified_model_interface.load_model(config)
def _ensure_model_loaded(self):
signature = self._build_load_signature()
if signature == self._load_signature and self._last_config is not None:
return
self.load_model(
model_variant=self.config.get("model_variant", "OmniVoice"),
device=self.config.get("device", "auto"),
dtype=self.config.get("dtype", "auto"),
)
self._load_signature = signature
def _get_engine(self):
if self._last_config is None:
self._ensure_model_loaded()
from utils.models.unified_model_interface import unified_model_interface
return unified_model_interface.load_model(self._last_config)
@staticmethod
def _seed_everything(seed: int):
if int(seed or 0) <= 0:
return
torch.manual_seed(int(seed))
if torch.cuda.is_available():
torch.cuda.manual_seed_all(int(seed))
if hasattr(torch, "xpu") and torch.xpu.is_available():
torch.xpu.manual_seed_all(int(seed))
@staticmethod
def _first_non_none(*values):
for value in values:
if value is None:
continue
if isinstance(value, str) and not value.strip():
continue
return value
return None
@staticmethod
def _normalize_waveform_tensor(waveform: torch.Tensor) -> torch.Tensor:
if waveform.dim() == 3:
waveform = waveform.squeeze(0)
if waveform.dim() == 1:
waveform = waveform.unsqueeze(0)
return waveform.detach().float().cpu()
def _extract_voice_reference(
self,
voice_ref: Optional[Dict[str, Any]],
) -> Tuple[Optional[Union[str, Tuple[torch.Tensor, int]]], str, str]:
if not voice_ref or not isinstance(voice_ref, dict):
return None, "", "default_voice"
prompt_text = (
voice_ref.get("reference_text")
or voice_ref.get("prompt_text")
or voice_ref.get("text")
or ""
).strip()
ref_audio = effective_voice_audio(voice_ref)
if ref_audio is None:
return None, prompt_text, "default_voice"
if isinstance(ref_audio, str):
return ref_audio, prompt_text, generate_stable_audio_component(audio_file_path=ref_audio)
if isinstance(ref_audio, dict) and "waveform" in ref_audio:
waveform = self._normalize_waveform_tensor(ref_audio["waveform"])
sample_rate = int(ref_audio.get("sample_rate", self.SAMPLE_RATE))
audio_dict = {"waveform": waveform, "sample_rate": sample_rate}
return (
(waveform, sample_rate),
prompt_text,
generate_stable_audio_component(reference_audio=audio_dict),
)
if torch.is_tensor(ref_audio):
waveform = self._normalize_waveform_tensor(ref_audio)
sample_rate = int(voice_ref.get("sample_rate", self.SAMPLE_RATE))
audio_dict = {"waveform": waveform, "sample_rate": sample_rate}
return (
(waveform, sample_rate),
prompt_text,
generate_stable_audio_component(reference_audio=audio_dict),
)
raise TypeError(f"Unsupported OmniVoice reference type: {type(ref_audio)}")
@staticmethod
def _normalize_language(language: Optional[str]) -> Optional[str]:
if language is None:
return None
normalized = str(language).strip()
if not normalized:
return None
lowered = normalized.lower()
if lowered in {"auto", "none"}:
return None
resolved = resolve_language_alias(normalized)
# OmniVoice uses the base ISO code for Portuguese rather than regional
# variants accepted by other suite engines.
if resolved in {"pt", "pt-br", "pt-pt"}:
return "pt"
if resolved and resolved.lower() != lowered:
return resolved
return normalized
def generate_single(
self,
text: str,
voice_ref: Optional[Dict[str, Any]],
seed: int = 0,
enable_audio_cache: bool = True,
character_name: Optional[str] = None,
) -> torch.Tensor:
stripped = (text or "").strip()
if not stripped:
return torch.zeros(1, 0, dtype=torch.float32)
self._ensure_model_loaded()
engine = self._get_engine()
ref_audio, prompt_text, audio_component = self._extract_voice_reference(voice_ref)
language = self._normalize_language(self.config.get("language", "Auto"))
instruct = str(self.config.get("instruct", "") or "").strip() or None
model_variant = self.config.get("model_variant", "OmniVoice")
device = self.config.get("device", "auto")
dtype = self.config.get("dtype", "auto")
num_step = int(self.config.get("num_steps", 32))
guidance_scale = float(self.config.get("guidance_scale", 2.0))
t_shift = float(self.config.get("t_shift", 0.1))
speed = float(self.config.get("speed", 1.0))
duration_value = float(self.config.get("duration", 0.0))
duration = duration_value if duration_value > 0 else None
layer_penalty_factor = float(self.config.get("layer_penalty_factor", 5.0))
position_temperature = float(self.config.get("position_temperature", 5.0))
class_temperature = float(self.config.get("class_temperature", 0.0))
denoise = bool(self.config.get("denoise", True))
preprocess_prompt = bool(self.config.get("preprocess_prompt", True))
postprocess_output = bool(self.config.get("postprocess_output", True))
audio_chunk_duration = float(self.config.get("audio_chunk_duration", 15.0))
audio_chunk_threshold = float(self.config.get("audio_chunk_threshold", 30.0))
cache_key = None
if enable_audio_cache:
cache_key = self.audio_cache.generate_cache_key(
"omnivoice",
text=stripped,
audio_component=audio_component,
prompt_text=prompt_text,
model_variant=model_variant,
language=language or "none",
instruct=instruct or "",
num_step=num_step,
guidance_scale=guidance_scale,
t_shift=t_shift,
speed=speed,
duration=duration if duration is not None else "auto",
layer_penalty_factor=layer_penalty_factor,
position_temperature=position_temperature,
class_temperature=class_temperature,
denoise=denoise,
preprocess_prompt=preprocess_prompt,
postprocess_output=postprocess_output,
audio_chunk_duration=audio_chunk_duration,
audio_chunk_threshold=audio_chunk_threshold,
seed=int(seed or 0),
device=device,
dtype=dtype,
character=character_name or "narrator",
)
cached_audio = self.audio_cache.get_cached_audio(cache_key)
if cached_audio:
print(f"💾 Using cached OmniVoice audio for '{character_name or 'narrator'}': '{stripped[:30]}...'")
return cached_audio[0]
voice_clone_prompt = None
if ref_audio is not None:
if not prompt_text:
raise ValueError(
"OmniVoice voice cloning requires reference text. "
"Direct audio-only narrator input is unsupported. "
"Use Character Voices or a narrator voice file with matching reference text."
)
voice_clone_prompt = engine.create_voice_clone_prompt(
ref_audio=ref_audio,
ref_text=prompt_text,
preprocess_prompt=preprocess_prompt,
)
self._seed_everything(seed)
result = engine.generate(
text=stripped,
language=language,
voice_clone_prompt=voice_clone_prompt,
instruct=instruct,
duration=duration,
speed=speed,
num_step=num_step,
guidance_scale=guidance_scale,
t_shift=t_shift,
denoise=denoise,
preprocess_prompt=preprocess_prompt,
postprocess_output=postprocess_output,
layer_penalty_factor=layer_penalty_factor,
position_temperature=position_temperature,
class_temperature=class_temperature,
audio_chunk_duration=audio_chunk_duration,
audio_chunk_threshold=audio_chunk_threshold,
)
audio_tensor = result[0]
if not isinstance(audio_tensor, torch.Tensor):
audio_tensor = torch.tensor(audio_tensor, dtype=torch.float32)
audio_tensor = audio_tensor.detach().float().cpu()
if audio_tensor.dim() == 1:
audio_tensor = audio_tensor.unsqueeze(0)
if enable_audio_cache and cache_key:
duration_seconds = self.audio_cache._calculate_duration(audio_tensor, "omnivoice")
self.audio_cache.cache_audio(cache_key, audio_tensor, duration_seconds)
return audio_tensor
+44 -39
View File
@@ -19,7 +19,8 @@ if project_root not in sys.path:
from engines.qwen3_tts.qwen3_tts import Qwen3TTSEngine
from utils.text.pause_processor import PauseTagProcessor
from utils.audio.cache import get_audio_cache
from utils.audio.cache import get_audio_cache
from utils.voice.reference import effective_voice_audio
import folder_paths
@@ -105,9 +106,14 @@ class Qwen3TTSEngineAdapter:
Returns:
Model type string: "CustomVoice", "VoiceDesign", or "Base"
"""
# Priority 1: Voice Designer node
if context.get("node_type") == "voice_designer":
return "VoiceDesign"
# Explicit engine selection is authoritative for refactored workflows.
explicit_model_type = context.get("model_type")
if explicit_model_type in {"Base", "CustomVoice", "VoiceDesign"}:
return explicit_model_type
# Legacy voice designer context.
if context.get("node_type") == "voice_designer":
return "VoiceDesign"
# Priority 2: Preset voice selected
voice_preset = context.get("voice_preset")
@@ -117,15 +123,17 @@ class Qwen3TTSEngineAdapter:
# Priority 3: Default to Base (zero-shot)
return "Base"
def load_base_model(self,
model_path: str,
device: str = "auto",
dtype: str = "auto",
model_size: str = "1.7B",
attn_implementation: str = "auto",
context: Optional[Dict[str, Any]] = None,
use_torch_compile: bool = False,
use_cuda_graphs: bool = False,
def load_base_model(self,
model_path: str,
device: str = "auto",
dtype: str = "auto",
model_size: str = "1.7B",
attn_implementation: str = "auto",
runtime_mode: str = "main_environment",
runtime_profile: Optional[str] = None,
context: Optional[Dict[str, Any]] = None,
use_torch_compile: bool = False,
use_cuda_graphs: bool = False,
compile_mode: str = "reduce-overhead"):
"""
Load Qwen3-TTS engine via unified interface with intelligent model selection.
@@ -153,8 +161,8 @@ class Qwen3TTSEngineAdapter:
model_size = "1.7B"
print("⚠️ VoiceDesign requires 1.7B model, auto-switching from 0.6B")
# Build model name
model_name = f"Qwen3-TTS-12Hz-{model_size}-{model_type}"
# Keep the canonical model name separate from a local: model path.
model_name = context.get("model_name") or f"Qwen3-TTS-12Hz-{model_size}-{model_type}"
# Track current model type (unified interface handles unloading automatically)
self.current_model_type = model_type
@@ -165,12 +173,14 @@ class Qwen3TTSEngineAdapter:
config = ModelLoadConfig(
engine_name="qwen3_tts",
model_type="tts",
model_name=model_name,
model_path=model_path if model_path else model_name,
device=resolve_torch_device(device),
additional_params={
"dtype": dtype,
"attn_implementation": attn_implementation,
model_name=model_name,
model_path=model_path if model_path else model_name,
device=resolve_torch_device(device),
runtime_mode=runtime_mode,
runtime_profile=runtime_profile,
additional_params={
"dtype": dtype,
"attn_implementation": attn_implementation,
"use_torch_compile": use_torch_compile,
"use_cuda_graphs": use_cuda_graphs,
"compile_mode": compile_mode
@@ -524,20 +534,18 @@ class Qwen3TTSEngineAdapter:
# Generate cache key using voice_ref dict (contains waveform + sample_rate)
# This ensures different voices generate different cache keys
from utils.audio.audio_hash import generate_stable_audio_component
if voice_ref and isinstance(voice_ref, dict):
# Check if voice_ref has audio tensor or file path
if "audio" in voice_ref:
# Unified Character Voices format: {"audio": {"waveform": ..., "sample_rate": ...}, "audio_path": ..., ...}
audio_dict = voice_ref.get("audio")
audio_component = generate_stable_audio_component(reference_audio=audio_dict)
elif ref_audio_original is not None and isinstance(ref_audio_original, str):
# File path format: {"audio_path": "/path/to/file.wav", "reference_text": "..."}
audio_component = generate_stable_audio_component(audio_file_path=ref_audio_original)
elif "waveform" in voice_ref:
# Direct tensor format: {"waveform": tensor, "sample_rate": 24000}
audio_component = generate_stable_audio_component(reference_audio=voice_ref)
else:
audio_component = "default_voice"
if voice_ref and isinstance(voice_ref, dict):
if isinstance(ref_audio_original, str):
audio_component = generate_stable_audio_component(audio_file_path=ref_audio_original)
elif isinstance(ref_audio_original, dict) and "waveform" in ref_audio_original:
audio_component = generate_stable_audio_component(reference_audio=ref_audio_original)
elif torch.is_tensor(ref_audio_original):
audio_component = generate_stable_audio_component(reference_audio={
"waveform": ref_audio_original,
"sample_rate": voice_ref.get("sample_rate", 24000),
})
else:
audio_component = "default_voice"
elif ref_audio_original is not None and isinstance(ref_audio_original, str):
# File path case (voice_ref is None but ref_audio_original extracted)
audio_component = generate_stable_audio_component(audio_file_path=ref_audio_original)
@@ -683,10 +691,7 @@ class Qwen3TTSEngineAdapter:
return None, None, False
# Extract reference audio (multiple possible keys)
ref_audio_original = (voice_ref.get('prompt_audio_path') or
voice_ref.get('audio_path') or
voice_ref.get('audio') or
voice_ref.get('waveform'))
ref_audio_original = effective_voice_audio(voice_ref)
# Extract reference text (multiple possible keys)
ref_text = (voice_ref.get('prompt_text') or
+54 -33
View File
@@ -84,20 +84,23 @@ class RVCEngineAdapter:
print(f"Error loading RVC models: {e}")
raise e
def convert_voice(
self,
audio_input: Union[torch.Tensor, np.ndarray, tuple],
rvc_model: dict = None,
pitch_shift: int = 0,
def convert_voice(
self,
audio_input: Union[torch.Tensor, np.ndarray, tuple],
rvc_model: dict = None,
pitch_shift: int = 0,
index_rate: float = 0.75,
rms_mix_rate: float = 0.25,
protect: float = 0.25,
f0_method: str = 'rmvpe',
f0_autotune: bool = False,
resample_sr: int = 0,
crepe_hop_length: int = 160,
**kwargs
) -> Tuple[np.ndarray, int]:
protect: float = 0.25,
f0_method: str = 'rmvpe',
f0_autotune: bool = False,
resample_sr: int = 0,
crepe_hop_length: int = 160,
filter_radius: int = 3,
use_cache: bool = True,
batch_size: int = 1,
**kwargs
) -> Tuple[np.ndarray, int]:
"""
Perform voice conversion using RVC.
@@ -109,10 +112,13 @@ class RVCEngineAdapter:
rms_mix_rate: RMS mixing rate (0.0-1.0)
protect: Consonant protection (0.0-0.5)
f0_method: Pitch extraction method
f0_autotune: Enable autotune
resample_sr: Resample rate (0 for no resampling)
crepe_hop_length: Crepe hop length (16-512)
**kwargs: Additional parameters
f0_autotune: Enable autotune
resample_sr: Resample rate (0 for no resampling)
crepe_hop_length: Crepe hop length (16-512)
filter_radius: Harvest smoothing radius
use_cache: Reuse cached loaded models where possible
batch_size: Crepe-family pitch batch size hint
**kwargs: Additional parameters
Returns:
Tuple of (converted_audio, sample_rate)
@@ -156,13 +162,16 @@ class RVCEngineAdapter:
f0_up_key=pitch_shift,
f0_method=f0_method,
index_rate=index_rate,
protect=protect,
rms_mix_rate=rms_mix_rate,
resample_sr=resample_sr,
f0_autotune=f0_autotune,
crepe_hop_length=crepe_hop_length,
**kwargs # Pass through hubert_path and other engine parameters
)
protect=protect,
rms_mix_rate=rms_mix_rate,
resample_sr=resample_sr,
f0_autotune=f0_autotune,
crepe_hop_length=crepe_hop_length,
filter_radius=filter_radius,
use_cache=use_cache,
batch_size=batch_size,
**kwargs # Pass through hubert_path and other engine parameters
)
if result is None:
raise ValueError("RVC conversion failed - no output from minimal wrapper")
@@ -290,11 +299,14 @@ class RVCEngineAdapter:
'index_rate': 0.75,
'rms_mix_rate': 0.25,
'protect': 0.25,
'f0_method': 'rmvpe',
'f0_autotune': False,
'resample_sr': 0,
'crepe_hop_length': 160
}
'f0_method': 'rmvpe',
'f0_autotune': False,
'resample_sr': 0,
'crepe_hop_length': 160,
'filter_radius': 3,
'use_cache': True,
'batch_size': 1,
}
def validate_parameters(self, **params) -> Dict[str, Any]:
"""
@@ -331,10 +343,19 @@ class RVCEngineAdapter:
if params['resample_sr'] in valid_rates:
validated['resample_sr'] = int(params['resample_sr'])
if 'crepe_hop_length' in params:
validated['crepe_hop_length'] = max(16, min(512, int(params['crepe_hop_length'])))
return validated
if 'crepe_hop_length' in params:
validated['crepe_hop_length'] = max(16, min(512, int(params['crepe_hop_length'])))
if 'filter_radius' in params:
validated['filter_radius'] = max(0, min(7, int(params['filter_radius'])))
if 'use_cache' in params:
validated['use_cache'] = bool(params['use_cache'])
if 'batch_size' in params:
validated['batch_size'] = max(1, min(32, int(params['batch_size'])))
return validated
def cleanup(self):
"""Clean up loaded models and free memory."""
@@ -345,4 +366,4 @@ class RVCEngineAdapter:
# CRITICAL: Clear minimal wrapper cache - this is where RVC models are actually stored
minimal_wrapper.clear_model_cache()
print("RVC adapter cleanup completed")
print("RVC adapter cleanup completed")
+7 -3
View File
@@ -93,7 +93,9 @@ class StepAudioEditXEngineAdapter:
model_path: str,
device: str = "auto",
torch_dtype: str = "auto",
quantization: Optional[str] = None):
quantization: Optional[str] = None,
runtime_mode: str = "shared_runtime",
runtime_profile: Optional[str] = None):
"""
Load Step Audio EditX engine via unified interface (with caching).
@@ -113,6 +115,8 @@ class StepAudioEditXEngineAdapter:
model_name="Step-Audio-EditX",
model_path=model_path, # Downloader will resolve this if it's "local:xxx"
device=resolve_torch_device(device),
runtime_mode=runtime_mode,
runtime_profile=runtime_profile,
additional_params={
"torch_dtype": torch_dtype,
"quantization": quantization
@@ -193,7 +197,7 @@ class StepAudioEditXEngineAdapter:
prompt_text=prompt_text,
temperature=params.get('temperature', 0.7),
do_sample=params.get('do_sample', True),
max_new_tokens=params.get('max_new_tokens', 8192),
max_new_tokens=params.get('max_new_tokens', 1024),
seed=params.get('seed', 0),
model_path=params.get('model_path', 'Step-Audio-EditX'),
device=params.get('device', 'auto'),
@@ -210,7 +214,7 @@ class StepAudioEditXEngineAdapter:
return cached_audio[0]
# Create ComfyUI progress bar for generation tracking with time prediction
max_new_tokens = params.get('max_new_tokens', 8192)
max_new_tokens = params.get('max_new_tokens', 1024)
# Estimate actual tokens based on text length (same heuristic as Qwen3-TTS)
# Rough heuristic: ~0.7 tokens per character for TTS (conservative estimate)
+113 -39
View File
@@ -16,10 +16,16 @@ if parent_dir not in sys.path:
sys.path.insert(0, parent_dir)
from utils.models.language_mapper import get_model_for_language
from utils.models.engine_registry import get_default_runtime_profile
from utils.models.factory_config import normalize_runtime_mode, runtime_uses_isolation
from utils.text.pause_processor import PauseTagProcessor
from utils.text.character_parser import character_parser
from engines.vibevoice_engine.vibevoice_downloader import VIBEVOICE_MODELS
from engines.vibevoice_engine.vibevoice_downloader import (
VIBEVOICE_MODELS,
is_kugelaudio_variant_name,
)
from utils.models.manager import model_manager
from utils.voice.character_logging import resolved_character_label
class VibeVoiceEngineAdapter:
@@ -48,11 +54,15 @@ class VibeVoiceEngineAdapter:
self._device = engine_config.get('device', 'auto')
self._attention_mode = engine_config.get('attention_mode', 'auto')
self._quantize_4bit = engine_config.get('quantize_llm_4bit', False)
self._runtime_mode = normalize_runtime_mode(engine_config.get('runtime_mode'))
self._runtime_profile = engine_config.get('runtime_profile')
else:
self._model_name = 'vibevoice-1.5B'
self._device = 'auto'
self._attention_mode = 'auto'
self._quantize_4bit = False
self._runtime_mode = 'main_environment'
self._runtime_profile = None
@property
def vibevoice_engine(self):
@@ -65,7 +75,9 @@ class VibeVoiceEngineAdapter:
model_name=self._model_name,
device=self._device,
attention_mode=self._attention_mode,
quantize_llm_4bit=self._quantize_4bit
quantize_llm_4bit=self._quantize_4bit,
runtime_mode=self._runtime_mode,
runtime_profile=self._runtime_profile or get_default_runtime_profile("vibevoice"),
)
@property
@@ -99,8 +111,8 @@ class VibeVoiceEngineAdapter:
# VibeVoice models support both English and Chinese
supported_languages = ['en', 'zh', 'zh-cn', 'chinese', 'english']
# KugelAudio supports 23 languages, skip language validation
if "kugelaudio" in default_model.lower():
# Kugel variants are multilingual, so skip base VibeVoice language gating.
if is_kugelaudio_variant_name(default_model):
return default_model
if lang_code.lower() in supported_languages:
@@ -455,7 +467,8 @@ class VibeVoiceEngineAdapter:
# Generate each character group using VibeVoice format
for group_idx, (character, text_list) in enumerate(character_groups):
print(f"🎤 Group {group_idx + 1}: Character '{character}' with {len(text_list)} segments")
display_name = resolved_character_label(character, voice_mapping.get(character))
print(f"🎤 Group {group_idx + 1}: Character '{display_name}' with {len(text_list)} segments")
# Format as Speaker 1 entries (VibeVoice style) and combine
formatted_lines = []
@@ -515,15 +528,11 @@ class VibeVoiceEngineAdapter:
Returns:
Combined audio dict
"""
# Get speaker voice inputs from engine config for priority system
# For manual Speaker format, the main narrator voice comes from the TTS Text node
# We need to get it from a different source since voice_mapping may not have 'narrator' key
main_narrator_voice = voice_mapping.get('narrator') # Try narrator first
if main_narrator_voice is None:
# If no 'narrator' key, get it from any available voice (fallback for manual Speaker format)
available_voices = [v for v in voice_mapping.values() if v is not None]
main_narrator_voice = available_voices[0] if available_voices else None
# print(f"🐛 Debug: No 'narrator' key, using fallback voice: {'✅ found' if main_narrator_voice else '❌ none available'}")
# Get speaker voice inputs from engine config for priority system.
# Speaker 1 must only come from an explicit narrator/speaker-1 input.
# Do not synthesize Speaker 1 from discovered character voices, or aliases
# will look like they were overridden by a connection that does not exist.
main_narrator_voice = voice_mapping.get('narrator')
speaker_inputs = {
1: main_narrator_voice, # Speaker 1 uses main narrator from TTS Text
2: params.get('speaker2_voice'),
@@ -533,19 +542,24 @@ class VibeVoiceEngineAdapter:
# print(f"🐛 Debug: speaker_inputs[1] (main narrator): {'✅ has voice' if speaker_inputs[1] else '❌ no voice'}")
# Build speaker mapping and format text
# Build speaker mapping and format text.
# For SRT/global processing we keep speaker numbering local to the current
# generation call (Speaker 1..N in order of first appearance), but preserve
# the character's global slot when choosing connected speaker override inputs.
character_map = {}
# Pre-fill speaker_voices with all 4 speaker slots (some may be None)
speaker_voices = [
speaker_inputs.get(1), # Speaker 1
speaker_inputs.get(2), # Speaker 2
speaker_inputs.get(3), # Speaker 3
speaker_inputs.get(4) # Speaker 4
]
character_global_slots = {}
speaker_voices = []
formatted_lines = []
segment_characters = [char for char, _ in segments]
print(f"🎭 Native multi-speaker: Processing {len(segments)} segments with characters: {[char for char, _ in segments]}")
print(f"🎭 Native multi-speaker: Processing {len(segments)} segments with characters: {segment_characters}")
print(f"🎤 Speaker inputs connected: {[f'Speaker {k}' for k, v in speaker_inputs.items() if v is not None]}")
narrator_exists_globally = bool(global_char_to_speaker and "narrator" in global_char_to_speaker)
if speaker_inputs.get(1) is not None and "narrator" not in segment_characters:
if narrator_exists_globally:
print("ℹ️ Narrator input is connected, but this subtitle has no narrator turn; it will not override tagged characters")
else:
print("ℹ️ No narrator turns exist in this SRT; Speaker 1 maps to the first named character by first appearance")
for character, text in segments:
# Check if this is already a manual "Speaker N:" format
@@ -556,50 +570,84 @@ class VibeVoiceEngineAdapter:
speaker_idx = manual_speaker - 1 # Convert to 0-based
if speaker_idx >= 4:
speaker_idx = 3
# Voice already in pre-filled speaker_voices array
voice = speaker_voices[speaker_idx]
voice = speaker_inputs.get(manual_speaker)
if manual_speaker == 1:
print(f"🎤 Manual format 'Speaker {manual_speaker}' -> using {'✅ main narrator (Tony)' if voice else '❌ no narrator, using default'}")
else:
print(f"🎤 Manual format 'Speaker {manual_speaker}' -> using {'✅ connected input' if voice else '❌ no input, using default'}")
while len(speaker_voices) <= speaker_idx:
speaker_voices.append(None)
speaker_voices[speaker_idx] = voice
formatted_lines.append(f"Speaker {manual_speaker}: {text.strip()}")
else:
# Character tag format - use global mapping if provided (for SRT consistency)
# Character tag format
if character not in character_map:
# Special handling for numeric characters: [1] [2] [3] [4] -> map directly to Speaker N
if character.isdigit() and 1 <= int(character) <= 4:
speaker_idx = int(character) - 1 # Convert [1] to Speaker 1 (0-based index)
global_speaker_num = int(character)
speaker_idx = len(character_map)
character_map[character] = speaker_idx
character_global_slots[character] = global_speaker_num
print(f"🔢 Numeric character '[{character}]' -> Speaker {int(character)} (direct mapping)")
elif global_char_to_speaker and character in global_char_to_speaker:
# Use global mapping for consistent SRT processing
speaker_idx = global_char_to_speaker[character] - 1 # Convert to 0-based
global_speaker_num = global_char_to_speaker[character]
speaker_idx = len(character_map)
character_map[character] = speaker_idx
character_global_slots[character] = global_speaker_num
else:
# Fallback to sequential assignment
speaker_idx = len(character_map)
if speaker_idx >= 4:
print(f"⚠️ VibeVoice: Limiting to 4 speakers, '{character}' will use Speaker 4")
speaker_idx = 3 # Use 0-based internally, will convert to 1-based for format
global_speaker_num = 4
else:
character_map[character] = speaker_idx
global_speaker_num = speaker_idx + 1
character_global_slots[character] = global_speaker_num
# Priority system: speaker inputs override character aliases
# Priority system:
# - Numeric [1]-[4] always map directly to speaker inputs 1-4.
# - If any narrator turn exists in the SRT, Speaker 1 is reserved for narrator.
# - Otherwise, Speaker 1..4 map to named characters by first global appearance order.
speaker_num = speaker_idx + 1
connected_voice = speaker_inputs.get(speaker_num)
global_speaker_num = character_global_slots.get(character, speaker_num)
connected_voice = None
is_numeric_direct = character.isdigit() and 1 <= int(character) <= 4
if is_numeric_direct:
connected_voice = speaker_inputs.get(global_speaker_num)
elif character == "narrator":
connected_voice = speaker_inputs.get(1)
elif narrator_exists_globally:
if global_speaker_num >= 2:
connected_voice = speaker_inputs.get(global_speaker_num)
else:
connected_voice = speaker_inputs.get(global_speaker_num)
character_voice = voice_mapping.get(character)
if connected_voice is not None and character_voice is not None:
print(f"⚠️ Priority: Speaker {speaker_num} input overrides ['{character}'] alias - using connected voice")
if global_speaker_num != speaker_num:
print(
f"⚠️ Priority: Speaker {global_speaker_num} input overrides ['{character}'] alias "
f"- using connected voice as local Speaker {speaker_num}"
)
else:
print(f"⚠️ Priority: Speaker {speaker_num} input overrides ['{character}'] alias - using connected voice")
voice = connected_voice
elif connected_voice is not None:
print(f"🎤 Speaker {speaker_num}: Using connected voice input")
if global_speaker_num != speaker_num:
print(f"🎤 Speaker {global_speaker_num}: Using connected voice input as local Speaker {speaker_num}")
else:
print(f"🎤 Speaker {speaker_num}: Using connected voice input")
voice = connected_voice
else:
print(f"🎭 Character '{character}' -> Speaker {speaker_num}, using character voice")
if global_speaker_num != speaker_num:
print(f"🎭 Character '{character}' -> global Speaker {global_speaker_num}, local Speaker {speaker_num}, using alias/character voice")
else:
print(f"🎭 Character '{character}' -> Speaker {speaker_num}, using alias/character voice")
voice = character_voice
# Ensure we have enough speaker_voices slots
@@ -618,6 +666,18 @@ class VibeVoiceEngineAdapter:
print(formatted_text)
print("="*60)
print(f"🎤 Using {len(speaker_voices)} voice samples for generation")
for idx, voice in enumerate(speaker_voices, start=1):
if voice is None:
print(f" Speaker {idx}: default / no reference")
elif isinstance(voice, dict) and voice.get("audio_path"):
print(f" Speaker {idx}: file reference -> {voice['audio_path']}")
elif isinstance(voice, dict) and "waveform" in voice:
waveform = voice["waveform"]
sample_rate = voice.get("sample_rate", "unknown")
shape = tuple(waveform.shape) if hasattr(waveform, "shape") else "unknown"
print(f" Speaker {idx}: waveform reference -> shape {shape}, sr {sample_rate}")
else:
print(f" Speaker {idx}: unexpected reference type {type(voice).__name__}")
# Validate and normalize voice references
normalized_voices = []
@@ -771,14 +831,20 @@ class VibeVoiceEngineAdapter:
# Process pause tags - returns list of ('text', content) or ('pause', duration)
pause_segments, _ = PauseTagProcessor.parse_pause_tags(text)
print(f"🎵 VibeVoice: Parsed {len(pause_segments)} pause-aware segments")
# Generate segment dicts for each pause-separated piece
segment_dicts = []
for seg_type, seg_content in pause_segments:
for idx, (seg_type, seg_content) in enumerate(pause_segments, start=1):
if seg_type == 'text':
# Extract edit tags from this text segment
clean_text, edit_tags = get_edit_tags_for_segment(seg_content)
preview = clean_text.replace('\n', ' ')[:120]
if len(clean_text) > 120:
preview += "..."
print(f" 🎤 Segment {idx}/{len(pause_segments)}: generating text ({len(clean_text)} chars)")
print(f" {preview}")
# Generate TTS for this text piece
result = self.generate_segment(clean_text, voice_ref, params, character)
@@ -802,6 +868,7 @@ class VibeVoiceEngineAdapter:
else: # seg_type == 'pause'
# Create silence segment
duration_seconds = seg_content
print(f" ⏸️ Segment {idx}/{len(pause_segments)}: inserting {duration_seconds:.2f}s silence")
silence = PauseTagProcessor.create_silence_segment(
duration_seconds, 24000, device=torch.device('cpu')
)
@@ -837,7 +904,14 @@ class VibeVoiceEngineAdapter:
return None
def cleanup(self):
"""Clean up resources - models are now managed by ModelManager"""
"""Clean up resources and unload isolated runtime workers when requested."""
self._character_speaker_map.clear()
self._speaker_voices.clear()
# Don't clean up vibevoice_engine - it just holds references to cached models
if runtime_uses_isolation(self._runtime_mode):
try:
from utils.models.unified_model_interface import unified_model_interface
unified_model_interface.clear_engine_models("vibevoice")
except Exception as e:
print(f"⚠️ Failed to unload isolated VibeVoice runtime during cleanup: {e}")
# Don't clean up vibevoice_engine - it just holds references to cached models
+26 -10
View File
@@ -18,12 +18,28 @@ import shutil
import soundfile as sf
class AudioTimingError(Exception):
"""Exception raised when audio timing operations fail"""
pass
class AudioTimingUtils:
class AudioTimingError(Exception):
"""Exception raised when audio timing operations fail"""
pass
def _stack_stretched_channels(
channels: List[torch.Tensor],
device: torch.device,
) -> torch.Tensor:
"""Stack independently stretched channels after reconciling tiny length drift."""
if not channels:
raise AudioTimingError("No audio channels were processed successfully")
common_length = min(channel.size(-1) for channel in channels)
if common_length <= 0:
raise AudioTimingError("Time stretching produced an empty audio channel")
return torch.stack(
[channel[..., :common_length] for channel in channels],
dim=0,
).to(device)
class AudioTimingUtils:
"""
Utilities for audio timing manipulation and synchronization
"""
@@ -211,7 +227,7 @@ class PhaseVocoderTimeStretcher:
stretched_channels.append(torch.from_numpy(stretched))
# Combine channels
result = torch.stack(stretched_channels, dim=0).to(audio.device)
result = _stack_stretched_channels(stretched_channels, audio.device)
# Restore original shape if input was 1D
if len(original_shape) == 1:
@@ -366,7 +382,7 @@ class FFmpegTimeStretcher:
if not stretched:
raise AudioTimingError("No audio was processed successfully")
result = torch.stack(stretched, dim=0).to(audio.device)
result = _stack_stretched_channels(stretched, audio.device)
return result.squeeze(0) if len(original_shape) == 1 else result
except Exception as e:
@@ -425,7 +441,7 @@ class FFmpegTimeStretcher:
try:
# Stack channels and restore shape
result = torch.stack(stretched, dim=0).to(audio.device)
result = _stack_stretched_channels(stretched, audio.device)
print(f"Successfully processed all channels")
return result.squeeze(0) if len(original_shape) == 1 else result
@@ -707,4 +723,4 @@ def calculate_timing_adjustments(natural_durations: List[float],
adjustments.append(adjustment)
return adjustments
return adjustments
+11 -9
View File
@@ -7,7 +7,8 @@ Works with character_grouper.py to process multiple segments simultaneously.
from typing import Dict, List, Any
import torch
from .character_grouper import CharacterGroup
from .character_grouper import CharacterGroup
from utils.voice.character_logging import resolved_character_label
class BatchProcessor:
@@ -49,7 +50,8 @@ class BatchProcessor:
character = character_group.character
segments = character_group.segments
print(f"🚀 BATCH PROCESSING: {character} - {len(segments)} segments in {language}")
display_name = resolved_character_label(character, voice_refs.get(character))
print(f"🚀 BATCH PROCESSING: {display_name} - {len(segments)} segments in {language}")
# Collect all texts for batching
batch_texts = []
@@ -75,7 +77,7 @@ class BatchProcessor:
char_audio_prompt = voice_refs[character]
# THE ACTUAL BATCH PROCESSING - this is the key improvement
print(f"⚡ Batch generating {len(batch_texts)} chunks for {character}")
print(f"⚡ Batch generating {len(batch_texts)} chunks for {display_name}")
print(f"🔧 DEBUG: batch_size from inputs = {inputs.get('batch_size', 4)}")
try:
batch_audio = self.tts_model.generate_batch(
@@ -137,15 +139,15 @@ class BatchProcessor:
character = character_group.character
segments = character_group.segments
print(f"→ SEQUENTIAL: {character} - {len(segments)} segments in {language}")
char_audio_prompt = voice_refs[character]
display_name = resolved_character_label(character, char_audio_prompt)
print(f"→ SEQUENTIAL: {display_name} - {len(segments)} segments in {language}")
results = {}
char_audio_prompt = voice_refs[character]
for segment in segments:
for segment in segments:
segment_display_idx = segment.original_idx + 1 # 1-based for display
print(f"🎤 Generating segment {segment_display_idx}/{total_segments} for '{character}' (lang: {language})")
print(f"🎤 Generating segment {segment_display_idx}/{total_segments} for '{display_name}' (lang: {language})")
# Apply chunking
if inputs["enable_chunking"] and len(segment.segment_text) > inputs["max_chars_per_chunk"]:
@@ -183,4 +185,4 @@ class BatchProcessor:
"""Apply crash protection padding to short texts."""
if len(text.strip()) < min_length:
return padding_template.format(seg=text)
return text
return text
+12 -3
View File
@@ -74,7 +74,10 @@ class MaskedDiffWithXvec(torch.nn.Module):
# concat text and prompt_text
mask = (~make_pad_mask(token_len)).float().unsqueeze(-1).to(device)
token = self.input_embedding(torch.clamp(token, min=0)) * mask
# TTS Audio Suite patch: bound speech-token indices to prevent CUDA embedding asserts.
token = self.input_embedding(
torch.clamp(token, min=0, max=self.input_embedding.num_embeddings - 1)
) * mask
# text encode
h, h_lengths = self.encoder(token, token_len)
@@ -124,7 +127,10 @@ class MaskedDiffWithXvec(torch.nn.Module):
token_len1, token_len2 = prompt_token.shape[1], token.shape[1]
token, token_len = torch.concat([prompt_token, token], dim=1), prompt_token_len + token_len
mask = (~make_pad_mask(token_len)).unsqueeze(-1).to(embedding)
token = self.input_embedding(torch.clamp(token, min=0)) * mask
# TTS Audio Suite patch: bound speech-token indices to prevent CUDA embedding asserts.
token = self.input_embedding(
torch.clamp(token, min=0, max=self.input_embedding.num_embeddings - 1)
) * mask
# text encode
h, h_lengths = self.encoder(token, token_len)
@@ -215,7 +221,10 @@ class CausalMaskedDiffWithXvec(torch.nn.Module):
# concat text and prompt_text
token, token_len = torch.concat([prompt_token, token], dim=1), prompt_token_len + token_len
mask = (~make_pad_mask(token_len)).unsqueeze(-1).to(embedding)
token = self.input_embedding(torch.clamp(token, min=0)) * mask
# TTS Audio Suite patch: bound speech-token indices to prevent CUDA embedding asserts.
token = self.input_embedding(
torch.clamp(token, min=0, max=self.input_embedding.num_embeddings - 1)
) * mask
# text encode
h, h_lengths = self.encoder(token, token_len)
@@ -1,3 +1,5 @@
import logging
from .s3tokenizer import (
S3_SR,
S3_HOP,
@@ -27,4 +29,20 @@ def drop_invalid_tokens(x):
e = None
x = x[s: e]
# T3's speech head spans `speech_tokens_dict_size` (8194) while S3Gen's
# input embedding only has SPEECH_VOCAB_SIZE (6561) rows, so a sampled id
# above the vocab that is not exactly SoS/EoS reaches the embedding gather
# and trips a device-side "index out of bounds" assert (which surfaces
# later, at the next CUDA sync, with a misleading traceback).
invalid = x >= SPEECH_VOCAB_SIZE
if invalid.any():
logging.warning(
"Dropping %d out-of-vocabulary speech token(s) (max id %d, vocab %d)",
int(invalid.sum()), int(x.max()), SPEECH_VOCAB_SIZE,
)
x = x[~invalid]
# TTS Audio Suite patch: do not pass an empty speech stream to S3Gen.
if x.numel() == 0:
raise RuntimeError("ChatterBox generated no valid speech tokens. Try another seed or text.")
return x
+3 -8
View File
@@ -16,7 +16,6 @@ from .modules.cond_enc import T3CondEnc, T3Cond
from .modules.t3_config import T3Config
from .llama_configs import LLAMA_CONFIGS
from .inference.t3_hf_backend import T3HuggingfaceBackend
from .inference.alignment_stream_analyzer import AlignmentStreamAnalyzer
logger = logging.getLogger(__name__)
@@ -277,13 +276,9 @@ class T3(nn.Module):
self.tfmr.use_cache = True
self.tfmr.return_dict = True
alignment_stream_analyzer = AlignmentStreamAnalyzer(
self.tfmr,
None,
text_tokens_slice=(len_cond, len_cond + text_tokens.size(-1)),
alignment_layer_idx=9,
eos_idx=self.hp.stop_speech_token,
)
# TTS Audio Suite patch: alignment analysis is disabled in the backend, so do
# not install its persistent attention hook/forward wrapper on every inference.
alignment_stream_analyzer = None
# Create backend with updated settings
patched_model = T3HuggingfaceBackend(
+23
View File
@@ -0,0 +1,23 @@
import warnings
def create_perth_watermarker():
"""
Lazily import perth only when watermarking is actually requested.
Watermarking is disabled by default, so Chatterbox should not fail model load
just because perth drags in a fragile librosa import path on some envs.
"""
with warnings.catch_warnings():
warnings.simplefilter("ignore")
import perth
watermarker_cls = getattr(perth, "PerthImplicitWatermarker", None)
if watermarker_cls is None:
raise AttributeError("PerthImplicitWatermarker not available in perth module")
watermarker = watermarker_cls()
if watermarker is None:
raise ValueError("PerthImplicitWatermarker returned None")
return watermarker
+26 -39
View File
@@ -4,13 +4,14 @@ import os
import torch
import torch.nn.functional as F
from huggingface_hub import hf_hub_download
import warnings
# Use librosa fallback for Python 3.13 compatibility
from utils.audio.librosa_fallback import safe_load, safe_resample
# Import safetensors for multilanguage model support
from safetensors.torch import load_file
from huggingface_hub import hf_hub_download
import warnings
# Use librosa fallback for Python 3.13 compatibility
from utils.audio.librosa_fallback import safe_load, safe_resample
from .perth_loader import create_perth_watermarker
# Import safetensors for multilanguage model support
from safetensors.torch import load_file
# Import folder_paths for model directory detection
try:
@@ -18,19 +19,9 @@ try:
except ImportError:
folder_paths = None
# Import perth with warnings disabled and graceful fallback
try:
with warnings.catch_warnings():
warnings.simplefilter("ignore")
import perth
PERTH_AVAILABLE = True
except ImportError:
perth = None
PERTH_AVAILABLE = False
from .models.t3 import T3
from .models.s3tokenizer import S3_SR, drop_invalid_tokens
from .models.s3gen import S3GEN_SR, S3Gen
from .models.t3 import T3
from .models.s3tokenizer import S3_SR, drop_invalid_tokens
from .models.s3gen import S3GEN_SR, S3Gen
from .models.tokenizers import EnTokenizer
from .models.voice_encoder import VoiceEncoder
from .models.t3.modules.cond_enc import T3Cond
@@ -197,20 +188,16 @@ class ChatterboxTTS:
return self
def _init_watermarker_if_needed(self):
"""Initialize watermarker on first use if enabled"""
if self.enable_watermarking and not self._watermarker_init_attempted:
self._watermarker_init_attempted = True
try:
with warnings.catch_warnings():
warnings.simplefilter("ignore")
self.watermarker = perth.PerthImplicitWatermarker()
if self.watermarker is None:
raise ValueError("PerthImplicitWatermarker returned None")
except Exception as e:
print(f"❌ Failed to initialize watermarker: {e}")
self.watermarker = None
self.enable_watermarking = False
def _init_watermarker_if_needed(self):
"""Initialize watermarker on first use if enabled"""
if self.enable_watermarking and not self._watermarker_init_attempted:
self._watermarker_init_attempted = True
try:
self.watermarker = create_perth_watermarker()
except Exception as e:
print(f"❌ Failed to initialize watermarker: {e}")
self.watermarker = None
self.enable_watermarking = False
@classmethod
def from_local(cls, ckpt_dir, device, language=None) -> 'ChatterboxTTS':
@@ -281,7 +268,7 @@ class ChatterboxTTS:
if custom_path.suffix == ".safetensors":
return load_file(custom_path, device=device)
if custom_path.suffix == ".pt":
return torch.load(custom_path, map_location=device)
return torch.load(custom_path, map_location=device, weights_only=True)
safetensors_path = ckpt_dir / f"{base_name}.safetensors"
pt_path = ckpt_dir / f"{base_name}.pt"
@@ -289,7 +276,7 @@ class ChatterboxTTS:
if safetensors_path.exists():
return load_file(safetensors_path, device=device)
elif pt_path.exists():
return torch.load(pt_path, map_location=device)
return torch.load(pt_path, map_location=device, weights_only=True)
elif not required:
return None
else:
@@ -312,7 +299,7 @@ class ChatterboxTTS:
return load_file(safetensors_path, device=device)
elif pt_path.exists():
print(f"📁 Loading {base_name} from local English model: {pt_path}")
return torch.load(pt_path, map_location=device)
return torch.load(pt_path, map_location=device, weights_only=True)
# Download English model if not available locally
print(f"📦 Downloading English model components for incomplete language model...")
@@ -328,7 +315,7 @@ class ChatterboxTTS:
return load_file(safetensors_path, device=device)
elif pt_path.exists():
print(f"📁 Loading {base_name} from downloaded English model: {pt_path}")
return torch.load(pt_path, map_location=device)
return torch.load(pt_path, map_location=device, weights_only=True)
raise FileNotFoundError(f"Could not load {base_name} from English fallback")
@@ -465,7 +452,7 @@ class ChatterboxTTS:
warnings.simplefilter("ignore")
print(f"📦 Loading checkpoint: {unified_model_path.name}")
checkpoint = torch.load(unified_model_path, map_location=device)
checkpoint = torch.load(unified_model_path, map_location=device, weights_only=True)
# Extract model configuration
model_config = checkpoint.get('model_config', {})
+20 -36
View File
@@ -2,11 +2,12 @@ from pathlib import Path
import os
import torch
import warnings
from huggingface_hub import hf_hub_download
# Use librosa fallback for Python 3.13 compatibility
from utils.audio.librosa_fallback import safe_load
import warnings
from huggingface_hub import hf_hub_download
# Use librosa fallback for Python 3.13 compatibility
from utils.audio.librosa_fallback import safe_load
from .perth_loader import create_perth_watermarker
# Import folder_paths for model directory detection
try:
@@ -14,18 +15,8 @@ try:
except ImportError:
folder_paths = None
# Import perth with warnings disabled and graceful fallback
try:
with warnings.catch_warnings():
warnings.simplefilter("ignore")
import perth
PERTH_AVAILABLE = True
except ImportError:
perth = None
PERTH_AVAILABLE = False
# Import safetensors for multilanguage model support
from safetensors.torch import load_file
# Import safetensors for multilanguage model support
from safetensors.torch import load_file
from .models.s3tokenizer import S3_SR
from .models.s3gen import S3GEN_SR, S3Gen
@@ -82,23 +73,16 @@ class ChatterboxVC:
return self
def _init_watermarker_if_needed(self):
"""Initialize watermarker on first use if enabled"""
if self.enable_watermarking and not self._watermarker_init_attempted:
self._watermarker_init_attempted = True
try:
if PERTH_AVAILABLE and hasattr(perth, 'PerthImplicitWatermarker'):
with warnings.catch_warnings():
warnings.simplefilter("ignore")
self.watermarker = perth.PerthImplicitWatermarker()
if self.watermarker is None:
raise ValueError("PerthImplicitWatermarker returned None")
else:
raise AttributeError("PerthImplicitWatermarker not available in perth module")
except Exception as e:
print(f"❌ Failed to initialize watermarker: {e}")
self.watermarker = None
self.enable_watermarking = False
def _init_watermarker_if_needed(self):
"""Initialize watermarker on first use if enabled"""
if self.enable_watermarking and not self._watermarker_init_attempted:
self._watermarker_init_attempted = True
try:
self.watermarker = create_perth_watermarker()
except Exception as e:
print(f"❌ Failed to initialize watermarker: {e}")
self.watermarker = None
self.enable_watermarking = False
@classmethod
def from_local(cls, ckpt_dir, device) -> 'ChatterboxVC':
@@ -114,7 +98,7 @@ class ChatterboxVC:
if safetensors_path.exists():
return load_file(safetensors_path, device=device)
elif pt_path.exists():
return torch.load(pt_path, map_location=device)
return torch.load(pt_path, map_location=device, weights_only=True)
else:
raise FileNotFoundError(f"Neither {base_name}.safetensors nor {base_name}.pt found in {ckpt_dir}")
@@ -123,7 +107,7 @@ class ChatterboxVC:
ref_dict = None
if (builtin_voice := ckpt_dir / "conds.pt").exists():
states = torch.load(builtin_voice, map_location=device)
states = torch.load(builtin_voice, map_location=device, weights_only=True)
ref_dict = states['gen']
s3gen = S3Gen()
@@ -7,7 +7,8 @@ Works with character_grouper.py to process multiple segments simultaneously.
from typing import Dict, List, Any
import torch
from .character_grouper import CharacterGroup
from .character_grouper import CharacterGroup
from utils.voice.character_logging import resolved_character_label
class BatchProcessor:
@@ -49,7 +50,8 @@ class BatchProcessor:
character = character_group.character
segments = character_group.segments
print(f"🚀 BATCH PROCESSING: {character} - {len(segments)} segments in {language}")
display_name = resolved_character_label(character, voice_refs.get(character))
print(f"🚀 BATCH PROCESSING: {display_name} - {len(segments)} segments in {language}")
# Collect all texts for batching
batch_texts = []
@@ -76,7 +78,7 @@ class BatchProcessor:
char_audio_prompt = voice_refs[character]
# THE ACTUAL BATCH PROCESSING - this is the key improvement
print(f"⚡ Batch generating {len(batch_texts)} chunks for {character}")
print(f"⚡ Batch generating {len(batch_texts)} chunks for {display_name}")
print(f"🔧 DEBUG: batch_size from inputs = {inputs.get('batch_size', 4)}")
try:
batch_audio = self.tts_model.generate_batch(
@@ -138,15 +140,15 @@ class BatchProcessor:
character = character_group.character
segments = character_group.segments
print(f"→ SEQUENTIAL: {character} - {len(segments)} segments in {language}")
char_audio_prompt = voice_refs[character]
display_name = resolved_character_label(character, char_audio_prompt)
print(f"→ SEQUENTIAL: {display_name} - {len(segments)} segments in {language}")
results = {}
char_audio_prompt = voice_refs[character]
for segment in segments:
for segment in segments:
segment_display_idx = segment.original_idx + 1 # 1-based for display
print(f"🎤 Generating segment {segment_display_idx}/{total_segments} for '{character}' (lang: {language})")
print(f"🎤 Generating segment {segment_display_idx}/{total_segments} for '{display_name}' (lang: {language})")
# Apply chunking
if inputs["enable_chunking"] and len(segment.segment_text) > inputs["max_chars_per_chunk"]:
@@ -184,4 +186,4 @@ class BatchProcessor:
"""Apply crash protection padding to short texts."""
if len(text.strip()) < min_length:
return padding_template.format(seg=text)
return text
return text
@@ -43,19 +43,30 @@ OFFICIAL_23LANG_MODELS = {
"required_files": {
"v1": [
"t3_23lang.safetensors", # Multilingual T3 model v1
"s3gen.pt", # S3Gen model (same as English)
"ve.pt", # Voice encoder (same as English)
"mtl_tokenizer.json", # Multilingual tokenizer
"conds.pt" # Conditioning (optional)
"s3gen.pt", # S3Gen model (same as English)
"ve.pt", # Voice encoder (same as English)
"mtl_tokenizer.json", # Multilingual tokenizer
"Cangjie5_TC.json", # Chinese Cangjie mapping
"conds.pt" # Conditioning (optional)
],
"v2": [
"t3_mtl23ls_v2.safetensors", # Multilingual T3 model v2 with enhanced tokenization
"s3gen.pt", # S3Gen model (same as English)
"ve.pt", # Voice encoder (same as English)
"grapheme_mtl_merged_expanded_v1.json", # Enhanced grapheme/phoneme mappings with special tokens
"mtl_tokenizer.json", # Multilingual tokenizer (may be updated for v2)
"conds.pt" # Conditioning (optional)
]
"v2": [
"t3_mtl23ls_v2.safetensors", # Multilingual T3 model v2 with enhanced tokenization
"s3gen.pt", # S3Gen model (same as English)
"ve.pt", # Voice encoder (same as English)
"grapheme_mtl_merged_expanded_v1.json", # Enhanced grapheme/phoneme mappings with special tokens
"mtl_tokenizer.json", # Multilingual tokenizer (may be updated for v2)
"Cangjie5_TC.json", # Chinese Cangjie mapping
"conds.pt" # Conditioning (optional)
],
"v3": [
"t3_mtl23ls_v3.safetensors", # Latest official multilingual T3 model
"s3gen.pt", # Official V3 API continues to use shared S3Gen
"ve.pt", # Shared voice encoder
"grapheme_mtl_merged_expanded_v1.json",
"mtl_tokenizer.json",
"Cangjie5_TC.json",
"conds.pt"
]
},
"multilingual": True
},
@@ -1,3 +1,5 @@
import logging
from .s3tokenizer import (
S3_SR,
S3_HOP,
@@ -27,4 +29,20 @@ def drop_invalid_tokens(x):
e = None
x = x[s: e]
# T3's speech head spans `speech_tokens_dict_size` (8194) while S3Gen's
# input embedding only has SPEECH_VOCAB_SIZE (6561) rows, so a sampled id
# above the vocab that is not exactly SoS/EoS reaches the embedding gather
# and trips a device-side "index out of bounds" assert (which surfaces
# later, at the next CUDA sync, with a misleading traceback).
invalid = x >= SPEECH_VOCAB_SIZE
if invalid.any():
logging.warning(
"Dropping %d out-of-vocabulary speech token(s) (max id %d, vocab %d)",
int(invalid.sum()), int(x.max()), SPEECH_VOCAB_SIZE,
)
x = x[~invalid]
# TTS Audio Suite patch: do not pass an empty speech stream to S3Gen.
if x.numel() == 0:
raise RuntimeError("ChatterBox generated no valid speech tokens. Try another seed or text.")
return x
@@ -60,6 +60,10 @@ class AlignmentStreamAnalyzer:
# using it for all layers slows things down too much. We can apply it to just one layer
# by intercepting the kwargs and adding a forward hook (credit: jrm)
self.last_aligned_attns = []
# TTS Audio Suite patch: retain hook handles so each inference can remove its
# analyzer hooks instead of accumulating three more on the shared model.
self._hook_handles = []
self._original_output_attentions = getattr(getattr(tfmr, "config", None), "output_attentions", None)
for i, (layer_idx, head_idx) in enumerate(LLAMA_ALIGNED_HEADS):
self.last_aligned_attns += [None]
self._add_attention_spy(tfmr, i, layer_idx, head_idx)
@@ -81,11 +85,18 @@ class AlignmentStreamAnalyzer:
target_layer = tfmr.layers[layer_idx].self_attn
# Register hook and store the handle
target_layer.register_forward_hook(attention_forward_hook)
self._hook_handles.append(target_layer.register_forward_hook(attention_forward_hook))
if hasattr(tfmr, 'config') and hasattr(tfmr.config, 'output_attentions'):
self.original_output_attentions = tfmr.config.output_attentions
tfmr.config.output_attentions = True
def close(self, tfmr):
"""Remove inference-scoped hooks and restore the transformer setting."""
for hook_handle in self._hook_handles:
hook_handle.remove()
self._hook_handles.clear()
if self._original_output_attentions is not None:
tfmr.config.output_attentions = self._original_output_attentions
def step(self, logits, next_token=None):
"""
Emits an AlignmentAnalysisResult into the output queue, and potentially modifies the logits to force an EOS.
@@ -235,6 +235,9 @@ class T3(nn.Module):
length_penalty=1.0,
repetition_penalty=1.2,
cfg_weight=0.5,
# TTS Audio Suite patch: V3 follows upstream by disabling the legacy
# multilingual alignment analyzer while V1/V2 retain existing behavior.
use_alignment_analyzer=True,
):
"""
Args:
@@ -265,9 +268,15 @@ class T3(nn.Module):
# TODO? synchronize the expensive compile function
# with self.compile_lock:
if not self.compiled:
# TTS Audio Suite patch: clean up hooks left by an interrupted prior inference.
previous_backend = getattr(self, "patched_model", None)
previous_analyzer = getattr(previous_backend, "alignment_stream_analyzer", None)
if previous_analyzer is not None:
previous_analyzer.close(self.tfmr)
# Default to None for English models, only create for multilingual
alignment_stream_analyzer = None
if self.hp.is_multilingual:
if self.hp.is_multilingual and use_alignment_analyzer:
alignment_stream_analyzer = AlignmentStreamAnalyzer(
self.tfmr,
None,
@@ -331,7 +340,7 @@ class T3(nn.Module):
inputs_embeds=inputs_embeds,
past_key_values=None,
use_cache=True,
output_attentions=True,
output_attentions=use_alignment_analyzer,
output_hidden_states=True,
return_dict=True,
)
@@ -399,4 +408,7 @@ class T3(nn.Module):
# Concatenate all predicted tokens along the sequence dimension.
predicted_tokens = torch.cat(predicted, dim=1) # shape: (B, num_tokens)
if self.patched_model.alignment_stream_analyzer is not None:
self.patched_model.alignment_stream_analyzer.close(self.tfmr)
self.patched_model.alignment_stream_analyzer = None
return predicted_tokens
@@ -6,7 +6,6 @@ import torch
from pathlib import Path
from unicodedata import category
from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download
from utils.text.russian_stress_support import get_russian_text_stresser
@@ -56,9 +55,6 @@ class EnTokenizer:
return txt
# Model repository
REPO_ID = "ResembleAI/chatterbox"
# Global instances for optional dependencies
_kakasi = None
_dicta = None
@@ -167,13 +163,13 @@ class ChineseCangjieConverter:
self._init_segmenter()
def _load_cangjie_mapping(self, model_dir=None):
"""Load Cangjie mapping from HuggingFace model repository."""
"""Load the Cangjie mapping from the organized local model folder."""
try:
cangjie_file = hf_hub_download(
repo_id=REPO_ID,
filename="Cangjie5_TC.json",
cache_dir=model_dir
)
# TTS Audio Suite patch: this asset is downloaded by the unified
# downloader; tokenization must never create a hidden HF cache.
cangjie_file = Path(model_dir) / "Cangjie5_TC.json"
if not cangjie_file.is_file():
raise FileNotFoundError(f"Missing local Cangjie mapping: {cangjie_file}")
with open(cangjie_file, "r", encoding="utf-8") as fp:
data = json.load(fp)
+36 -23
View File
@@ -29,7 +29,7 @@ except ImportError:
PERTH_AVAILABLE = False
from .models.t3 import T3
from .models.s3tokenizer import S3_SR, drop_invalid_tokens
from .models.s3tokenizer import S3_SR, S3_TOKEN_RATE, drop_invalid_tokens
from .models.s3gen import S3GEN_SR, S3Gen
from .models.tokenizers import EnTokenizer, MTLTokenizer
from .models.voice_encoder import VoiceEncoder
@@ -219,7 +219,8 @@ class ChatterboxOfficial23LangTTS:
"""
Load ChatterBox Official 23-Lang multilingual model from local directory.
Expected files:
- t3_23lang.safetensors (multilingual T3 model v1) OR t3_mtl23ls_v2.safetensors (v2)
- t3_23lang.safetensors (v1), t3_mtl23ls_v2.safetensors (v2),
or t3_mtl23ls_v3.safetensors (v3)
- s3gen.pt (S3Gen model)
- ve.pt (Voice encoder)
- mtl_tokenizer.json (multilingual tokenizer)
@@ -281,8 +282,8 @@ class ChatterboxOfficial23LangTTS:
print("📦 Loading multilingual tokenizer...")
tokenizer_path = None
if version_for_files == "v2":
# Try v2 enhanced tokenizer first
if version_for_files in ("v2", "v3"):
# V2 and V3 use the expanded multilingual tokenizer.
candidate_path = ckpt_dir / "grapheme_mtl_merged_expanded_v1.json"
if candidate_path.exists():
tokenizer_path = candidate_path
@@ -327,23 +328,26 @@ class ChatterboxOfficial23LangTTS:
# Support multiple T3 filename patterns:
# - Official v1: t3_23lang.safetensors
# - Official v2: t3_mtl23ls_v2.safetensors
# - Official v2/v3: t3_mtl23ls_v2.safetensors / t3_mtl23ls_v3.safetensors
# - Vietnamese Viterbox: t3_ml24ls_v2.safetensors
# - Egyptian Arabic: t3_mtl23ls_v2.safetensors
# - Future variants: any t3_*.safetensors
t3_path = None
if version_for_files == "v2":
# Try specific v2 patterns first
for pattern in ["t3_mtl23ls_v2.safetensors", "t3_ml24ls_v2.safetensors"]:
if version_for_files in ("v2", "v3"):
patterns = (
["t3_mtl23ls_v3.safetensors"]
if version_for_files == "v3"
else ["t3_mtl23ls_v2.safetensors", "t3_ml24ls_v2.safetensors"]
)
for pattern in patterns:
candidate = ckpt_dir / pattern
if candidate.exists():
t3_path = candidate
break
# Fallback: find any t3_*_v2.safetensors file
if not t3_path:
import glob
matches = list(ckpt_dir.glob("t3_*_v2.safetensors"))
# Fallback stays version-specific so V2 and V3 cannot be mixed.
if not t3_path:
matches = list(ckpt_dir.glob(f"t3_*_{version_for_files}.safetensors"))
if matches:
t3_path = matches[0]
else:
@@ -428,7 +432,7 @@ class ChatterboxOfficial23LangTTS:
warnings.simplefilter("ignore")
print(f"📦 Loading checkpoint: {unified_model_path.name}")
checkpoint = torch.load(unified_model_path, map_location=actual_device)
checkpoint = torch.load(unified_model_path, map_location=actual_device, weights_only=True)
# Extract model configuration
model_config = checkpoint.get('model_config', {})
@@ -505,7 +509,7 @@ class ChatterboxOfficial23LangTTS:
Args:
device: Device to load model on
model_name: Model to load (defaults to "ChatterBox Official 23-Lang")
model_version: Model version - "v1" or "v2" (defaults to "v2")
model_version: Model version - "v1", "v2", or "v3"
"""
# Get model configuration
model_config = get_model_config(model_name)
@@ -689,9 +693,10 @@ class ChatterboxOfficial23LangTTS:
max_new_tokens=1000, # TODO: use the value in config
temperature=temperature,
cfg_weight=cfg_weight,
repetition_penalty=repetition_penalty,
min_p=min_p,
top_p=top_p,
repetition_penalty=repetition_penalty,
min_p=min_p,
top_p=top_p,
use_alignment_analyzer=self.model_version != "v3",
)
# Extract only the conditional batch.
speech_tokens = speech_tokens[0]
@@ -700,12 +705,20 @@ class ChatterboxOfficial23LangTTS:
speech_tokens = drop_invalid_tokens(speech_tokens)
speech_tokens = speech_tokens.to(self.device)
wav, _ = self.s3gen.inference(
speech_tokens=speech_tokens,
ref_dict=self.conds.gen,
)
wav = wav.squeeze(0).detach().cpu().numpy()
if self.enable_watermarking:
wav, _ = self.s3gen.inference(
speech_tokens=speech_tokens,
ref_dict=self.conds.gen,
)
wav = wav.squeeze(0).detach().cpu().numpy()
if self.model_version == "v3":
# TTS Audio Suite patch: match official V3 by dropping the
# final degraded pre-EOS speech-token artifact.
token_count = int(speech_tokens.shape[-1])
clean_token_count = max(1, token_count - 1)
wav = wav[: clean_token_count * (S3GEN_SR // S3_TOKEN_RATE)]
if self.enable_watermarking:
self._init_watermarker_if_needed()
if self.watermarker is not None:
watermarked_wav = self.watermarker.apply_watermark(wav, sample_rate=self.sr)
+2 -2
View File
@@ -114,7 +114,7 @@ class ChatterboxVC:
if safetensors_path.exists():
return load_file(safetensors_path, device=device)
elif pt_path.exists():
return torch.load(pt_path, map_location=device)
return torch.load(pt_path, map_location=device, weights_only=True)
else:
raise FileNotFoundError(f"Neither {base_name}.safetensors nor {base_name}.pt found in {ckpt_dir}")
@@ -123,7 +123,7 @@ class ChatterboxVC:
ref_dict = None
if (builtin_voice := ckpt_dir / "conds.pt").exists():
states = torch.load(builtin_voice)
states = torch.load(builtin_voice, weights_only=True)
ref_dict = states['gen']
s3gen = S3Gen()
+11
View File
@@ -28,6 +28,7 @@ import tempfile
import folder_paths
import numpy as np
from typing import Optional, List, Dict, Any, Generator
import comfy.model_management as model_management
from utils.models.unified_model_interface import unified_model_interface
from utils.models.factory_config import ModelLoadConfig
@@ -59,6 +60,12 @@ class CosyVoiceEngine:
# Generation modes
MODES = ["zero_shot", "instruct", "cross_lingual"]
@staticmethod
def _check_interrupt():
"""Raise immediately when ComfyUI interrupt was requested."""
if model_management.interrupt_processing:
raise InterruptedError("CosyVoice3 generation interrupted by user")
def __init__(self, model_dir: str = "Fun-CosyVoice3-0.5B-RL", device: str = "auto",
use_fp16: bool = False, load_trt: bool = False, load_vllm: bool = False):
@@ -293,6 +300,7 @@ class CosyVoiceEngine:
speed=speed,
text_frontend=text_frontend
)):
self._check_interrupt()
audio_chunk = output['tts_speech']
audio_chunks.append(audio_chunk)
@@ -385,6 +393,7 @@ class CosyVoiceEngine:
speed=speed,
text_frontend=text_frontend
)):
self._check_interrupt()
audio_chunk = output['tts_speech']
audio_chunks.append(audio_chunk)
@@ -471,6 +480,7 @@ class CosyVoiceEngine:
speed=speed,
text_frontend=text_frontend
)):
self._check_interrupt()
audio_chunk = output['tts_speech']
audio_chunks.append(audio_chunk)
@@ -609,6 +619,7 @@ class CosyVoiceEngine:
stream=stream,
speed=speed
):
self._check_interrupt()
audio_chunks.append(output['tts_speech'])
# Concatenate all chunks
@@ -73,7 +73,8 @@ def main():
assert num == len(path_list)
for path in path_list:
print('Processing {}'.format(path))
states = torch.load(path, map_location=torch.device('cpu'))
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
states = torch.load(path, map_location=torch.device('cpu'), weights_only=True)
for k in states.keys():
if k not in ['step', 'epoch']:
if k not in avg.keys():
@@ -134,7 +134,8 @@ def main():
start_step, start_epoch = 0, -1
if args.checkpoint is not None:
if os.path.exists(args.checkpoint):
state_dict = torch.load(args.checkpoint, map_location='cpu')
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
state_dict = torch.load(args.checkpoint, map_location='cpu', weights_only=True)
model.load_state_dict(state_dict, strict=False)
if 'step' in state_dict:
start_step = state_dict['step']
@@ -161,7 +162,8 @@ def main():
# DPO related
if args.dpo is True:
ref_model = deepcopy(configs[args.model])
state_dict = torch.load(args.ref_model, map_location='cpu')
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
state_dict = torch.load(args.ref_model, map_location='cpu', weights_only=True)
ref_model.load_state_dict(state_dict, strict=False)
dpo_loss = DPOLoss(beta=0.01, label_smoothing=0.0, ipo=False)
# NOTE maybe it is not needed to wrap ref_model as ddp because its parameter is not updated
@@ -270,7 +270,6 @@ class CosyVoice3(CosyVoice2):
# PATCH: Added llm_filename parameter to support model variants (llm.pt vs llm.rl.pt)
def __init__(self, model_dir, load_trt=False, load_vllm=False, fp16=False, trt_concurrent=1, llm_filename='llm.pt'):
self.model_dir = model_dir
self.fp16 = fp16
if not os.path.exists(model_dir):
model_dir = snapshot_download(model_dir)
hyper_yaml_path = '{}/cosyvoice3.yaml'.format(model_dir)
@@ -289,6 +288,15 @@ class CosyVoice3(CosyVoice2):
if torch.cuda.is_available() is False and (load_trt is True or fp16 is True):
load_trt, fp16 = False, False
logging.warning('no cuda device, set load_trt/fp16 to False')
# TTS Audio Suite patch: ROCm cannot use CosyVoice3's FP16 flow/vocoder
# path reliably; CosyVoice3Model applies BF16 only to the bundled Qwen LLM.
elif torch.version.hip:
if fp16:
logging.warning('ROCm detected: CosyVoice3 disables FP16 for its flow/vocoder and uses BF16 only for the Qwen LLM.')
if load_trt:
logging.warning('ROCm detected: TensorRT is unavailable, disabling CosyVoice3 TensorRT loading.')
load_trt, fp16 = False, False
self.fp16 = fp16
self.model = CosyVoice3Model(configs['llm'], configs['flow'], configs['hift'], fp16)
# PATCH: Use llm_filename parameter instead of hardcoded 'llm.pt'
self.model.load('{}/{}'.format(model_dir, llm_filename),
@@ -47,7 +47,8 @@ class CosyVoiceFrontEnd:
providers=["CUDAExecutionProvider" if torch.cuda.is_available() else
"CPUExecutionProvider"])
if os.path.exists(spk2info):
self.spk2info = torch.load(spk2info, map_location=self.device)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
self.spk2info = torch.load(spk2info, map_location=self.device, weights_only=True)
else:
self.spk2info = {}
self.allowed_special = allowed_special
+75 -26
View File
@@ -38,6 +38,7 @@ class CosyVoiceModel:
self.flow = flow
self.hift = hift
self.fp16 = fp16
self._rocm_bf16_llm_autocast = False
self.token_min_hop_len = 2 * self.flow.input_frame_rate
self.token_max_hop_len = 4 * self.flow.input_frame_rate
self.token_overlap_len = 20
@@ -57,12 +58,14 @@ class CosyVoiceModel:
# dict used to store session related variable
self.tts_speech_token_dict = {}
self.llm_end_dict = {}
self.llm_error_dict = {}
self.mel_overlap_dict = {}
self.flow_cache_dict = {}
self.hift_cache_dict = {}
def load(self, llm_model, flow_model, hift_model):
self.llm.load_state_dict(torch.load(llm_model, map_location=self.device), strict=True)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
self.llm.load_state_dict(torch.load(llm_model, map_location=self.device, weights_only=True), strict=True)
self.llm.to(self.device).eval()
# FIX: Bundled code fix - Ensure all nested modules are moved to device
# Issue: Some nested layers (embed_tokens, etc.) weren't being moved with .to(device)
@@ -70,14 +73,16 @@ class CosyVoiceModel:
for module in self.llm.modules():
module.to(self.device)
self.flow.load_state_dict(torch.load(flow_model, map_location=self.device), strict=True)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
self.flow.load_state_dict(torch.load(flow_model, map_location=self.device, weights_only=True), strict=True)
self.flow.to(self.device).eval()
# FIX: Bundled code fix - Ensure all nested modules are moved to device
for module in self.flow.modules():
module.to(self.device)
# in case hift_model is a hifigan model
hift_state_dict = {k.replace('generator.', ''): v for k, v in torch.load(hift_model, map_location=self.device).items()}
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
hift_state_dict = {k.replace('generator.', ''): v for k, v in torch.load(hift_model, map_location=self.device, weights_only=True).items()}
self.hift.load_state_dict(hift_state_dict, strict=True)
self.hift.to(self.device).eval()
# FIX: Bundled code fix - Ensure all nested modules are moved to device
@@ -111,29 +116,61 @@ class CosyVoiceModel:
input_names = ["x", "mask", "mu", "t", "spks", "cond"]
return {'min_shape': min_shape, 'opt_shape': opt_shape, 'max_shape': max_shape, 'input_names': input_names}
# TTS Audio Suite patch: ROCm needs BF16 for the bundled Qwen LLM while
# CosyVoice's flow model and vocoder remain in FP32 to prevent silent audio.
def _llm_autocast_context(self):
if hasattr(self.llm, 'vllm'):
return nullcontext()
if self._rocm_bf16_llm_autocast:
return torch.amp.autocast('cuda', dtype=torch.bfloat16)
if self.fp16:
return torch.amp.autocast('cuda', dtype=torch.float16)
return nullcontext()
# TTS Audio Suite patch: forward bundled LLM worker failures to the caller
# instead of trying to decode an empty token sequence.
def _raise_if_llm_failed(self, uuid):
error = self.llm_error_dict.pop(uuid, None)
if error is None:
return
with self.lock:
self.tts_speech_token_dict.pop(uuid, None)
self.llm_end_dict.pop(uuid, None)
self.hift_cache_dict.pop(uuid, None)
if hasattr(self, 'mel_overlap_dict'):
self.mel_overlap_dict.pop(uuid, None)
if hasattr(self, 'flow_cache_dict'):
self.flow_cache_dict.pop(uuid, None)
raise RuntimeError(f'CosyVoice LLM inference failed: {error}') from error
def llm_job(self, text, prompt_text, llm_prompt_speech_token, llm_embedding, uuid, progress_callback=None):
with self.llm_context, torch.cuda.amp.autocast(self.fp16 is True and hasattr(self.llm, 'vllm') is False):
if isinstance(text, Generator):
assert (self.__class__.__name__ != 'CosyVoiceModel') and not hasattr(self.llm, 'vllm'), 'streaming input text is only implemented for CosyVoice2/3 and do not support vllm!'
for i in self.llm.inference_bistream(text=text,
prompt_text=prompt_text.to(self.device),
prompt_text_len=torch.tensor([prompt_text.shape[1]], dtype=torch.int32).to(self.device),
prompt_speech_token=llm_prompt_speech_token.to(self.device),
prompt_speech_token_len=torch.tensor([llm_prompt_speech_token.shape[1]], dtype=torch.int32).to(self.device),
embedding=llm_embedding.to(self.device)):
self.tts_speech_token_dict[uuid].append(i)
else:
for i in self.llm.inference(text=text.to(self.device),
text_len=torch.tensor([text.shape[1]], dtype=torch.int32).to(self.device),
prompt_text=prompt_text.to(self.device),
prompt_text_len=torch.tensor([prompt_text.shape[1]], dtype=torch.int32).to(self.device),
prompt_speech_token=llm_prompt_speech_token.to(self.device),
prompt_speech_token_len=torch.tensor([llm_prompt_speech_token.shape[1]], dtype=torch.int32).to(self.device),
embedding=llm_embedding.to(self.device),
uuid=uuid,
progress_callback=progress_callback):
self.tts_speech_token_dict[uuid].append(i)
self.llm_end_dict[uuid] = True
# TTS Audio Suite patch: capture worker failures for _raise_if_llm_failed().
try:
with self.llm_context, self._llm_autocast_context():
if isinstance(text, Generator):
assert (self.__class__.__name__ != 'CosyVoiceModel') and not hasattr(self.llm, 'vllm'), 'streaming input text is only implemented for CosyVoice2/3 and do not support vllm!'
for i in self.llm.inference_bistream(text=text,
prompt_text=prompt_text.to(self.device),
prompt_text_len=torch.tensor([prompt_text.shape[1]], dtype=torch.int32).to(self.device),
prompt_speech_token=llm_prompt_speech_token.to(self.device),
prompt_speech_token_len=torch.tensor([llm_prompt_speech_token.shape[1]], dtype=torch.int32).to(self.device),
embedding=llm_embedding.to(self.device)):
self.tts_speech_token_dict[uuid].append(i)
else:
for i in self.llm.inference(text=text.to(self.device),
text_len=torch.tensor([text.shape[1]], dtype=torch.int32).to(self.device),
prompt_text=prompt_text.to(self.device),
prompt_text_len=torch.tensor([prompt_text.shape[1]], dtype=torch.int32).to(self.device),
prompt_speech_token=llm_prompt_speech_token.to(self.device),
prompt_speech_token_len=torch.tensor([llm_prompt_speech_token.shape[1]], dtype=torch.int32).to(self.device),
embedding=llm_embedding.to(self.device),
uuid=uuid,
progress_callback=progress_callback):
self.tts_speech_token_dict[uuid].append(i)
except Exception as error:
self.llm_error_dict[uuid] = error
finally:
self.llm_end_dict[uuid] = True
def vc_job(self, source_speech_token, uuid):
self.tts_speech_token_dict[uuid] = source_speech_token.flatten().tolist()
@@ -188,6 +225,7 @@ class CosyVoiceModel:
this_uuid = str(uuid.uuid1())
with self.lock:
self.tts_speech_token_dict[this_uuid], self.llm_end_dict[this_uuid] = [], False
self.llm_error_dict[this_uuid] = None
self.hift_cache_dict[this_uuid] = None
self.mel_overlap_dict[this_uuid] = torch.zeros(1, 80, 0)
self.flow_cache_dict[this_uuid] = torch.zeros(1, 80, 0, 2)
@@ -217,6 +255,7 @@ class CosyVoiceModel:
if self.llm_end_dict[this_uuid] is True and len(self.tts_speech_token_dict[this_uuid]) < token_hop_len + self.token_overlap_len:
break
p.join()
self._raise_if_llm_failed(this_uuid)
# deal with remain tokens, make sure inference remain token len equals token_hop_len when cache_speech is not None
this_tts_speech_token = torch.tensor(self.tts_speech_token_dict[this_uuid]).unsqueeze(dim=0)
this_tts_speech = self.token2wav(token=this_tts_speech_token,
@@ -229,6 +268,7 @@ class CosyVoiceModel:
else:
# deal with all tokens
p.join()
self._raise_if_llm_failed(this_uuid)
this_tts_speech_token = torch.tensor(self.tts_speech_token_dict[this_uuid]).unsqueeze(dim=0)
this_tts_speech = self.token2wav(token=this_tts_speech_token,
prompt_token=flow_prompt_speech_token,
@@ -241,6 +281,7 @@ class CosyVoiceModel:
with self.lock:
self.tts_speech_token_dict.pop(this_uuid)
self.llm_end_dict.pop(this_uuid)
self.llm_error_dict.pop(this_uuid, None)
self.mel_overlap_dict.pop(this_uuid)
self.hift_cache_dict.pop(this_uuid)
self.flow_cache_dict.pop(this_uuid)
@@ -274,6 +315,7 @@ class CosyVoice2Model(CosyVoiceModel):
# dict used to store session related variable
self.tts_speech_token_dict = {}
self.llm_end_dict = {}
self.llm_error_dict = {}
self.hift_cache_dict = {}
def load_jit(self, flow_encoder_model):
@@ -336,6 +378,7 @@ class CosyVoice2Model(CosyVoiceModel):
this_uuid = str(uuid.uuid1())
with self.lock:
self.tts_speech_token_dict[this_uuid], self.llm_end_dict[this_uuid] = [], False
self.llm_error_dict[this_uuid] = None
self.hift_cache_dict[this_uuid] = None
if source_speech_token.shape[1] == 0:
p = threading.Thread(target=self.llm_job, args=(text, prompt_text, llm_prompt_speech_token, llm_embedding, this_uuid, progress_callback))
@@ -363,6 +406,7 @@ class CosyVoice2Model(CosyVoiceModel):
if self.llm_end_dict[this_uuid] is True and len(self.tts_speech_token_dict[this_uuid]) - token_offset < this_token_hop_len + self.flow.pre_lookahead_len:
break
p.join()
self._raise_if_llm_failed(this_uuid)
# deal with remain tokens, make sure inference remain token len equals token_hop_len when cache_speech is not None
this_tts_speech_token = torch.tensor(self.tts_speech_token_dict[this_uuid]).unsqueeze(dim=0)
this_tts_speech = self.token2wav(token=this_tts_speech_token,
@@ -376,6 +420,7 @@ class CosyVoice2Model(CosyVoiceModel):
else:
# deal with all tokens
p.join()
self._raise_if_llm_failed(this_uuid)
this_tts_speech_token = torch.tensor(self.tts_speech_token_dict[this_uuid]).unsqueeze(dim=0)
this_tts_speech = self.token2wav(token=this_tts_speech_token,
prompt_token=flow_prompt_speech_token,
@@ -389,6 +434,7 @@ class CosyVoice2Model(CosyVoiceModel):
with self.lock:
self.tts_speech_token_dict.pop(this_uuid)
self.llm_end_dict.pop(this_uuid)
self.llm_error_dict.pop(this_uuid, None)
self.hift_cache_dict.pop(this_uuid)
if torch.cuda.is_available():
torch.cuda.empty_cache()
@@ -406,7 +452,9 @@ class CosyVoice3Model(CosyVoice2Model):
self.llm = llm
self.flow = flow
self.hift = hift
self.fp16 = fp16
# TTS Audio Suite patch: enable BF16 LLM autocast on ROCm only.
self._rocm_bf16_llm_autocast = torch.cuda.is_available() and bool(torch.version.hip)
self.fp16 = fp16 and not self._rocm_bf16_llm_autocast
# NOTE must matching training static_chunk_size
self.token_hop_len = 25
# rtf and decoding related
@@ -415,6 +463,7 @@ class CosyVoice3Model(CosyVoice2Model):
# dict used to store session related variable
self.tts_speech_token_dict = {}
self.llm_end_dict = {}
self.llm_error_dict = {}
self.hift_cache_dict = {}
def token2wav(self, token, prompt_token, prompt_feat, embedding, token_offset, uuid, stream=False, finalize=False, speed=1.0):
@@ -84,7 +84,8 @@ def assert_required_models_available(args):
def load_hifigan(checkpoint_path, device):
h = AttrDict(v1)
hifigan = HiFiGAN(h).to(device)
hifigan.load_state_dict(torch.load(checkpoint_path, map_location=device)["generator"])
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
hifigan.load_state_dict(torch.load(checkpoint_path, map_location=device, weights_only=True)["generator"])
_ = hifigan.eval()
hifigan.remove_weight_norm()
return hifigan
@@ -41,7 +41,8 @@ def get_padding(kernel_size, dilation=1):
def load_checkpoint(filepath, device):
assert os.path.isfile(filepath)
print(f"Loading '{filepath}'")
checkpoint_dict = torch.load(filepath, map_location=device)
# TTS Audio Suite patch: checkpoint data must not execute pickle globals.
checkpoint_dict = torch.load(filepath, map_location=device, weights_only=True)
print("Complete.")
return checkpoint_dict
+6
View File
@@ -0,0 +1,6 @@
"""Dots TTS engine integration."""
from .dots_tts_engine import DotsTTSEngine
from .dots_tts_downloader import DotsTTSDownloader
__all__ = ["DotsTTSEngine", "DotsTTSDownloader"]
+146
View File
@@ -0,0 +1,146 @@
"""
Dots TTS model downloader.
Downloads official Rednote dots.tts checkpoints into
ComfyUI/models/TTS/dots_tts/ instead of hidden Hugging Face cache folders.
"""
import os
from typing import Dict, Optional
import folder_paths
from huggingface_hub import snapshot_download
from utils.hf_download_logging import quiet_hf_download_logs
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
from utils.security.path_access import allowed_path, validate_model_paths, allowed_path
class DotsTTSDownloader:
"""Resolve and download official dots.tts model folders."""
MODELS: Dict[str, Dict[str, str]] = {
"dots.tts-base": {
"repo_id": "rednote-hilab/dots.tts-base",
"description": "Official Dots TTS base checkpoint",
},
"dots.tts-soar": {
"repo_id": "rednote-hilab/dots.tts-soar",
"description": "Official Dots TTS SOAR checkpoint",
},
"dots.tts-mf": {
"repo_id": "rednote-hilab/dots.tts-mf",
"description": "Official Dots TTS MeanFlow checkpoint",
},
}
REQUIRED_FILES = [
"added_tokens.json",
"chat_template.jinja",
"config.json",
"latent_stats.pt",
"llm_config.json",
"merges.txt",
"model.safetensors",
"speaker_encoder.safetensors",
"special_tokens_map.json",
"tokenizer.json",
"tokenizer_config.json",
"vocab.json",
"vocoder.safetensors",
]
def __init__(self, base_path: Optional[str] = None):
if base_path is None:
try:
self.base_path = get_preferred_download_path(model_type="TTS", engine_name="dots_tts")
except Exception:
self.base_path = os.path.join(folder_paths.models_dir, "TTS", "dots_tts")
else:
self.base_path = base_path
os.makedirs(self.base_path, exist_ok=True)
@validate_model_paths
def resolve_model_path(self, model_identifier: str) -> str:
"""Resolve local: names, absolute paths, or known official model identifiers."""
if not model_identifier:
model_identifier = "dots.tts-soar"
if os.path.isabs(model_identifier) and os.path.exists(model_identifier):
return model_identifier
if os.path.isdir(model_identifier):
return model_identifier
if model_identifier.startswith("local:"):
local_name = model_identifier[6:]
for base_path in get_all_tts_model_paths("TTS"):
for folder_name in ("dots_tts", "DOTS", ""):
candidate = os.path.join(base_path, folder_name, local_name) if folder_name else os.path.join(base_path, local_name)
if os.path.exists(candidate):
print(f"📁 Using local Dots TTS model: {candidate}")
return candidate
raise FileNotFoundError(f"Local Dots TTS model not found: {local_name}")
if model_identifier in self.MODELS:
return self.get_model_path(model_identifier)
raise FileNotFoundError(f"Unknown Dots TTS model identifier: {model_identifier}")
def get_model_path(self, model_name: str) -> str:
model_dir = os.path.join(self.base_path, model_name)
if not os.path.exists(model_dir) or not self._is_model_complete(model_dir):
return self.download_model(model_name, force=os.path.exists(model_dir))
return model_dir
def download_model(self, model_name: str, force: bool = False) -> str:
if model_name not in self.MODELS:
raise ValueError(f"Unknown Dots TTS model: {model_name}")
model_info = self.MODELS[model_name]
repo_id = model_info["repo_id"]
model_dir = os.path.join(self.base_path, model_name)
print(f"\n{'=' * 60}")
print("📦 Dots TTS Model Download")
print(f"{'=' * 60}")
print(f"Model: {model_name}")
print(f"Description: {model_info['description']}")
print(f"Repository: {repo_id}")
print(f"Target: {model_dir}")
print(f"{'=' * 60}\n")
try:
with quiet_hf_download_logs():
snapshot_download(
repo_id=repo_id,
local_dir=allowed_path(model_dir),
local_dir_use_symlinks=False,
resume_download=True,
force_download=force,
)
except Exception as e:
raise RuntimeError(f"Failed to download {model_name} from {repo_id}: {e}") from e
if not self._is_model_complete(model_dir):
raise RuntimeError(f"Downloaded Dots TTS model is incomplete: {model_dir}")
print(f"✅ Dots TTS model ready: {model_dir}")
return model_dir
def _is_model_complete(self, model_dir: str) -> bool:
if not os.path.isdir(model_dir):
return False
missing = [
rel_path
for rel_path in self.REQUIRED_FILES
if not os.path.exists(os.path.join(model_dir, rel_path))
]
if missing:
print(f"❌ Dots TTS model incomplete. Missing {len(missing)} file(s):")
for rel_path in missing:
print(f" - {rel_path}")
return False
return True
+479
View File
@@ -0,0 +1,479 @@
"""
Dots TTS engine wrapper.
Wraps the official dots.tts runtime with ComfyUI-friendly model lifecycle hooks.
"""
from utils.security.data_loading import RestrictedLoadModule, load_dependency_checkpoint
import os
import sys
import types
import importlib
import logging
import time
import warnings
import gc
from contextlib import contextmanager
from typing import Any, Dict, Optional
import torch
current_dir = os.path.dirname(__file__)
engines_dir = os.path.dirname(current_dir)
project_root = os.path.dirname(engines_dir)
if project_root not in sys.path:
sys.path.insert(0, project_root)
from engines.dots_tts.dots_tts_downloader import DotsTTSDownloader
from engines.dots_tts.progress_callback import DotsTTSProgressTracker
from utils.device import resolve_torch_device
class _SuppressDotsModelTypeWarning(logging.Filter):
"""Drop the upstream bogus model-type warning while keeping real ones."""
def filter(self, record: logging.LogRecord) -> bool:
try:
message = record.getMessage()
except Exception:
return True
return "model of type `dots_tts` to instantiate a model of type" not in message
class DotsTTSEngine:
"""Thin wrapper around the official DotsTtsRuntime."""
SAMPLE_RATE = 48000
_normalizer_warning_shown = False
def __init__(
self,
model_name: str = "dots.tts-soar",
device: str = "auto",
precision: str = "auto",
optimize: bool = False,
max_generate_length: int = 500,
model_dir: Optional[str] = None,
):
self.model_name = model_name
self.device = resolve_torch_device(device)
self.precision = self._resolve_precision(precision, self.device)
self.optimize = bool(optimize)
self.max_generate_length = int(max_generate_length)
self.model_dir = model_dir or DotsTTSDownloader().resolve_model_path(model_name)
self._runtime = None
@staticmethod
def _resolve_precision(precision: str, device: str) -> str:
normalized = str(precision or "auto").lower()
if normalized in {"bfloat16", "float16", "float32"}:
if device == "cpu" and normalized != "float32":
return "float32"
return normalized
if device == "cpu":
return "float32"
if torch.cuda.is_available():
major, _minor = torch.cuda.get_device_capability()
if major >= 8:
return "bfloat16"
return "float16"
def _import_runtime(self):
importlib.invalidate_caches()
nodes_dir = os.path.join(project_root, "nodes")
blocked_paths = {
os.path.abspath(engines_dir),
os.path.abspath(nodes_dir),
}
original_sys_path = list(sys.path)
# ComfyUI can place the suite's `engines/` directory directly on sys.path.
# That makes our local `engines/dots_tts` package shadow the pip-installed
# official `dots_tts` package we actually need here.
sys.path[:] = [
path for path in sys.path
if os.path.abspath(path or os.getcwd()) not in blocked_paths
]
stale_modules = []
for module_name, module in list(sys.modules.items()):
if not (module_name == "dots_tts" or module_name.startswith("dots_tts.")):
continue
module_file = getattr(module, "__file__", "") or ""
if module_file and os.path.abspath(module_file).startswith(os.path.abspath(project_root)):
stale_modules.append((module_name, module))
del sys.modules[module_name]
try:
with self._text_normalizer_compat():
from dots_tts.runtime import DotsTtsRuntime
from dots_tts.models.dots_tts import core
if not isinstance(core.torch, RestrictedLoadModule):
core.torch = RestrictedLoadModule(core.torch, load_dependency_checkpoint)
except Exception as e:
for module_name, module in stale_modules:
sys.modules.setdefault(module_name, module)
raise ImportError(
"Failed to import the official `dots_tts` package in the active ComfyUI Python "
"environment. This is usually either a missing install or an import-path "
f"collision. Active Python: {sys.executable}"
) from e
finally:
sys.path[:] = original_sys_path
return DotsTtsRuntime
@staticmethod
@contextmanager
def _suppress_noisy_runtime_logs():
"""Mute verbose Dots chatter while keeping actionable Transformers warnings."""
dots_logger = None
dots_logger_disabled = False
transformers_filter = _SuppressDotsModelTypeWarning()
transformer_loggers = []
try:
from loguru import logger as dots_logger
dots_logger.disable("dots_tts")
dots_logger_disabled = True
transformer_loggers = [
logging.getLogger("transformers"),
logging.getLogger("transformers.configuration_utils"),
logging.getLogger("transformers.modeling_utils"),
logging.getLogger("transformers.tokenization_utils_base"),
]
for logger in transformer_loggers:
logger.addFilter(transformers_filter)
except Exception:
pass
try:
with warnings.catch_warnings():
warnings.filterwarnings(
"ignore",
message=r".*model of type `dots_tts` to instantiate a model of type.*",
)
yield
finally:
if dots_logger is not None and dots_logger_disabled:
try:
dots_logger.enable("dots_tts")
except Exception:
pass
for logger in transformer_loggers:
try:
logger.removeFilter(transformers_filter)
except Exception:
pass
@staticmethod
@contextmanager
def _patch_tokenizer_regex_fix():
"""Force the upstream Dots tokenizer load to opt into the Mistral regex fix."""
try:
from transformers import AutoTokenizer
except Exception:
yield
return
original_from_pretrained = AutoTokenizer.from_pretrained
def patched_from_pretrained(*args, **kwargs):
kwargs.setdefault("fix_mistral_regex", True)
return original_from_pretrained(*args, **kwargs)
AutoTokenizer.from_pretrained = patched_from_pretrained
try:
yield
finally:
AutoTokenizer.from_pretrained = original_from_pretrained
@staticmethod
@contextmanager
def _text_normalizer_compat():
"""Temporarily provide Dots' expected tn package when unavailable."""
normalizers_available = False
try:
from tn.chinese.normalizer import Normalizer as _ZhNormalizer # noqa: F401
from tn.english.normalizer import Normalizer as _EnNormalizer # noqa: F401
normalizers_available = True
except Exception:
pass
if normalizers_available:
yield
return
class _NoOpNormalizer:
def normalize(self, text: str) -> str:
return text
tn_module = types.ModuleType("tn")
chinese_module = types.ModuleType("tn.chinese")
chinese_normalizer_module = types.ModuleType("tn.chinese.normalizer")
english_module = types.ModuleType("tn.english")
english_normalizer_module = types.ModuleType("tn.english.normalizer")
# Mark package modules as packages so nested imports work even when an
# unrelated top-level module named `tn` is already installed.
tn_module.__path__ = []
chinese_module.__path__ = []
english_module.__path__ = []
chinese_normalizer_module.Normalizer = _NoOpNormalizer
english_normalizer_module.Normalizer = _NoOpNormalizer
chinese_module.normalizer = chinese_normalizer_module
english_module.normalizer = english_normalizer_module
tn_module.chinese = chinese_module
tn_module.english = english_module
fallback_modules = {
"tn": tn_module,
"tn.chinese": chinese_module,
"tn.chinese.normalizer": chinese_normalizer_module,
"tn.english": english_module,
"tn.english.normalizer": english_normalizer_module,
}
missing = object()
previous_modules = {
name: sys.modules.get(name, missing)
for name in fallback_modules
}
sys.modules.update(fallback_modules)
try:
if not DotsTTSEngine._normalizer_warning_shown:
print("[Dots TTS] WeTextProcessing not available; normalize_text will use a no-op fallback")
DotsTTSEngine._normalizer_warning_shown = True
yield
finally:
for name, previous in previous_modules.items():
if previous is missing:
sys.modules.pop(name, None)
else:
sys.modules[name] = previous
def _ensure_runtime_loaded(self):
if self._runtime is not None:
return
DotsTtsRuntime = self._import_runtime()
with self._patch_tokenizer_regex_fix(), self._suppress_noisy_runtime_logs():
self._runtime = DotsTtsRuntime.from_pretrained(
self.model_dir,
precision=self.precision,
optimize=self.optimize,
max_generate_length=self.max_generate_length,
)
self._ensure_runtime_device()
def _current_runtime_device(self) -> Optional[str]:
if self._runtime is None:
return None
try:
first_param = next(self._runtime.model.parameters())
return str(first_param.device)
except Exception:
return str(getattr(self._runtime, "device", self.device))
def _move_runtime_to(self, target_device: str):
if self._runtime is None:
return
device_obj = torch.device(target_device)
runtime = self._runtime
runtime.model = runtime.model.to(device_obj).eval()
runtime.device = device_obj
def _ensure_runtime_device(self):
if self._runtime is None:
return
target_device = resolve_torch_device(self.device)
current_device = self._current_runtime_device()
if self._devices_equivalent(current_device, target_device):
return
if current_device != target_device:
print(f"🔄 Dots TTS: moving runtime from {current_device} to {target_device}")
self._move_runtime_to(target_device)
@staticmethod
def _devices_equivalent(current_device: Optional[str], target_device: Optional[str]) -> bool:
if current_device == target_device:
return True
if not current_device or not target_device:
return False
current = str(current_device)
target = str(target_device)
if current.startswith("cuda") and target.startswith("cuda"):
return current in {"cuda", "cuda:0"} and target in {"cuda", "cuda:0"}
return False
@staticmethod
def _clear_runtime_caches(runtime: Any) -> None:
if runtime is None:
return
model = getattr(runtime, "model", None)
if model is not None:
try:
if hasattr(model, "set_optimize"):
model.set_optimize(False)
except Exception:
pass
for attr_name in (
"_compiled_models",
"_prompt_feature_cache",
"_static_generate_workspaces",
"_fm_decode_workspaces",
):
try:
cache = getattr(model, attr_name, None)
if hasattr(cache, "clear"):
cache.clear()
except Exception:
pass
for attr_name in (
"_double_streaming_prompt_g_cond_cache",
"_double_streaming_silence_audio_patch_cache",
):
try:
cache = getattr(runtime, attr_name, None)
if hasattr(cache, "clear"):
cache.clear()
except Exception:
pass
def unload_runtime(self):
"""Fully tear down the Dots runtime so Clear VRAM actually releases it."""
runtime = self._runtime
if runtime is None:
if torch.cuda.is_available():
torch.cuda.empty_cache()
if hasattr(torch.cuda, "ipc_collect"):
try:
torch.cuda.ipc_collect()
except Exception:
pass
gc.collect()
return
self._clear_runtime_caches(runtime)
model = getattr(runtime, "model", None)
try:
if model is not None and hasattr(model, "to"):
model.to("cpu")
except Exception:
pass
try:
runtime.model = None
except Exception:
pass
self._runtime = None
try:
import torch._dynamo as torch_dynamo
torch_dynamo.reset()
except Exception:
pass
del model
del runtime
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
if hasattr(torch.cuda, "ipc_collect"):
try:
torch.cuda.ipc_collect()
except Exception:
pass
def to(self, device):
"""ComfyUI unload/reload hook."""
self.device = str(device) if isinstance(device, str) else str(torch.device(device))
self._ensure_runtime_device()
return self
def unload(self):
"""Explicit unload hook used by engine-specific ComfyUI handlers."""
self.unload_runtime()
def generate(
self,
text: str,
prompt_audio_path: Optional[str] = None,
prompt_text: Optional[str] = None,
language: Optional[str] = None,
num_steps: int = 10,
guidance_scale: float = 1.2,
speaker_scale: float = 1.5,
normalize_text: bool = False,
template_name: Optional[str] = "tts",
seed: int = 42,
) -> Dict[str, Any]:
self._ensure_runtime_loaded()
self._ensure_runtime_device()
seed_value = int(seed or 0)
if seed_value > 0:
torch.manual_seed(seed_value)
if torch.cuda.is_available():
torch.cuda.manual_seed_all(seed_value)
chunks = []
tracker = DotsTTSProgressTracker(
max_generate_length=self.max_generate_length,
sample_rate=self.SAMPLE_RATE,
)
start_time = time.time()
with self._patch_tokenizer_regex_fix(), self._suppress_noisy_runtime_logs():
stream = self._runtime.generate_stream(
text=text,
prompt_audio_path=prompt_audio_path,
prompt_text=prompt_text,
language=language,
template_name=template_name,
num_steps=int(num_steps),
guidance_scale=float(guidance_scale),
speaker_scale=float(speaker_scale),
normalize_text=bool(normalize_text),
)
for chunk in stream:
if not isinstance(chunk, torch.Tensor):
chunk = torch.tensor(chunk, dtype=torch.float32)
chunk = chunk.detach().float().cpu()
if chunk.dim() > 1:
chunk = chunk.squeeze()
chunks.append(chunk)
tracker.update(chunk.shape[-1] if chunk.numel() > 0 else 0)
tracker.end()
if chunks:
audio = torch.cat(chunks, dim=-1)
else:
audio = torch.zeros(0, dtype=torch.float32)
time_used = time.time() - start_time
duration_seconds = audio.shape[-1] / self.SAMPLE_RATE if audio.numel() > 0 else 0.0
rtf = time_used / duration_seconds if duration_seconds > 0 else float("inf")
return {
"fid": None,
"audio": audio,
"sample_rate": self.SAMPLE_RATE,
"time_used": time_used,
"rtf": rtf,
"profiling": None,
}

Some files were not shown because too many files have changed in this diff Show More