feat: add ASR, Podcast Splitter, and Podcast Stitcher nodes for anti-leakage pipeline
- Add AIIA_ASR node with FunASR (paraformer-zh + SenseVoiceSmall) - Add AIIA_Podcast_Splitter for per-speaker text splitting - Add AIIA_Podcast_Stitcher with 3-tier fuzzy alignment (exact, Levenshtein, fallback) - Register 5 previously missing modules in __init__.py - Update README with installation guide and node documentation
This commit is contained in:
Hawk Lee
Executable → Regular
Executable → Regular
+1
-2
@@ -2,5 +2,4 @@ GEMINI.md
|
||||
.gemini/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.DS_Store
|
||||
.env
|
||||
.DS_Store
|
||||
@@ -1,6 +1,6 @@
|
||||
|
||||
|
||||
<h1><img src="assets/logo_small.png" alt="AIIA Logo" align="center" /> AIIA Nodes for ComfyUI</h1>
|
||||
# AIIA Nodes for ComfyUI
|
||||
|
||||
欢迎来到 AIIA Nodes for ComfyUI 仓库！这是一个旨在为 ComfyUI 提供一系列强大、直观且高度可定制的节点的集合。这些节点专注于简化复杂的工作流，并为创意工作者提供最大的灵活性。
|
||||
|
||||
@@ -13,7 +13,6 @@
|
||||
**还在费力地翻找 `output` 文件夹，或者对着一堆时间戳命名的文件猜内容吗？**
|
||||
|
||||
我们隆重推出 **AIIA 媒体浏览器**——一个完全集成在 ComfyUI 内部的、功能完备的媒体文件管理中心。它的诞生，旨在彻底改变你管理和使用生成结果的方式，让整个过程变得高效、直观且充满乐趣。
|
||||
|
||||
|
||||
|
||||
### ✨ 为何选择 AIIA 浏览器？
|
||||
@@ -59,21 +58,14 @@
|
||||
|
||||
处理成百上千张高清图像帧时，轻易就会耗尽 VRAM 和系统内存，导致工作流中断。AIIA 节点通过 **增量式处理（Incremental Processing）** 的策略从根本上解决了这个问题。
|
||||
|
||||
#### v1.9.21+ 内存优化
|
||||
无论是从磁盘流式读取帧进行视频合并，还是将生成结果逐帧保存到磁盘，我们的节点都**永远不需要将所有图像一次性加载到内存中**。这意味着您可以轻松生成数千帧的视频，而无需再为内存限制而烦恼。
|
||||
|
||||
自 v1.9.21 起，**AIIA Body Sway** 和 **AIIA Video Combine** 节点实现了激进的内存管理策略：
|
||||
* **分批处理**：每次只处理 50 帧，处理完立即释放中间变量。
|
||||
* **逐帧释放**：每帧转换后立即 `del` 并定期 `gc.collect()`。
|
||||
* **GPU 内存同步释放**：处理前移至 CPU 并调用 `torch.cuda.empty_cache()`。
|
||||
|
||||
这意味着即使处理 **1500+ 帧的高分辨率视频**（如 1288×1920），也能在合理的内存占用下完成，**无需磁盘中转**。
|
||||
|
||||
#### 两种工作模式
|
||||
### ✨ 无缝与高效的平衡
|
||||
|
||||
我们提供了两种工作模式，以适应不同场景：
|
||||
|
||||
- **内存模式 (推荐)**: 直接将上游节点的 `IMAGE` 张量输入。v1.9.21+ 的优化使其可处理数千帧而不 OOM。
|
||||
- **磁盘模式**: 对于极端长序列或内存受限环境，仍可通过 `frames_directory` 从磁盘流式读取帧。
|
||||
- **内存模式**: 对于短序列或测试，可以直接将上游节点的 `IMAGE` 张量输入，实现无缝、快速的内存内处理。
|
||||
- **磁盘模式**: 对于长序列的最终生成，节点会高效地从磁盘流式读取/写入帧，保证了稳定性和极低的内存占用。
|
||||
|
||||
### 🔧 强大且可扩展的预设系统
|
||||
|
||||
@@ -91,15 +83,7 @@
|
||||
- **强烈建议**将 FFmpeg 的 `bin` 目录添加到您系统的 `PATH` 环境变量中。
|
||||
- 在终端中运行 `ffmpeg -version` 和 `ffprobe -version` 来验证安装。
|
||||
|
||||
### 2. 安装 SoX (VibeVoice 变速不变调必须)
|
||||
|
||||
VibeVoice 节点的 `speed` 参数依赖系统级 `sox` 命令。
|
||||
|
||||
- **Ubuntu/Debian**: `sudo apt-get update && sudo apt-get install -y libsox-dev sox`
|
||||
- **macOS**: `brew install sox`
|
||||
- **Windows**: 下载 [SoX 编译版](https://sourceforge.net/projects/sox/files/sox/) 并将目录添加到 `PATH`。
|
||||
|
||||
### 3. 安装 NeMo 模型 (音频AI节点必须)
|
||||
### 2. 安装 NeMo 模型 (音频AI节点必须)
|
||||
|
||||
音频处理节点（如说话人日志）依赖 NeMo 模型。
|
||||
|
||||
@@ -124,6 +108,43 @@ hf download nvidia/nemo-models diar_sortformer_4spk-v1.nemo --local-dir nemo_mod
|
||||
hf download nvidia/nemo-models diar_streaming_sortformer_4spk-v2.1.nemo --local-dir nemo_models
|
||||
```
|
||||
|
||||
### 3. 安装 FunASR 模型 (ASR 节点 / 播客防泄漏管线)
|
||||
|
||||
ASR 节点使用阿里达摩院的 **FunASR** 框架进行语音识别，支持字级时间戳。模型需手动下载至 `ComfyUI/models/funasr/` 目录。
|
||||
|
||||
```text
|
||||
ComfyUI/models/funasr/
|
||||
├── paraformer-zh/ <-- 中文 ASR (推荐，支持字级时间戳)
|
||||
│ ├── model.pt
|
||||
│ ├── configuration.json
|
||||
│ └── ...
|
||||
└── SenseVoiceSmall/ <-- 多语言 ASR (中/英/日/韩/粤，无时间戳)
|
||||
├── model.pt
|
||||
└── ...
|
||||
```
|
||||
|
||||
**下载命令**:
|
||||
|
||||
```bash
|
||||
cd ComfyUI/models
|
||||
mkdir -p funasr
|
||||
|
||||
# 推荐: Paraformer-zh (中文，支持字级时间戳，~950MB)
|
||||
modelscope download --model iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch --local_dir funasr/paraformer-zh
|
||||
|
||||
# 可选: SenseVoiceSmall (多语言，无时间戳，~450MB)
|
||||
modelscope download --model iic/SenseVoiceSmall --local_dir funasr/SenseVoiceSmall
|
||||
```
|
||||
|
||||
**Python 依赖**:
|
||||
|
||||
```bash
|
||||
pip install funasr
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> **Paraformer-zh** 是播客防泄漏管线（Stitcher 节点）的**必需**模型，因为它提供精确的字级时间戳用于音频切分对齐。SenseVoiceSmall 适合通用多语言识别场景，但不输出时间戳。
|
||||
|
||||
### 4. 安装本节点套件
|
||||
|
||||
进入 ComfyUI 的自定义节点目录，然后克隆本仓库：
|
||||
@@ -151,7 +172,7 @@ git clone https://github.com/havvk/ComfyUI_AIIA.git
|
||||
|
||||
### 2. 视频生成与合成 (Video Generation & Compositing)
|
||||
|
||||
#### 2.1 视频合并 (AIIA, 图像或目录)
|
||||
#### 视频合并 (AIIA, 图像或目录)
|
||||
|
||||
这是一个功能强大且高度可定制的视频合并节点，是您工作流中处理视频生成的终极解决方案。
|
||||
|
||||
@@ -164,7 +185,7 @@ git clone https://github.com/havvk/ComfyUI_AIIA.git
|
||||
- **全面的音频控制**: 支持 `AUDIO` 张量和外部文件，并提供对编解码器和码率的精细控制。
|
||||
- **智能自动配置**: `auto` 模式能自动应用格式预设中的音频参数，并能自动检测源文件的码率。
|
||||
|
||||
#### 2.2 FLOAT 影片生成 (内存与磁盘模式)
|
||||
#### FLOAT 影片生成 (内存与磁盘模式)
|
||||
|
||||
这组节点封装了先进的 **FLOAT** 模型，能够根据参考图像和音频生成高质量的口型同步影片。我们提供了两种模式，以应对不同长度的生成需求。
|
||||
|
||||
@@ -182,7 +203,7 @@ git clone https://github.com/havvk/ComfyUI_AIIA.git
|
||||
- **优势**: 在解码过程中，节点以小批量方式处理帧并**逐帧保存到磁盘**，内存占用极低，可以处理任意长度的音频。
|
||||
- **工作流**: 此节点的输出目录可以直接作为 **视频合并节点** 的 `frames_directory` 输入，构建一个完整的、内存高效的 talking head 视频生成管线。
|
||||
|
||||
#### 2.3 PersonaLive 视频驱动 (AIIA Integrated)
|
||||
#### PersonaLive 视频驱动 (AIIA Integrated)
|
||||
|
||||
这组节点基于强大的 [PersonaLive](https://github.com/GVCLab/PersonaLive) 模型，专为生成高质量的 Talking Head 视频而设计。我们将原版代码完全重构并集成到 ComfyUI 中，通过特有的分块处理和磁盘流式技术，**彻底解决了长视频生成时的显存和内存溢出 (OOM) 问题**。
|
||||
|
||||
@@ -203,262 +224,7 @@ git clone https://github.com/havvk/ComfyUI_AIIA.git
|
||||
- **输出**: `STRING` (包含生成帧的目录路径) 和 `INT` (帧数)。
|
||||
- **最佳实践**: 将此节点的输出目录直接连接到 **AIIA Video Combine** 节点，即可实现从生成到合成的全流程 OOM-Safe。
|
||||
|
||||
#### 2.4 EchoMimic V3 (AIIA Integrated)
|
||||
|
||||
这组节点集成了最新的 **EchoMimic V3** (1.3B Parameters) 模型，它是目前开源界效果最惊艳的 Talking Head 解决方案之一。
|
||||
|
||||
**特点**:
|
||||
|
||||
- **多模态驱动**: 支持 **Audio Only** (仅音频驱动) 和 **Audio + Reference Pose** (音频+参考姿态) 驱动。
|
||||
- **自然度极高**: 相比 float 等早期模型，V3 在头部运动、表情微表情的自然度上有巨大提升。
|
||||
- **ComfyUI 原生**: 我们将其封装为标准的 Loader 和 Sampler 节点，支持流式生成和内存优化。
|
||||
|
||||
**1. EchoMimic V3 Loader**
|
||||
|
||||
- **用途**: 加载模型权重 (Transformer, VAE, Wav2Vec, etc.)。
|
||||
- **参数**:
|
||||
- `model_subfolder`: 模型子目录名 (默认 `Wan2.1-Fun-V1.1-1.3B-InP`)。
|
||||
- `device`: 指定运行设备 (CUDA)。
|
||||
|
||||
**2. EchoMimic V3 Sampler**
|
||||
|
||||
- **用途**: 执行推理生成。
|
||||
- **输入**:
|
||||
- `ref_image`: 参考人物图片 (建议 1:1 比例，如 768x768)。
|
||||
- `ref_audio`: 驱动音频。
|
||||
- **参数**:
|
||||
- `cfg`: 视觉引导系数 (默认 4.0)。
|
||||
- `audio_cfg`: 音频引导系数 (默认 2.9)。
|
||||
- `enable_teacache`: **True** (默认)。开启后生成速度提升 1.5 倍以上，且质量无损。
|
||||
- `keep_model_loaded`: **True** (默认)。即使显存占用增加，也强制将模型保留在 GPU 上，显著减少多段视频生成时的加载时间。
|
||||
- `negative_prompt`: 已内置优化过的 **眼部修复 (Eye Correction)** 提示词，有效防止翻白眼和眼神飘忽。
|
||||
|
||||
**🚀 性能优化 (Performance)**:
|
||||
|
||||
- **Flash Attention 2**: 强烈推荐安装。检测到时会自动启用，大幅提升推理速度。
|
||||
- **TeaCache**: 默认启用。通过缓存部分 Transformer 层计算，大幅加速生成。
|
||||
- **Full GPU Mode**: 默认启用。适合显存充足 (24GB+) 用户，享受极致流畅的生成体验。
|
||||
|
||||
**🛠️ 模型下载指南 (Manual Download Guide)**
|
||||
|
||||
由于 EchoMimic V3 模型较大且组件较多，目前**不支持自动下载**，请按以下步骤手动准备模型。
|
||||
|
||||
目标目录: `ComfyUI/models/EchoMimicV3/`
|
||||
|
||||
**目录结构**:
|
||||
|
||||
```text
|
||||
ComfyUI/models/EchoMimicV3/
|
||||
├── Wan2.1-Fun-V1.1-1.3B-InP/ <-- 主模型目录
|
||||
│ ├── transformer/
|
||||
│ │ ├── config.json
|
||||
│ │ └── diffusion_pytorch_model.safetensors
|
||||
│ ├── vae/
|
||||
│ │ ├── config.json
|
||||
│ │ └── diffusion_pytorch_model.safetensors
|
||||
│ ├── text_encoder/
|
||||
│ ├── tokenizer/
|
||||
│ ├── image_encoder/
|
||||
│ └── scheduler/
|
||||
└── wav2vec2-base-960h/ <-- 音频编码器 (必需)
|
||||
├── config.json
|
||||
├── pytorch_model.bin
|
||||
└── ...
|
||||
```
|
||||
|
||||
**下载地址**:
|
||||
|
||||
1. **主模型 (EchoMimicV3)**:
|
||||
|
||||
- HuggingFace: [BadToBest/EchoMimicV3](https://huggingface.co/BadToBest/EchoMimicV3)
|
||||
- **下载命令 (推荐)**:
|
||||
```bash
|
||||
hf download BadToBest/EchoMimicV3 --local-dir models/EchoMimicV3/EchoMimicV3
|
||||
```
|
||||
- *注意：此模型包含 EchoMimic 特有的 Transformer 权重，是生成嘴型的核心。*
|
||||
2. **底模 (Wan2.1-Fun-V1.1-1.3B-InP)**:
|
||||
|
||||
- HuggingFace: [alibaba-pai/Wan2.1-Fun-V1.1-1.3B-InP](https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-1.3B-InP)
|
||||
- **下载命令**:
|
||||
```bash
|
||||
hf download alibaba-pai/Wan2.1-Fun-V1.1-1.3B-InP --local-dir models/EchoMimicV3/Wan2.1-Fun-V1.1-1.3B-InP
|
||||
```
|
||||
- *注意：作为 fallback 来源，提供 VAE、Text Encoder 和 Image Encoder权重。*
|
||||
3. **音频编码器 (wav2vec2-base-960h)**:
|
||||
|
||||
- HuggingFace: [facebook/wav2vec2-base-960h](https://huggingface.co/facebook/wav2vec2-base-960h)
|
||||
- **下载命令**:
|
||||
```bash
|
||||
hf download facebook/wav2vec2-base-960h --local-dir models/EchoMimicV3/wav2vec2-base-960h
|
||||
```
|
||||
|
||||
**环境依赖**:
|
||||
|
||||
- 请确保安装了 `requirements.txt` 中的依赖，如 `diffusers>=0.30.1`。节点加载时会尝试自动引用，但如果报错缺包，请手动安装。
|
||||
|
||||
#### 2.5 Ditto Talking Head (AIIA Integrated)
|
||||
|
||||
这组节点集成了 [Ditto](https://github.com/antgroup/ditto-talkinghead) 数字人模型。我们采用了 **PyTorch** 原生实现，避免了复杂的 TensorRT 编译过程，让用户能够“开箱即用”地生成高质量的 Talking Head 视频。
|
||||
|
||||
**特点**:
|
||||
|
||||
- **PyTorch Native**: 无需安装 TensorRT，兼容性更好。
|
||||
- **In-Memory Pipeline**: 针对 ComfyUI 优化的内存内处理流程，无需生成中间视频文件。
|
||||
- **自动模型管理**: 支持自动下载模型权重。
|
||||
|
||||
**1. AIIA Ditto Loader**
|
||||
|
||||
- **用途**: 下载并加载 Ditto 模型 (约 1.2GB)。
|
||||
- **参数**:
|
||||
- `model_name`: 模型名称 (默认 `ditto-talkinghead`)。
|
||||
- `device`: 运行设备 (CUDA/CPU)。
|
||||
|
||||
**2. AIIA Ditto Sampler**
|
||||
|
||||
- **用途**: 执行推理生成。
|
||||
- **输入**:
|
||||
- `pipe`: 来自 Loader 的模型管道。
|
||||
- `ref_image`: 参考人物图片 (建议正方形，人脸居中)。
|
||||
- `audio`: 驱动音频。
|
||||
- `fps`: 建议 **25** (Ditto 针对 25FPS 训练)。即使输入其他值，目前内部逻辑也会优先保证 25FPS 的同步率。
|
||||
- **输出**: `IMAGE` (视频帧), `AUDIO`。
|
||||
- **高级参数 (Advanced Parameters)**:
|
||||
- `seed`: **随机种子 (Random Seed)**。
|
||||
- 控制扩散模型的噪声生成。
|
||||
- **关键作用**: 在长语音生成中，当触发“静音重置”时，种子会被重置，从而确保每一句话的生成条件都与第一句话完全一致，彻底消除“长语音嘴型漂移”和“对口型不准”的问题。
|
||||
- `crop_scale`: (默认 2.3) **面部工作区视野 (Face Context Scale)**。
|
||||
- **注意**: 此参数**不会改变输出视频的分辨率**，它决定了模型“看”到了多少人脸周围的内容。
|
||||
- **数值越大 (如 2.5)**: **广角视野**。模型能覆盖更多头发、脖子和背景。
|
||||
- ✅ 优点：适合头部运动幅度大的场景，不容易出框。
|
||||
- ❌ 缺点：在固定的推理画布中，人脸占比变小，生成的五官细节（如牙齿、眼神）可能会变糊。
|
||||
- **数值越小 (如 2.0)**: **特写视野**。模型聚焦于面部核心区域。
|
||||
- ✅ 优点：人脸占比大，五官细节极其清晰锐利。
|
||||
- ❌ 缺点：容易裁掉下巴或额头，头部大幅运动时可能会出现“断头”或伪影。
|
||||
- **推荐值**:
|
||||
- 标准场景: **2.3**
|
||||
- 大动态/全身/半身: **2.5** (牺牲细节换稳定性)
|
||||
- 大头照/证件照: **2.0** (追求极致细节)
|
||||
- `emo`: (默认 Neutral) 表情控制。可选 Angry, Happy, Sad 等。
|
||||
- `drive_eye`: (默认 True) 是否驱动眼睛。关闭后眼睛将保持参考图状态(或微动)，适合原图眼神较好的情况。
|
||||
- `chk_eye_blink`: (已废弃，请使用 `blink_mode`)。
|
||||
- `blink_mode`: (默认 Natural) **眨眼模式控制**。
|
||||
- `Natural`: **拟人化随机眨眼**。
|
||||
- 基础频率：90-150帧/次 (约 3.6s - 6.0s)。
|
||||
- `Slow`: 慢速沉稳眨眼 (120-200帧/次)。
|
||||
- `Fast`: 快速频繁眨眼 (10-40帧/次)。
|
||||
- `None`: **彻底关闭眨眼**。
|
||||
- `blink_amp`: (v1.9.1 New) **眨眼幅度控制**。
|
||||
- **1.0 (默认)**: 标准闭眼幅度。
|
||||
- **< 1.0 (推荐 0.8)**: 适合**男性角色**或眼睛较小的人物，避免“用力挤眼”的感觉。
|
||||
- **> 1.0**: 加深闭眼力度。
|
||||
- `mouth_amp`: (v1.9.1 New) **嘴型幅度控制**。
|
||||
- **1.0 (默认)**: 标准嘴部开合幅度。
|
||||
- **> 1.0 (推荐 1.1-1.2)**: 适合**大声说话**或需要更夸张表情的场景，增强口型辨识度。
|
||||
- **< 1.0**: 适合轻声细语。
|
||||
- `relax_on_silence`: (默认 True) **静音归位 (Relax Face on Silence)**。
|
||||
- 结合下方的 `silence_release` 参数，针对静音片段进行**平滑过渡**（慢速闭合），避免“紧绷抿嘴”。
|
||||
- **智能预测 (Predictive Logic)**: 自动识别短停顿（如逗号）与长静音（如句号）。短停顿不触发闭嘴动画，由模型自由发挥；长静音则触发优雅的慢速闭合。
|
||||
- **防止累积误差 (Drift Correction)**:
|
||||
- **精密网格对齐 (Exact Onset Alignment)**: 将模型的时间步长精确对齐到语音的开始（Onset），而不是固定的处理块。
|
||||
- **状态重置 (State & RNG Reset)**: 在长静音后，强制重置模型状态和**随机种子**，确保每一句话的生成质量一致，彻底消除“长语音嘴型漂移”。
|
||||
- `silence_release`: (v1.9.2 New) **静音闭嘴速度 (Adsr Release Control)**。
|
||||
- **Natural (0.8s) [默认]**:
|
||||
- 自然平衡模式。适合大多数常速对话。
|
||||
- 触发阈值: >0.88s。
|
||||
- **Fast (0.5s)**:
|
||||
- 快速响应模式。适合语速极快、充满激情的演讲。
|
||||
- 触发阈值: >0.56s。
|
||||
- **Deep (1.3s)**:
|
||||
- 深沉模式。适合朗诵、讲故事或情感类内容。超长尾韵，极度平滑。
|
||||
- 触发阈值: >1.4s。
|
||||
- `ref_threshold`: (默认 0.005) **静音检测相对阈值 (Relative Silence Threshold)**。
|
||||
- 现在的阈值是基于**全段音频峰值音量**的比例 (例如 0.005 = 0.5% 的峰值音量)。
|
||||
- 这意味着无论音频整体是大声还是小声，系统都能自动适应，准确捕捉微弱的语音片段。
|
||||
- 只有低于此比例的音量才会被视为静音。
|
||||
- `smo_k_d`: (默认 3) 运动平滑系数。数值越大动作越柔和，可抑制面部抖动。
|
||||
- `hd_rot_p` / `y` / `r`: 头部旋转微调 (Pitch/Yaw/Roll)。
|
||||
- `speech_pitch`: (v1.10.0 New) **说话时俯仰角补偿 (Speech Pitch Offset)**。
|
||||
- 用于修正"说话时头抬得太高"或"需要低头说话"的场景。此偏移量仅在说话期间生效，并随语音强度平滑切入切出。
|
||||
- **正值 (+) = 低头 (Look Down)**。例如 `5.0` 表示说话时微微低头。
|
||||
- **负值 (-) = 抬头 (Look Up)**。
|
||||
- `mouth_smoothing`: (v1.9.5 New) **嘴型惯性平滑 (Mouth Motion Inertia)**。
|
||||
- 防止模型输出的嘴型瞬间开合（如爆破音时），增加物理惯性感。
|
||||
- **`None (Raw)`**: 无平滑，模型原始输出。追求极致对口型，容忍偶尔快速开合。
|
||||
- **`Light`** (0.3): 轻微平滑，**推荐快语速使用**。
|
||||
- **`Normal`** (0.5) [默认]: 适中平滑，常规对话推荐。
|
||||
- **`Heavy`** (0.7): 强力平滑，适合低质量音频或模型输出抖动严重的情况。
|
||||
- `save_to_disk`: (v1.9.24 New) **OOM 安全模式 (OOM-Safe Mode)**。
|
||||
- **`Memory (Default)`**: 传统模式，所有帧保存在内存中。适合短视频（<1000帧）。
|
||||
- **`Disk (OOM-Safe)`**: **长视频推荐**。边生成边保存到磁盘，无 OOM 风险。
|
||||
- 选择 Disk 模式时，`frames_dir` 输出会包含帧保存路径，可直接连接 **AIIA Video Combine** 节点的 `frames_directory` 输入。
|
||||
- ⚠️ Disk 模式下 `images` 输出为占位符，请使用 `frames_dir` 连接后续节点。
|
||||
- **输出**:
|
||||
- `images`: 生成的视频帧序列（Memory 模式）或占位符（Disk 模式）。
|
||||
- `audio`: 透传的音频。
|
||||
- `frames_dir`: (v1.9.24 New) Disk 模式下的帧保存路径。Memory 模式下为空字符串。
|
||||
|
||||
**🛠️ 模型下载指南 (Manual Download Guide)**
|
||||
|
||||
如果自动下载失败，请手动下载模型并放入 `ComfyUI/models/ditto/` 目录。
|
||||
|
||||
**目标目录结构**:
|
||||
|
||||
```text
|
||||
ComfyUI/models/ditto/
|
||||
├── ditto_pytorch/
|
||||
│ ├── audio2motion.pth
|
||||
│ ├── ...
|
||||
└── ditto_cfg/
|
||||
├── v0.4_hubert_cfg_pytorch.pkl
|
||||
├── ...
|
||||
```
|
||||
|
||||
**下载地址**:
|
||||
|
||||
- HuggingFace: [digital-avatar/ditto-talkinghead](https://huggingface.co/digital-avatar/ditto-talkinghead)
|
||||
|
||||
**下载命令**:
|
||||
|
||||
```bash
|
||||
# 进入 models 目录
|
||||
cd ComfyUI/models
|
||||
|
||||
# 下载模型 (直接下载到 ditto 目录，避免多层嵌套)
|
||||
hf download digital-avatar/ditto-talkinghead --local-dir ditto
|
||||
```
|
||||
|
||||
#### 2.6 身体微动后处理 (Body Sway Post-Processing)
|
||||
|
||||
这个轻量级后处理节点可以为 Ditto 等 Talking Head 模型的输出添加**模拟的身体晃动效果**，让人物看起来更加自然、有呼吸感。
|
||||
|
||||
**工作原理**:
|
||||
* 通过**裁切平移 + 轻微旋转**模拟人体站立或坐着时的自然重心漂移和呼吸起伏。
|
||||
* 使用多频正弦波叠加（基于黄金比例）生成平滑、有机的运动轨迹。
|
||||
* **纯裁切**方式，不放大图像，保持原始画质。
|
||||
|
||||
**AIIA Body Sway 节点**
|
||||
|
||||
- **输入**:
|
||||
- `images` (可选): 来自 Ditto 等节点的视频帧张量 (Memory 模式)
|
||||
- `frames_directory` (可选, v1.9.25 New): 帧目录路径 (Disk 模式，连接 Ditto 的 `frames_dir` 输出)
|
||||
- **参数**:
|
||||
- `crop_ratio`: (默认 0.99) 输出尺寸占输入的比例。
|
||||
- 0.99 = 保留 99%，晃动幅度较小 (推荐)
|
||||
- 0.98 = 保留 98%，晃动幅度中等
|
||||
- 支持三位小数 (如 0.995)
|
||||
- `rotation_amplitude`: (默认 0.1) 最大旋转角度 (度)。
|
||||
- `smoothness`: (默认 0.02) Perlin 噪声平滑度。数值越小，运动越缓慢。
|
||||
- `seed`: 控制随机轨迹。
|
||||
- **输出**:
|
||||
- `images`: 应用了微动效果的帧 (Memory 模式) 或占位符 (Disk 模式)。
|
||||
- `output_frames_dir` (v1.9.25 New): Disk 模式下处理后的帧保存路径。
|
||||
|
||||
> [!NOTE]
|
||||
> v1.9.17 改进：使用 **Perlin 噪声** 替代正弦波，运动更有机自然。**已移除垂直方向位移**，减少叠加 Ditto 头部运动时的"晕船"感。
|
||||
|
||||
> [!TIP]
|
||||
> **OOM-Safe 工作流** (v1.9.24+): Ditto (`Disk`) → BodySway (`frames_directory`) → VideoCombine (`frames_directory`)，全流程无 OOM 风险。
|
||||
> **性能无损** (v1.9.28+): 采用并行 I/O 和零压缩策略，**Disk 模式生成速度与 Memory 模式完全一致** (~30fps+)，且极大降低 RAM 占用。强烈推荐长视频生成使用！
|
||||
---
|
||||
|
||||
### 3. 音频智能处理 (Intelligent Audio Processing)
|
||||
|
||||
@@ -839,7 +605,7 @@ hf download digital-avatar/ditto-talkinghead --local-dir ditto
|
||||
#### 1. 🗣️ VibeVoice TTS (Standard)
|
||||
|
||||
- **适用模型**: `VibeVoice-1.5B`, `VibeVoice-7B`
|
||||
- **参考音频 (Reference Audio)**: 可选 (`optional`)。如果不连接，将自动使用内置的高品质女声种子 (Fallback Seed) 进行生成。
|
||||
- **必选参数**: `reference_audio` (参考音频) - **必须连接**。
|
||||
- **功能**: 支持零样本音色克隆 (Zero-shot Cloning)。输入任何音频，它都会模仿该音色。
|
||||
- **不支持**: `voice_preset` (预设)。
|
||||
|
||||
@@ -964,136 +730,18 @@ hf download digital-avatar/ditto-talkinghead --local-dir ditto
|
||||
|
||||
经过深度测试，我们在三个主流模型中整理了以下对比，助您选择最适合的引擎：
|
||||
|
||||
| 维度 | **VoxCPM 1.5** (800M) | **CosyVoice 3.0** (0.5B/1.5B) | **VibeVoice** (1.5B/7B) |
|
||||
| :--------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------- | :-------------------------------------------------------------- |
|
||||
| **音质 (Fidelity)** | **44.1kHz 格式** `<br>`虽然物理格式为 44.1k，但因采用 **Neural Upsampling** (神经升频) 技术，听感上会有**含混 (Muffled)** 或**金属感**，且伴有底噪。 | **优秀** `<br>`听感最自然，但采样率稍低 (22/24kHz)，有时需 AI 增强。 | **良好** `<br>`主要强在语气自然度，纯音质略逊。 |
|
||||
| **推理速度 (Speed)** | **🚀 冠军 (RTF ~0.17)**`<br>`得益于 Tokenizer-free，极其高效。 | **极快** `<br>`流式响应仅 150ms，且支持 TensorRT 加速。 | **一般/较慢** `<br>`7B 版本较重，更适合离线生成。 |
|
||||
| **克隆能力 (Cloning)** | **SOTA** (Zero-Shot)`<br>`只需 3-10秒，对**音色质感**还原极高。 | **SOTA** (稳定性)`<br>`对**说话韵律/口音**的捕捉最准。 | **良好** `<br>`适合克隆特定语气，而非纯粹音色。 |
|
||||
| **多语言/方言** | **中/英** (双语优化) | **👑 霸主** (9种语言 + 18种方言) | **中/英** |
|
||||
| **语音转换 (VC)** (Audio-to-Audio) | ❌**不支持** `<br>`仅支持 TTS (Text-to-Speech)。无法改变已有音频的音色。 | ✅**支持** `<br>`可以将任意音频转换为任意音色 (保留语调/停顿)。 | ❌**不支持** `<br>`纯 TTS 模型。仅支持 Text-to-Speech。 |
|
||||
| **Qwen3-TTS** (1.7B/0.6B) | ✅**支持** `<br>`支持 Presets (内置音色) 和 VoiceDesign (描述)。 | ✅**支持** `<br>`支持 3秒极速 Clone (克隆) 模型。 | ✅**支持** `<br>`支持 10 种语言。 |
|
||||
|
||||
#### 3.13 Qwen3-TTS (New! 🔥)
|
||||
|
||||
- **用途**: 阿里巴巴 Qwen 团队推出的最新旗舰级 TTS 模型，支持 10 种主要语言及多种方言，具备极高的稳定性和表现力。
|
||||
- **核心能力**:
|
||||
- **Base (Clone)**: 核心能力为 **3秒极速音色克隆**。支持 X-Vector 模式提升稳定性。
|
||||
- **CustomVoice (Presets)**: 阿里巴巴官方提供的 **9 种高品质内置音色** (如 Vivian, Zack 等)，支持极强的情感和方言控制。
|
||||
- **VoiceDesign**: 通过自然语言描述（如“活泼的少女音，带点羞涩”）从零设计音色。
|
||||
- **环境要求**:
|
||||
- **qwen-tts**: `pip install qwen-tts` (插件会自动尝试安装)。
|
||||
- **Flash Attention 2**: 强烈推荐以获得最佳推理性能。
|
||||
- **节点**:
|
||||
- `🤖 Qwen3-TTS Loader`: 加载模型。支持 `Base (Clone)`、`CustomVoice (Presets)` 和 `VoiceDesign` 模型。
|
||||
- `🗣️ Qwen3-TTS Synthesis`: 执行合成。支持单模型连接或通过 Router 连接的 Bundle。
|
||||
- `🔌 Qwen3-Model Router (Bundle)`: **[新]** 路由节点。将多个分立的 Qwen 模型捆绑成一个，供对话节点自动调用。
|
||||
- `🎙️ Qwen3-TTS Dialogue (Specialist)`: **[旗帜级]** 专为 Qwen3 设计的对话节点。单输入设计，支持通过 Router 实现混合克隆/捏人。
|
||||
- **模型列表**:
|
||||
- `Qwen/Qwen3-TTS-12Hz-1.7B-Base` (或 0.6B-Base)
|
||||
- `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` (或 0.6B-CustomVoice)
|
||||
- `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign`
|
||||
|
||||
#### 📊 模型功能映射表 (Model Capability Mapping)
|
||||
|
||||
| 模型版本 | **音色克隆 (Clone)** | **情感控制 (Emotion)** | **文字捏人 (Design)** | **方言支持 (Dialect)** |
|
||||
| :--- | :---: | :---: | :---: | :---: |
|
||||
| **Base** (1.7B/0.6B) | **👑 最强** | ❌ 仅限录音自带 | ❌ 不支持 | ⚠️ 仅限录音自带 |
|
||||
| **CustomVoice** (1.7B) | ⚠️ 效果极差 | ✅ 支持 | ⚠️ 指令干扰严重 | ⚠️ 效果一般 |
|
||||
| **VoiceDesign** (1.7B) | ❌ 不支持 | **👑 专家** | **👑 专家** | **👑 完美支持** |
|
||||
| **CustomVoice** (0.6B) | ⚠️ 效果极差 | ✅ 支持 | ✅ 表现优异 | ✅ 表现优异 |
|
||||
|
||||
> [!TIP]
|
||||
> **关于 UI 简化**：
|
||||
> 现在的对话节点只有一个 `qwen_model` 输入槽。
|
||||
> - 如果你只需要一种模型，直接连上即可。
|
||||
> - 如果你想实现“Speaker A 克隆，Speaker B 捏人”的混合效果，请使用 `🔌 Qwen3-Model Router` 节点进行打包连接。
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **结论**：
|
||||
> 1. 做 **3秒音色克隆**：必须连 `Base` 模型。
|
||||
> 2. 说 **方言** 或 **文字定制音色**：优先连 `VoiceDesign`（1.7B）或 `CustomVoice`（0.6B）。
|
||||
> 3. 使用 **Vivian/Zack 内置音色**：连接 `CustomVoice` 模型。
|
||||
|
||||
**🛠️ 手工下载指南 (Manual Download Guide)**:
|
||||
|
||||
如果节点无法自动下载，或您需要在离线环境使用，请手动从 HuggingFace 或 ModelScope 下载模型文件夹，并放入以下目录（文件夹建议保留原名）：
|
||||
|
||||
```text
|
||||
ComfyUI/models/qwen_tts/Qwen/
|
||||
├── Qwen3-TTS-12Hz-1.7B-Base/ <-- 对应 1.7B Base (Clone)
|
||||
├── Qwen3-TTS-12Hz-1.7B-CustomVoice/ <-- 对应 1.7B CustomVoice
|
||||
├── Qwen3-TTS-12Hz-1.7B-VoiceDesign/ <-- 对应 1.7B VoiceDesign
|
||||
├── Qwen3-TTS-12Hz-0.6B-Base/ <-- 对应 0.6B Base (Clone)
|
||||
└── Qwen3-TTS-12Hz-0.6B-CustomVoice/ <-- 对应 0.6B CustomVoice/VoiceDesign
|
||||
```
|
||||
|
||||
**下载命令 (HuggingFace CLI)**:
|
||||
|
||||
```bash
|
||||
mkdir -p models/qwen_tts/Qwen
|
||||
|
||||
# 1.7B 系列
|
||||
hf download Qwen/Qwen3-TTS-12Hz-1.7B-Base --local-dir models/qwen_tts/Qwen/Qwen3-TTS-12Hz-1.7B-Base
|
||||
hf download Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --local-dir models/qwen_tts/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice
|
||||
hf download Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign --local-dir models/qwen_tts/Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign
|
||||
|
||||
# 0.6B 系列
|
||||
hf download Qwen/Qwen3-TTS-12Hz-0.6B-Base --local-dir models/qwen_tts/Qwen/Qwen3-TTS-12Hz-0.6B-Base
|
||||
hf download Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --local-dir models/qwen_tts/Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice
|
||||
```
|
||||
|
||||
**ModelScope 下载 (国内推荐)**:
|
||||
|
||||
```bash
|
||||
# 0.6B 示例
|
||||
pip install modelscope
|
||||
modelscope download --model qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --local_dir models/qwen_tts/Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> 对于 **0.6B** 系列，官方目前将 `CustomVoice` 和 `VoiceDesign`（文字设计）能力集成在同一个模型中。因此在设计模式下，加载 `0.6B-CustomVoice` 即可获得极佳效果。
|
||||
|
||||
#### 🎭 掌握指令控制 (Instruct Control)
|
||||
|
||||
Qwen3-TTS 最强大的特性之一是其**自然语言指令驱动**的能力。与传统的“固定标签”不同，你可以直接在 `instruct` 输入框中用一段描述来控制声音的表现。
|
||||
|
||||
**1. 情感与语气控制 (Emotion & Tone)**
|
||||
虽然官方没有强制的固定标签列表，但以下描述词被证明效果极佳（支持中文或英文）：
|
||||
- **基础情感**: "开心" (Happy), "悲伤" (Sad), "生气" (Angry), "兴奋" (Excited), "温柔" (Gentle), "严肃" (Serious)。
|
||||
- **微表情控制 (New!)**: 在 Specialist 节点中，你可以叠加更细腻的语气，如 "带点羞涩的" (With a hint of shyness), "语气充满诱惑力" (Seductive tone), "语气带着哭腔" (Crying tone), "语气充满笑意" (Cheerful tone) 等。
|
||||
- **提示**: 这些指令可以组合，例如 `生气且激动的。` 或 `Very happy and excited.`
|
||||
|
||||
**2. 语速与节奏 (Prosody)**
|
||||
虽然节点有专门的 `speed` 滑块，但通过 `instruct` 可以实现更自然的节奏控制：
|
||||
- "语速极快" (Very fast speaking rate), "缓慢且深情地" (Slow and soulful), "中间有明显的停顿" (Dramatic pauses)。
|
||||
|
||||
**3. 音色设计 (Voice Design)**
|
||||
在加载 **VoiceDesign** 模型时，指令框即为你的“捏人”引擎：
|
||||
- **特征描述**: "沙哑的男低音" (Raspy deep male voice), "甜美的少女音" (Sweet young girl's voice), "充满磁性的中年女性" (Magnetic middle-aged female)。
|
||||
- **示例**: `A young woman with a clear, bright voice, speaking with great confidence.`
|
||||
|
||||
- **示例**: `A young woman with a clear, bright voice, speaking with great confidence.`
|
||||
- **方言与口音 (Dialect & Accent)**:
|
||||
- 虽然官方称全系列支持，但实测发现不同模型遵循度不同：
|
||||
- **VoiceDesign (1.7B/0.6B-Custom)**: **👑 效果最强**。因为没有固定身份限制，能完美呈现粤语、上海话等方言的韵律。
|
||||
- **CustomVoice (Presets)**: 效果一般。由于 Vivian 等音色有固定的标准语设定，方言指令常会被弱化以维持音色一致性。
|
||||
- **Base (Clone)**: 效果最弱。主要取决于你的参考音频本身是什么口音。
|
||||
|
||||
**4. 使用技巧**:
|
||||
- **句尾符号**: 指令末尾建议加一个句号（如 `开心地。`），这有助于模型更稳定地理解指令边界。
|
||||
- **对话剧本**: 在 `🎙️ Qwen3-TTS Dialogue (Specialist)` 节点中，如果某位 Speaker 处于 `Preset` 或 `Design` 模式，系统会自动将剧本中的情感标签（如 `[开心]`）转换为对应的 `instruct` 指令。
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
#### 💡 用户实测与选型指南 (Model Comparison & Selection)
|
||||
| 维度 | **VoxCPM 1.5** (800M) | **CosyVoice 3.0** (0.5B/1.5B) | **VibeVoice** (1.5B/7B) |
|
||||
| :--------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------- | :------------------------------------------------------------- |
|
||||
| **音质 (Fidelity)** | **44.1kHz 格式**`<br>`虽然物理格式为 44.1k，但因采用 **Neural Upsampling** (神经升频) 技术，听感上会有**含混 (Muffled)** 或**金属感**，且伴有底噪。 | **优秀**`<br>`听感最自然，但采样率稍低 (22/24kHz)，有时需 AI 增强。 | **良好**`<br>`主要强在语气自然度，纯音质略逊。 |
|
||||
| **推理速度 (Speed)** | **🚀 冠军 (RTF ~0.17)**`<br>`得益于 Tokenizer-free，极其高效。 | **极快**`<br>`流式响应仅 150ms，且支持 TensorRT 加速。 | **一般/较慢**`<br>`7B 版本较重，更适合离线生成。 |
|
||||
| **克隆能力 (Cloning)** | **SOTA** (Zero-Shot)`<br>`只需 3-10秒，对**音色质感**还原极高。 | **SOTA** (稳定性)`<br>`对**说话韵律/口音**的捕捉最准。 | **良好**`<br>`适合克隆特定语气，而非纯粹音色。 |
|
||||
| **多语言/方言** | **中/英** (双语优化) | **👑 霸主** (9种语言 + 18种方言) | **中/英** |
|
||||
| **语音转换 (VC)** (Audio-to-Audio) | ❌**不支持**`<br>`仅支持 TTS (Text-to-Speech)。无法改变已有音频的音色。 | ✅**支持**`<br>`可以将任意音频转换为任意音色 (保留语调/停顿)。 | ❌**不支持**`<br>`纯 TTS 模型。仅支持 Text-to-Speech。 |
|
||||
|
||||
**选型建议**:
|
||||
|
||||
- **追求“听起来最像真人” (音质+音色)**: 选 **VoxCPM 1.5**。它的 Tokenizer-free 架构带来了质的飞跃。
|
||||
- **追求“方言/多语言/稳定性”**: 选 **CosyVoice 3.0**。目前依然是生产环境最稳的选择。
|
||||
- **追求“多样化音色设计/最新 Qwen 生态/长语音流畅度”**: 选 **Qwen3-TTS**。其 VoiceDesign 功能能让你用描述语“捏”出从未听过的声音。
|
||||
- **要做“长篇广播剧/播客”**: 选 **VibeVoice**。它的长窗口上下文优势依然不可替代。
|
||||
|
||||
### 4. 播客与对话生成 (Podcast & Dialogue Generation)
|
||||
@@ -1122,71 +770,29 @@ https://github.com/user-attachments/assets/9a5502c5-79e3-4fc8-8a2d-2cbdbdbbc860
|
||||
|
||||
- **TTS Engine**: 后端引擎选择。
|
||||
- **CosyVoice**: 精准控制型。
|
||||
- **Qwen3-TTS**: 万能旗舰型。支持混合模式：通过连接多个 Qwen 模型，可实现在一个对话中同时使用克隆和内置音色。
|
||||
- **Qwen Model Pins (Multi-Routing)**:
|
||||
- `qwen_model`: 默认主模型。
|
||||
- `qwen_base_model` (可选): 连接 `Base` 模型，专门处理有参考音频 (Clone) 的角色。
|
||||
- `qwen_custom_model` (可选): 连接 `Custom` 模型，专门处理使用内置 ID (Presets) 的角色。
|
||||
- `qwen_design_model` (可选): 连接 `VoiceDesign` 模型，专门处理复杂描述的角色。
|
||||
- **VibeVoice**: 自然演绎型。
|
||||
- **Speaker A/B/C**:
|
||||
- **Ref Audio**: 参考音频 (用于 Zero-Shot 克隆)。
|
||||
- **ID**: 内置音色 ID (如 CosyVoice 的 `Chinese Female`)。
|
||||
- **Batch Mode**: 生成模式控制。
|
||||
- `Natural (Hybrid)`: 混合批处理。仅在 `(Pause)` 处断开。语流最自然，但可能发生音色泄漏。
|
||||
- `Strict (Per-Speaker)`: 严格模式。每句话都会强制断开重置。彻底杜绝音色泄漏，但对话流畅度略低。
|
||||
- `Whole (Single Batch)`: 全量模式。无视所有暂停，一次性生成整本剧本。连贯性最强，但无法控制停顿时间。
|
||||
- **Batching Parameters**:
|
||||
- `max_batch_char` (Default 1500): 单次批处理的最大字符上限。增加此值可大幅提升 Qwen3 的对话连贯性和情感一致性。最高支持模型上限 **32,768**。
|
||||
- **Emotion Safeguard (New!)**:
|
||||
- **智能检测**: 系统会自动嗅探加载模型的元数据。如果你使用 CosyVoice SFT/Base 或 VibeVoice 等不支持 `Instruct` 功能的模型，系统将自动跳过 `[Emotion]` 标签插入，防止模型读出方括号。
|
||||
|
||||
#### 4.6 AIIA Qwen Dialogue TTS (Qwen 旗舰对话节点)
|
||||
|
||||
**[v1.11.0 New]** 深度集成 Qwen3-TTS 的多模式特性，支持复杂的混合角色场景。
|
||||
|
||||
- **Parameters**:
|
||||
- `seed`: 随机种子。
|
||||
- `speed`: 语速调节。
|
||||
- `cfg_scale`: 指令遵循强度 (Classifier-Free Guidance)。建议值 1.5 - 7.0。
|
||||
- `emotion`: **[v1.11.1 New]** 选中预设情感（开心、悲伤、幽默、愤怒等系统预置微调）。
|
||||
- `dialect`: **[v1.11.1 New]** 选中预设方言（粤语、上海话、东北话、四川话等）。
|
||||
- `temperature`: 采样温度。
|
||||
- `max_batch_char`: 单次批处理上限（最高 32,768）。
|
||||
- **Speaker A/B/C Configuration**:
|
||||
- **Mode**: 选择 `Clone` (音色克隆)、`Preset` (官方预设) 或 `Design` (文字设计)。
|
||||
- **ID**: 当模式为 Preset 时，输入预设音色名 (如 `Vivian`, `Serena`, `Uncle_Fu`, `Dylan`, `Eric`, `Ryan`, `Aiden`, `Ono_Anna`, `Sohee`)。
|
||||
- **Expression**: (New!) 为当前角色选择专属微表情描述。
|
||||
- **Dialect**: **[v1.11.1 New]** 为当前角色选择方言/口音（支持粤语、上海话、四川话、东北话等）。
|
||||
- **Design Description**: 当模式为 Design 时，输入对音色的详细自然语言描述。
|
||||
- **Ref Audio**: 当模式为 Clone 时，连接参考音频。
|
||||
- **特点**: 相对于通用对话节点，此节点能根据每个人的模式自动路由到最合适的 Qwen 引擎，且支持在 UI 直接输入设计描述。
|
||||
|
||||
#### 4.7 AIIA Subtitle Gen (字幕生成器)
|
||||
#### 4.3 AIIA Subtitle Gen (字幕生成器)
|
||||
|
||||
**[v1.7.0 New]** 无需 STT，直接从生成过程中提取精准时间轴。
|
||||
|
||||
- **Input**:
|
||||
- `segments_info`: 来自 `AIIA Dialogue TTS` 或 `AIIA Generate Segments` 的输出。
|
||||
- `calibration_info` (可选): **[v1.10.2 新增]** 接入 `AIIA Generate Speaker Segments` 的输出。用于将估算的时间轴自动“吸附”到真实的 VAD 语音活动区间，解决 VibeVoice 等批处理引擎的时间轴偏移问题。
|
||||
- `segments_info`: 来自 `AIIA Dialogue TTS` 的输出。
|
||||
- **Output**:
|
||||
- `SRT`: 通用字幕格式。
|
||||
- `ASS`: 高级排版字幕格式 (自动区分角色颜色)。
|
||||
- **原理**:
|
||||
- **CosyVoice**: 使用生成时的精确时长。
|
||||
- **VibeVoice**: 使用**智能插值算法 (Smart Interpolation)**，根据字符长度自动计算长音频段内的单句时间轴。
|
||||
- **Qwen3-TTS**: 基于生成的音频振幅精准断句，支持多角色时间轴导出。
|
||||
|
||||
#### 4.4 AIIA Subtitle to Segments (字幕转分段)
|
||||
|
||||
**[v1.10.3 New]** 将现有的 SRT/ASS 字幕文件转换为 `segments_info` 格式，以便进行时间轴重新校准。
|
||||
|
||||
- **Input**:
|
||||
- `subtitle_text`: SRT 或 ASS 格式的文本内容。
|
||||
- `subtitle_path` (可选): 字幕文件的本地路径（如果提供，将优先读取文件）。
|
||||
- **Output**:
|
||||
- `segments_info`: 标准化的 JSON 字符串，可直接输入到 `AIIA Subtitle Gen`。
|
||||
- **用途**: 结合 `Subtitle Gen` 的 `calibration_info` 输入，可以将**旧的、不准的字幕**自动对齐到**新的、精准的音轨**上。
|
||||
|
||||
#### 4.5 AIIA Subtitle Preview (字幕预览)
|
||||
#### 4.4 AIIA Subtitle Preview (字幕预览)
|
||||
|
||||
**[v1.7.1 New]** 实时校验音画同步效果。
|
||||
|
||||
@@ -1199,31 +805,100 @@ https://github.com/user-attachments/assets/9a5502c5-79e3-4fc8-8a2d-2cbdbdbbc860
|
||||
|
||||
#### 4.5 Interactive Teaching (Web Export) (互动式教学导出)
|
||||
|
||||
> [!TIP]
|
||||
> **音色泄漏问题？** 如果 VibeVoice 在多角色对话中出现音色混串（Speaker Leakage），请使用下方的 **4.6-4.8 防泄漏管线** 代替直接使用 Dialogue TTS 节点。
|
||||
|
||||
**[v1.8.1 New]** 将播客升级为视听同步的互动网页。支持“读写分离”的缓存优化，修改 Visual 标签无需重跑 TTS。
|
||||
|
||||
- **工作流 (Workflow)**:
|
||||
1. `Script Parser` 输出 `tts_data` (连接到 TTS) 和 `full_script` (连接到 Merge)。
|
||||
2. `AIIA Dialogue TTS` 生成音频和 `segments_info`。
|
||||
3. `AIIA Segment Merge` 将 `full_script` 中的 Visual 标签重新贴回到 `segments_info` 时间轴上。
|
||||
4. `AIIA Web Export` 生成最终 HTML。
|
||||
1. `Script Parser` 输出 `tts_data` (连接到 TTS) 和 `full_script` (连接到 Merge)。
|
||||
2. `AIIA Dialogue TTS` 生成音频和 `segments_info`。
|
||||
3. `AIIA Segment Merge` 将 `full_script` 中的 Visual 标签重新贴回到 `segments_info` 时间轴上。
|
||||
4. `AIIA Web Export` 生成最终 HTML。
|
||||
- **Input**:
|
||||
- `audio`: 音频信号。
|
||||
- `segments_info`: 来自 Merge 节点的包含 Visual 信息的 JSON。
|
||||
- `template`: `Split Screen` (适合宽屏) 或 `Presentation` (适合演示)。
|
||||
- `audio`: 音频信号。
|
||||
- `segments_info`: 来自 Merge 节点的包含 Visual 信息的 JSON。
|
||||
- `template`: `Split Screen` (适合宽屏) 或 `Presentation` (适合演示)。
|
||||
- **Visual Tag 语法**:
|
||||
- 在剧本中插入 `(Visual: url)`。
|
||||
- 支持绝对 URL: `(Visual: https://example.com)`
|
||||
- 支持相对路径: `(Visual: ./slides/01.jpg)` (相对于导出 HTML 的位置)
|
||||
- 在剧本中插入 `(Visual: url)`。
|
||||
- 支持绝对 URL: `(Visual: https://example.com)`
|
||||
- 支持相对路径: `(Visual: ./slides/01.jpg)` (相对于导出 HTML 的位置)
|
||||
|
||||
#### 4.6 🎙️ AIIA ASR (通用语音识别)
|
||||
|
||||
**[v1.9.0 New]** 基于 FunASR 的通用语音识别节点，提供**字级时间戳**，是防泄漏管线的核心组件。
|
||||
|
||||
- **Input**:
|
||||
- `audio`: 待识别的音频信号。
|
||||
- `model`: 选择 ASR 模型（自动扫描 `ComfyUI/models/funasr/` 目录）。
|
||||
- `device`: `cuda` 或 `cpu`。
|
||||
- `batch_size_s`: 动态 batch 大小（秒），越大越快但越占显存。
|
||||
- `hotword` (可选): 热词列表，提高特定词汇识别率。
|
||||
- **Output**:
|
||||
- `asr_result` (`ASR_RESULT`): 包含 `text`（完整文本）和 `words`（词级时间戳列表 `[{word, start, end}]`，单位：秒）。
|
||||
- `text` (`STRING`): 识别出的纯文本。
|
||||
- **支持模型**:
|
||||
| 模型 | 语言 | 时间戳 | 大小 |
|
||||
|------|------|--------|------|
|
||||
| **paraformer-zh** | 中文（含中英混合） | ✅ 字级 | ~950MB |
|
||||
| SenseVoiceSmall | 中/英/日/韩/粤 | ❌ | ~450MB |
|
||||
- **特性**:
|
||||
- 🔄 模型缓存：加载一次，后续调用直接复用。
|
||||
- 🎵 自动重采样：非 16kHz 音频自动转换。
|
||||
- 📝 热词增强：提高专业术语识别率（仅 Paraformer）。
|
||||
|
||||
#### 4.7 ✂️ AIIA Podcast Splitter (文本拆分)
|
||||
|
||||
**[v1.9.0 New]** 将多角色对话脚本按说话人拆分为独立文本段，用于分轨 TTS。
|
||||
|
||||
- **Input**:
|
||||
- `dialogue_json`: 来自 `Script Parser` 的 `dialogue_json` 输出。
|
||||
- **Output**:
|
||||
- `speaker_A_text`: Speaker A 的所有台词拼接（换行分隔）。
|
||||
- `speaker_B_text`: Speaker B 的所有台词拼接（换行分隔）。
|
||||
- `split_map`: 原始对话顺序映射 JSON（记录每句话的说话人、索引、文本）。
|
||||
- **原理**: 解析对话 JSON，按出场顺序将前两个说话人分别归为 A/B，保留 `(Pause)` 暂停信息。
|
||||
|
||||
#### 4.8 🧵 AIIA Podcast Stitcher (音频拼接)
|
||||
|
||||
**[v1.9.0 New]** 利用 ASR 时间戳精确切分分轨音频，按原始对话顺序重组为最终播客。**彻底消除 VibeVoice 的音色泄漏问题。**
|
||||
|
||||
- **Input**:
|
||||
- `split_map`: 来自 Splitter 的顺序映射。
|
||||
- `audio_A` / `audio_B`: 分别为两个说话人独立生成的 TTS 音频。
|
||||
- `asr_A` / `asr_B`: 分别对应的 ASR 识别结果（含字级时间戳）。
|
||||
- `gap_duration`: 说话人交替时插入的静音时长（默认 0.3s）。
|
||||
- `padding`: 每个切片前后保留的余量，保护呼吸声和尾音（默认 0.05s）。
|
||||
- **Output**:
|
||||
- `audio`: 最终拼接好的完整播客音频。
|
||||
- `segments_info`: 包含每个语音段时间轴的 JSON（可直接用于字幕生成）。
|
||||
- **核心算法**:
|
||||
1. **文本-ASR 字符对齐**: 将 ASR 词级文本与原始句子逐字匹配，精确定位每句话在音频中的起止时间。
|
||||
2. **边界扩展到中点**: 切割点扩展到相邻句间隙的中点，避免截断尾音。
|
||||
3. **模糊匹配回退**: ASR 识别与原文不完全一致时，使用前缀模糊匹配。
|
||||
4. **等分回退**: ASR 完全失败时，按字符数等比例分配时间。
|
||||
|
||||
#### 🔗 防泄漏管线连线方式 (Anti-Leakage Pipeline)
|
||||
|
||||
```text
|
||||
┌─ speaker_A_text → VibeVoice TTS (A) → audio_A → ASR → asr_A ─┐
|
||||
Script Parser → Splitter ─────┤ ├→ Stitcher → Final Audio
|
||||
├─ speaker_B_text → VibeVoice TTS (B) → audio_B → ASR → asr_B ─┘
|
||||
└─ split_map ──────────────────────────────────────────────────────→
|
||||
```
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **关键原理**: 每个说话人的音频由独立的 TTS 节点生成（各自使用不同的参考音频），从根本上杜绝了音色泄漏。Stitcher 节点再利用 ASR 时间戳精确地将各段重新交错拼接，还原原始对话节奏。
|
||||
|
||||
#### 💡 引擎选型与最佳实践 (Best Practices)
|
||||
|
||||
| 特性 | **CosyVoice** | **VibeVoice** | **Qwen3-TTS** |
|
||||
| :----------------- | :----------------------------------------- | :----------------------------------------------------------------------------------------------- | :------------------------------------------ |
|
||||
| **核心优势** | **精准控制 (Instruction)** | **自然演绎 (Context-Aware)** | **万能旗舰 (Voice Design)** |
|
||||
| **情感控制** | ✅**支持** (使用 `[Happy]` 等标签) | ❌ 不支持显式标签 (依赖上下文) | ✅**支持** (通过 `instruct` 或标签) |
|
||||
| **生成逻辑** | **逐句生成** (严格遵循每句话的指令) | **混合批处理** (Hybrid Batching) | **动态引擎** (支持流式与批处理) |
|
||||
| **最佳场景** | 需要精确指定某句话语气、方言时 | 长篇对话、广播剧、闲聊 | 音色定制、高质量配音、极速克隆 |
|
||||
| **使用建议** | 可以在剧本中详细标注情感。 | **尽量减少 `(Pause)`**！`<br>`让多句对话连在一起，模型能更好地联系上下文产生自然语气。 | 尝试使用其 Voice Design 进行创意捏人。 |
|
||||
| 特性 | **CosyVoice** | **VibeVoice** |
|
||||
| :----------------- | :----------------------------------------- | :----------------------------------------------------------------------------------------------- |
|
||||
| **核心优势** | **精准控制 (Instruction)** | **自然演绎 (Context-Aware)** |
|
||||
| **情感控制** | ✅**支持** (使用 `[Happy]` 等标签) | ❌ 不支持显式标签 (依赖上下文) |
|
||||
| **生成逻辑** | **逐句生成** (严格遵循每句话的指令) | **混合批处理** (Hybrid Batching) |
|
||||
| **最佳场景** | 需要精确指定某句话语气、方言时 | 长篇对话、广播剧、闲聊 |
|
||||
| **使用建议** | 可以在剧本中详细标注情感。 | **尽量减少 `(Pause)`**！`<br>`让多句对话连在一起，模型能更好地联系上下文产生自然语气。 |
|
||||
|
||||
#### 📝 综合测试剧本 (Example Script)
|
||||
|
||||
@@ -1273,38 +948,7 @@ B: 太神奇了！那我们快去生成试试吧！
|
||||
- 可自动调整其中一个图像序列的尺寸以匹配另一个，并保持宽高比。
|
||||
- 可自定义背景填充颜色。
|
||||
- **输出**: `STRING` (包含所有拼接后帧的新目录路径)。
|
||||
|
||||
#### AIIA Image Smart Crop (智能图像裁切)
|
||||
|
||||
- **用途**: 一个功能全面的智能裁切节点，专为解决人脸比例、视频构图等问题设计。
|
||||
- **场景**: 强烈建议在 **Ditto Sampler** 或其他视频生成节点之前使用，以确保输入图像（特别是人脸）处于最佳位置和比例，避免“嘴巴太大”或“五官漂移”等问题。
|
||||
- **参数**:
|
||||
- `crop_basis`: 裁切基准。
|
||||
- `fixed_width` / `fixed_height`: 锁定一条边 (使用 width/height 参数)，另一条边自适应。
|
||||
- `fixed_long_side`: **匹配原图长边**。裁切出的长边长度等于原图长边长度 (忽略 width/height 参数)。
|
||||
- `fixed_short_side`: **匹配原图短边**。裁切出的短边长度等于原图短边长度 (忽略 width/height 参数)。适合“最大化裁切”。
|
||||
- `custom_size`: 强制裁切为指定的 `width` x `height`。
|
||||
- `aspect_ratio`: 裁切比例。
|
||||
- 默认为 `original` (保持原图比例或使用 custom_size 的宽高)。
|
||||
- 可选 `1:1`, `16:9`, `custom` 等。
|
||||
- 选择非 original 时，会根据 `crop_basis` 自动计算另一条边的长度。
|
||||
- `custom_aspect_ratio`: 自定义比例值 (例如 2.35)。仅在 `aspect_ratio` 选 `custom` 时生效。
|
||||
- `position`: 锚点位置 (九宫格)。支持 `center`, `top`, `bottom_left` 等。
|
||||
- `offset_x` / `offset_y`: 相对偏移量。用于在自动定位的基础上进行微调 (范围 -1.0 到 1.0)。
|
||||
- **输出**: `IMAGE` (裁切后的图像)。
|
||||
|
||||
---
|
||||
|
||||
### 6. 调试与实用工具 (Debug & Utilities)
|
||||
|
||||
#### 6.1 文本调试拼接 (Text Debug Splicer)
|
||||
|
||||
- **用途**: 方便地将多段文本拼接为一个字符串，支持自定义标题和分隔符，常用于构建和调试复杂的 Prompt 或记录中间结果。
|
||||
- **功能**:
|
||||
- **多路输入**: 支持最多 3 路文本输入 (`text_1`~`3`) 和自定义标题 (`title`)。
|
||||
- **灵活分隔**: 内置多种常用分隔符 (换行、分割线等)。
|
||||
- **自动归档**: 支持将拼接结果自动保存为 `.txt` 文件，文件名支持**自定义前缀** (save_prefix)，方便回溯。
|
||||
- **输出**: `STRING` (拼接后的文本)。
|
||||
- **输出**: `STRING` (包含所有拼接后帧的新目录路径)。
|
||||
|
||||
---
|
||||
|
||||
@@ -1323,63 +967,14 @@ B: 太神奇了！那我们快去生成试试吧！
|
||||
|
||||
## Changelog
|
||||
|
||||
### [1.11.0] - 2026-02-04
|
||||
### [1.9.0] - 2026-02-15
|
||||
|
||||
- **Qwen3-TTS**: 新增阿里巴巴 **Qwen3-TTS** 全系列支持。
|
||||
- **🤖 Qwen3-TTS Loader**: 支持加载 Base, CustomVoice, VoiceDesign 及其 1.7B/0.6B 版本。
|
||||
- **🗣️ Qwen3-TTS Synthesis**: 实现全功能生成，包括 Zero-shot 克隆、音色设计和内置音色合成。
|
||||
- **Podcast Integration**: **AIIA Dialogue TTS** 节点现在正式集成 Qwen3-TTS 引擎。
|
||||
- 支持多角色混合场景下的 Qwen3 驱动，支持使用脚本标签触发 `instruct`。
|
||||
- **Auto-Dependency**: 首次运行 Qwen3 节点会自动检测并安装 `qwen-tts` 库。
|
||||
|
||||
### [1.10.17] - 2026-02-03
|
||||
|
||||
- **Subtitle**: 引入“说话人 ID 为了映射 (Speaker Mapping)”机制。
|
||||
- 在字幕校准过程中，系统会建立脚本角色与 VAD 检测角色的对应关系。这确保了即使在短句重叠（Spillover）的情况下，字幕也能强制匹配到正确的说话人音频，避免被相邻的音量大/时长长的角色“抢走”。
|
||||
|
||||
### [1.10.16] - 2026-02-02
|
||||
|
||||
- **Subtitle**: 优化了多片段合并逻辑，引入“贪婪说话人占用”原则。
|
||||
- 对于同一说话人的连续音频片段，只要中间没有被其他说话人占用且停顿小于 3s，都会自动合并到当前行字幕中。解决多句/长句被意外截断的问题。
|
||||
|
||||
### [1.10.15] - 2026-02-02
|
||||
|
||||
- **Subtitle**: 修复了字幕时间校准逻辑中的 Bug。
|
||||
- 增加了说话人一致性检查，防止上一句音频片段（VAD Chunk）被错误地共享给下一个不同说话人的句子，从而导致当前句字幕被截断。
|
||||
|
||||
### [1.10.14] - 2026-02-02
|
||||
|
||||
- **Ditto Sampler**: 修复了由于 `comfy.model_management` 接口版本差异导致的 `AttributeError`。
|
||||
|
||||
### [1.10.13] - 2026-02-02
|
||||
|
||||
- **Ditto Sampler**: 修复了采样过程中无法正常响应 ComfyUI 中断/取消信号的问题。
|
||||
- 为所有工作线程增加了超时检测和状态轮询，支持在长任务执行期间即时退出。
|
||||
|
||||
### [1.10.12] - 2026-02-02
|
||||
|
||||
- **Debug & Utilities**: 新增 **Text Debug Splicer** 节点。
|
||||
- 支持多路文本拼接、自定义分隔符和自动文件归档。
|
||||
|
||||
### [1.9.0] - 2026-01-20
|
||||
|
||||
- **Ditto Talking Head**: 新增 Ditto 模型支持 (PyTorch 版)。
|
||||
- **AIIA Ditto Loader**: 支持自动下载与加载。
|
||||
- **AIIA Ditto Sampler**: 支持内存内流式生成。
|
||||
- **EchoMimic V3**: 优化了音频同步逻辑，修复了唇形漂移问题。
|
||||
|
||||
### [1.8.4] - 2026-01-19
|
||||
|
||||
- **VibeVoice Speed Control**: 实现了基于系统 `sox` 命令的**变速不变调**（Time Stretching）。
|
||||
- **稳定性修复**:
|
||||
- 修复了 VibeVoice 在调整速度时由于张量类型不匹配（Half vs Float）导致的崩溃。
|
||||
- 强制所有音频输出为 `float32`，解决了在 `speed=1.0` 且有参考音频时，下游节点（如 Resemble Enhance）报错的问题。
|
||||
- **依赖更新**: 新增 `sox` 依赖。Linux 服务器用户请确保安装系统库：`sudo apt-get install libsox-dev sox`。
|
||||
|
||||
### [1.8.3] - 2026-01-07
|
||||
|
||||
- **VibeVoice TTS (Standard)**: `reference_audio` 变为可选参数。如果不输入，节点会自动加载内置的高品质女声种子，方便快速测试。
|
||||
- **Fix**: 修复 GitHub Actions 发布的子模块错误。
|
||||
- **防音色泄漏管线 (Anti-Leakage Pipeline)**: 新增 3 个节点，彻底解决 VibeVoice 多角色对话中的音色混串问题。
|
||||
- **🎙️ AIIA ASR**: 通用语音识别节点，基于 FunASR Paraformer，提供字级时间戳。
|
||||
- **✂️ AIIA Podcast Splitter**: 按说话人拆分对话脚本，输出分轨文本和顺序映射。
|
||||
- **🧵 AIIA Podcast Stitcher**: 利用 ASR 时间戳精确切分分轨音频，按原始对话顺序重组。
|
||||
- **Speaker Tag 优化**: VibeVoice 的说话人标签从 `Speaker N:` 改为 `[N]:` 格式，减少注意力泄漏。
|
||||
- **FunASR 模型支持**: 支持 Paraformer-zh（中文/字级时间戳）和 SenseVoiceSmall（多语言）。
|
||||
|
||||
### [1.8.1] - 2026-01-05
|
||||
|
||||
|
||||
Executable → Regular
+24
-11
@@ -53,9 +53,13 @@ else:
|
||||
NODE_DISPLAY_NAME_MAPPINGS.update(module_object.NODE_DISPLAY_NAME_MAPPINGS)
|
||||
|
||||
except ImportError as e_import:
|
||||
import traceback
|
||||
print(f"错误: 导入 {module_alias_for_log} ({module_name_relative}) 失败: {e_import}")
|
||||
traceback.print_exc()
|
||||
except Exception as e_generic:
|
||||
import traceback
|
||||
print(f"错误: 在 {module_alias_for_log} ({module_name_relative}) 导入或处理过程中发生错误: {e_generic}")
|
||||
traceback.print_exc()
|
||||
|
||||
|
||||
# 1. 处理 aiia_float_nodes.py
|
||||
@@ -124,21 +128,30 @@ else:
|
||||
# 21. 处理 aiia_web_export_nodes.py (新增网页导出)
|
||||
_load_nodes_from_module(".aiia_web_export_nodes", "aiia_web_export_nodes")
|
||||
|
||||
# 22. 处理 aiia_echomimic_nodes.py (新增 EchoMimic V3)
|
||||
_load_nodes_from_module(".aiia_echomimic_nodes", "aiia_echomimic_nodes")
|
||||
|
||||
# 23. 处理 aiia_ditto_nodes.py (新增 Ditto)
|
||||
_load_nodes_from_module(".aiia_ditto_nodes", "aiia_ditto_nodes")
|
||||
|
||||
# 24. 处理 aiia_image_nodes.py (新增 Smart Crop)
|
||||
_load_nodes_from_module(".aiia_image_nodes", "aiia_image_nodes")
|
||||
|
||||
# 25. 处理 aiia_debug_nodes.py (新增调试节点)
|
||||
# 22. 处理 aiia_debug_nodes.py
|
||||
_load_nodes_from_module(".aiia_debug_nodes", "aiia_debug_nodes")
|
||||
|
||||
# 26. 处理 aiia_qwen_nodes.py (新增 Qwen3-TTS)
|
||||
# 23. 处理 aiia_ditto_nodes.py (Ditto TTS)
|
||||
_load_nodes_from_module(".aiia_ditto_nodes", "aiia_ditto_nodes")
|
||||
|
||||
# 24. 处理 aiia_echomimic_nodes.py (EchoMimic)
|
||||
_load_nodes_from_module(".aiia_echomimic_nodes", "aiia_echomimic_nodes")
|
||||
|
||||
# 25. 处理 aiia_image_nodes.py (图像工具)
|
||||
_load_nodes_from_module(".aiia_image_nodes", "aiia_image_nodes")
|
||||
|
||||
# 26. 处理 aiia_qwen_nodes.py (Qwen 模型)
|
||||
_load_nodes_from_module(".aiia_qwen_nodes", "aiia_qwen_nodes")
|
||||
|
||||
# 27. 处理 aiia_asr_nodes.py (ASR 语音识别)
|
||||
_load_nodes_from_module(".aiia_asr_nodes", "aiia_asr_nodes")
|
||||
|
||||
# 23. 处理 aiia_podcast_splitter.py (播客文本拆分)
|
||||
_load_nodes_from_module(".aiia_podcast_splitter", "aiia_podcast_splitter")
|
||||
|
||||
# 24. 处理 aiia_podcast_stitcher.py (播客音频拼接)
|
||||
_load_nodes_from_module(".aiia_podcast_stitcher", "aiia_podcast_stitcher")
|
||||
|
||||
# 告诉 ComfyUI 这个节点包有一个包含网页资源的 'js' 目录
|
||||
WEB_DIRECTORY = "js"
|
||||
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
import torch
|
||||
import os
|
||||
import json
|
||||
import tempfile
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
import folder_paths
|
||||
|
||||
# --- 模型路径初始化 ---
|
||||
_FUNASR_MODELS_DIR = None
|
||||
_AVAILABLE_MODELS = {}
|
||||
|
||||
try:
|
||||
_models_base = os.path.join(folder_paths.base_path, "models", "funasr")
|
||||
if os.path.isdir(_models_base):
|
||||
_FUNASR_MODELS_DIR = _models_base
|
||||
for entry in os.listdir(_models_base):
|
||||
full_path = os.path.join(_models_base, entry)
|
||||
if os.path.isdir(full_path):
|
||||
_AVAILABLE_MODELS[entry] = full_path
|
||||
print(f"[AIIA ASR] 发现模型: {entry} -> {full_path}")
|
||||
else:
|
||||
print(f"[AIIA ASR] 警告: funasr 模型目录不存在: {_models_base}")
|
||||
except Exception as e:
|
||||
print(f"[AIIA ASR] 模型路径初始化错误: {e}")
|
||||
|
||||
|
||||
class AIIA_ASR:
|
||||
"""通用 ASR 语音识别节点，基于 FunASR，支持字级时间戳输出。"""
|
||||
|
||||
NODE_NAME = "AIIA ASR"
|
||||
_model_cache = {} # 类级别模型缓存: {model_key: model_instance}
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
model_choices = list(_AVAILABLE_MODELS.keys()) if _AVAILABLE_MODELS else ["NO_MODELS_FOUND"]
|
||||
default_model = "paraformer-zh" if "paraformer-zh" in _AVAILABLE_MODELS else model_choices[0]
|
||||
|
||||
return {
|
||||
"required": {
|
||||
"audio": ("AUDIO",),
|
||||
"model": (model_choices, {"default": default_model}),
|
||||
},
|
||||
"optional": {
|
||||
"device": (["cuda", "cpu"], {"default": "cuda"}),
|
||||
"batch_size_s": ("INT", {
|
||||
"default": 300, "min": 1, "max": 3600, "step": 10,
|
||||
"tooltip": "以秒为单位的动态 batch 大小。越大越快但占用更多显存。"
|
||||
}),
|
||||
"hotword": ("STRING", {
|
||||
"default": "",
|
||||
"tooltip": "热词列表，每行一个词。提高这些词的识别准确率。"
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("ASR_RESULT", "STRING",)
|
||||
RETURN_NAMES = ("asr_result", "text",)
|
||||
FUNCTION = "recognize"
|
||||
CATEGORY = "AIIA/Audio"
|
||||
|
||||
def _ensure_model(self, model_name: str, device: str):
|
||||
"""加载或从缓存获取模型实例。"""
|
||||
cache_key = f"{model_name}_{device}"
|
||||
if cache_key in self._model_cache:
|
||||
print(f"[{self.NODE_NAME}] 使用缓存模型: {model_name} on {device}")
|
||||
return self._model_cache[cache_key]
|
||||
|
||||
model_path = _AVAILABLE_MODELS.get(model_name)
|
||||
if not model_path:
|
||||
raise RuntimeError(f"模型 '{model_name}' 未找到。可用模型: {list(_AVAILABLE_MODELS.keys())}")
|
||||
|
||||
from funasr import AutoModel
|
||||
|
||||
# 检测是否为 SenseVoice 系列（需要 trust_remote_code）
|
||||
is_sensevoice = "sensevoice" in model_name.lower()
|
||||
|
||||
print(f"[{self.NODE_NAME}] 加载模型: {model_path} on {device}...")
|
||||
model = AutoModel(
|
||||
model=model_path,
|
||||
device=device,
|
||||
disable_update=True,
|
||||
trust_remote_code=is_sensevoice,
|
||||
)
|
||||
print(f"[{self.NODE_NAME}] 模型加载完成。")
|
||||
|
||||
self._model_cache[cache_key] = model
|
||||
return model
|
||||
|
||||
def _audio_to_numpy(self, audio: dict) -> tuple:
|
||||
"""将 ComfyUI AUDIO 格式转换为 16kHz mono numpy 数组。"""
|
||||
waveform = audio["waveform"] # (batch, channels, samples)
|
||||
sample_rate = audio["sample_rate"]
|
||||
|
||||
# 取第一个 batch
|
||||
if waveform.ndim == 3:
|
||||
wav = waveform[0]
|
||||
else:
|
||||
wav = waveform
|
||||
|
||||
# 转 mono
|
||||
if wav.ndim == 2 and wav.shape[0] > 1:
|
||||
wav = wav.mean(dim=0)
|
||||
elif wav.ndim == 2:
|
||||
wav = wav.squeeze(0)
|
||||
|
||||
wav_np = wav.cpu().numpy().astype(np.float32)
|
||||
|
||||
# 重采样到 16kHz（FunASR 要求）
|
||||
if sample_rate != 16000:
|
||||
try:
|
||||
import torchaudio.functional as F
|
||||
wav_tensor = torch.from_numpy(wav_np).unsqueeze(0)
|
||||
wav_resampled = F.resample(wav_tensor, sample_rate, 16000)
|
||||
wav_np = wav_resampled.squeeze(0).numpy()
|
||||
print(f"[{self.NODE_NAME}] 重采样: {sample_rate}Hz -> 16000Hz")
|
||||
except ImportError:
|
||||
# 如果 torchaudio 不可用，写临时文件让 FunASR 自行处理
|
||||
print(f"[{self.NODE_NAME}] 警告: torchaudio 不可用，尝试直接传入音频")
|
||||
sample_rate = 16000
|
||||
|
||||
return wav_np, sample_rate
|
||||
|
||||
def recognize(self, audio, model, device="cuda", batch_size_s=300, hotword=""):
|
||||
log = f"[{self.NODE_NAME}]"
|
||||
|
||||
if model == "NO_MODELS_FOUND":
|
||||
error_result = {
|
||||
"text": "",
|
||||
"words": [],
|
||||
"error": "未找到 FunASR 模型。请将模型放在 ComfyUI/models/funasr/ 目录下。"
|
||||
}
|
||||
return (error_result, "")
|
||||
|
||||
# 验证音频
|
||||
if audio is None or "waveform" not in audio:
|
||||
error_result = {"text": "", "words": [], "error": "输入音频无效"}
|
||||
return (error_result, "")
|
||||
|
||||
wav_np, sr = self._audio_to_numpy(audio)
|
||||
duration = len(wav_np) / sr
|
||||
print(f"{log} 音频时长: {duration:.2f}s, 采样率: {sr}Hz")
|
||||
|
||||
if duration < 0.1:
|
||||
print(f"{log} 音频太短 ({duration:.3f}s)，跳过识别")
|
||||
return ({"text": "", "words": []}, "")
|
||||
|
||||
# 加载模型
|
||||
asr_model = self._ensure_model(model, device)
|
||||
|
||||
# 构建生成参数
|
||||
generate_kwargs = {
|
||||
"input": wav_np,
|
||||
"batch_size_s": batch_size_s,
|
||||
}
|
||||
|
||||
# 热词支持（仅 Paraformer 支持）
|
||||
if hotword and hotword.strip() and "paraformer" in model.lower():
|
||||
generate_kwargs["hotword"] = hotword.strip()
|
||||
print(f"{log} 使用热词: {hotword.strip()[:50]}...")
|
||||
|
||||
# SenseVoice 特殊参数
|
||||
if "sensevoice" in model.lower():
|
||||
generate_kwargs["language"] = "auto"
|
||||
generate_kwargs["use_itn"] = True
|
||||
|
||||
print(f"{log} 开始识别...")
|
||||
results = asr_model.generate(**generate_kwargs)
|
||||
|
||||
if not results or len(results) == 0:
|
||||
print(f"{log} 识别结果为空")
|
||||
return ({"text": "", "words": []}, "")
|
||||
|
||||
result = results[0]
|
||||
raw_text = result.get("text", "")
|
||||
raw_timestamps = result.get("timestamp", [])
|
||||
|
||||
# 构建 words 列表
|
||||
words = []
|
||||
if raw_timestamps and raw_text:
|
||||
# FunASR paraformer: text 是空格分隔的词, timestamp 是 [[start_ms, end_ms], ...]
|
||||
text_tokens = raw_text.split()
|
||||
if len(text_tokens) == len(raw_timestamps):
|
||||
for token, ts in zip(text_tokens, raw_timestamps):
|
||||
words.append({
|
||||
"word": token,
|
||||
"start": round(ts[0] / 1000.0, 3), # ms -> s
|
||||
"end": round(ts[1] / 1000.0, 3),
|
||||
})
|
||||
else:
|
||||
print(f"{log} 警告: 词数 ({len(text_tokens)}) 与时间戳数 ({len(raw_timestamps)}) 不匹配")
|
||||
# 尽力匹配
|
||||
for i, ts in enumerate(raw_timestamps):
|
||||
token = text_tokens[i] if i < len(text_tokens) else "?"
|
||||
words.append({
|
||||
"word": token,
|
||||
"start": round(ts[0] / 1000.0, 3),
|
||||
"end": round(ts[1] / 1000.0, 3),
|
||||
})
|
||||
|
||||
# 去掉空格，构建完整文本
|
||||
clean_text = raw_text.replace(" ", "") if raw_text else ""
|
||||
|
||||
asr_result = {
|
||||
"text": clean_text,
|
||||
"words": words,
|
||||
}
|
||||
|
||||
print(f"{log} 识别完成: {len(words)} 个词, 文本: {clean_text[:80]}...")
|
||||
return (asr_result, clean_text)
|
||||
|
||||
|
||||
# --- ComfyUI 节点注册 ---
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"AIIA_ASR": AIIA_ASR,
|
||||
}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"AIIA_ASR": "🎙️ AIIA ASR (Word Timestamps)",
|
||||
}
|
||||
Executable → Regular
Executable → Regular
Executable → Regular
-7
@@ -624,13 +624,6 @@ class AIIA_Audio_Enhance:
|
||||
new_splice_info["sample_rate"] = result_sr
|
||||
new_splice_info["scale_factor"] = scale
|
||||
|
||||
# Cleanup: Move global enhancer to CPU
|
||||
try:
|
||||
if _cached_enhancer is not None:
|
||||
_cached_enhancer.to("cpu")
|
||||
torch.cuda.empty_cache()
|
||||
except: pass
|
||||
|
||||
return ({"waveform": processed_wav, "sample_rate": result_sr}, new_splice_info)
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
|
||||
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
-30
@@ -248,35 +248,6 @@ async def get_video_poster(request):
|
||||
traceback.print_exc()
|
||||
return web.Response(status=500, text=str(e))
|
||||
|
||||
async def delete_item(request):
|
||||
data = await request.json()
|
||||
relative_path_str = data.get("path", "")
|
||||
filename = data.get("filename", "")
|
||||
if not filename:
|
||||
return web.Response(status=400, text="Filename is required.")
|
||||
|
||||
try:
|
||||
file_path = get_safe_path(output_dir, os.path.join(relative_path_str, filename))
|
||||
|
||||
if file_path.is_file():
|
||||
os.remove(file_path)
|
||||
# Also cleanup potential cache files
|
||||
cache_thumb = get_safe_path(image_thumb_dir, os.path.join(relative_path_str, f"{file_path.stem}.jpg"))
|
||||
if cache_thumb.exists(): os.remove(cache_thumb)
|
||||
cache_poster = get_safe_path(video_poster_dir, os.path.join(relative_path_str, f"{file_path.stem}.jpg"))
|
||||
if cache_poster.exists(): os.remove(cache_poster)
|
||||
|
||||
return web.json_response({"status": "success", "message": f"File {filename} deleted."})
|
||||
elif file_path.is_dir():
|
||||
shutil.rmtree(file_path)
|
||||
return web.json_response({"status": "success", "message": f"Directory {filename} deleted."})
|
||||
else:
|
||||
return web.Response(status=404, text="Item not found")
|
||||
|
||||
except Exception as e:
|
||||
traceback.print_exc()
|
||||
return web.Response(status=500, text=str(e))
|
||||
|
||||
async def get_batch_metadata(request):
|
||||
data = await request.json()
|
||||
path = data.get("path", "")
|
||||
@@ -365,7 +336,6 @@ server.PromptServer.instance.app.router.add_post('/api/aiia/v1/browser/get_batch
|
||||
server.PromptServer.instance.app.router.add_get('/api/aiia/v1/browser/thumbnail', get_thumbnail)
|
||||
server.PromptServer.instance.app.router.add_get('/api/aiia/v1/browser/poster', get_video_poster)
|
||||
server.PromptServer.instance.app.router.add_get('/api/aiia/v1/browser/get_workflow', get_workflow)
|
||||
server.PromptServer.instance.app.router.add_post('/api/aiia/v1/browser/delete_item', delete_item)
|
||||
|
||||
NODE_CLASS_MAPPINGS = {}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {}
|
||||
Executable → Regular
+8
-8
@@ -543,14 +543,14 @@ class AIIA_CosyVoice_TTS:
|
||||
return {
|
||||
"required": {
|
||||
"model": ("COSYVOICE_MODEL",),
|
||||
"prompt_label_1": ("STRING", {"default": "Step 1: Enter TTS Text here.", "is_label": True}),
|
||||
"tts_text": ("STRING", {"multiline": True, "default": "Hello, this is a test of CosyVoice 3.0."}),
|
||||
"prompt_label_2": ("STRING", {"default": "Step 2: Enter Style Description here.", "is_label": True}),
|
||||
"instruct_text": ("STRING", {"multiline": True, "default": "Slow speed, magnetic tone, full of emotion.", "tooltip": "Text description to control style/emotion."}),
|
||||
"base_gender": (["Female", "Male"], {"default": "Female", "tooltip": "Base gender for description-based synthesis."}),
|
||||
"dialect": (["None (Auto)", "Cantonese", "Northeastern", "Sichuan", "Henan", "Tianjin", "Shanghai", "Shandong", "Hubei", "Hunan", "Shaanxi", "Shanxi", "Gansu", "Ningxia", "Hokkien", "Guizhou", "Yunnan", "Jiangxi"], {"default": "None (Auto)", "tooltip": "Preset dialect instruction."}),
|
||||
"emotion": (["None (Neutral)", "Happy", "Sad", "Angry", "Robotic", "Peppa Pig"], {"default": "None (Neutral)", "tooltip": "Preset emotion instruction."}),
|
||||
"spk_id": ("STRING", {"default": "", "tooltip": "Fixed Speaker ID (e.g. pure_1). Leave empty for Zero-Shot models."}),
|
||||
"提示1_说的内容": ("STRING", {"default": "📖 第一步：在此输入您想让 AI 说的话 (TTS Text)", "is_label": True}),
|
||||
"tts_text": ("STRING", {"multiline": True, "default": "你好，这是 CosyVoice 3.0 的全能模式测试。"}),
|
||||
"提示2_音色描述": ("STRING", {"default": "🎨 第二步：在此输入对表现力/情感的文字描述 (Style Description)", "is_label": True}),
|
||||
"instruct_text": ("STRING", {"multiline": True, "default": "语速非常慢，语气充满磁性，情感饱满。", "tooltip": "文字描述：在 0.5B 中主要控制情感、方言、语速等‘表现风格’，而非从零生成音色身份。"}),
|
||||
"base_gender": (["Female", "Male"], {"default": "Female", "tooltip": "基础性别底色。在“描述生成”模式下，这提供初始的声音身份（性别/音感底色）。"}),
|
||||
"dialect": (["None (Auto)", "广东话 (Cantonese)", "东北话 (Northeastern)", "四川话 (Sichuan)", "河南话 (Henan)", "天津话 (Tianjin)", "上海话 (Shanghai)", "山东话 (Shandong)", "湖北话 (Hubei)", "湖南话 (Hunan)", "陕西话 (Shaanxi)", "山西话 (Shanxi)", "甘肃话 (Gansu)", "宁夏话 (Ningxia)", "闽南话 (Hokkien)", "贵州话 (Guizhou)", "云南话 (Yunnan)", "江西话 (Jiangxi)"], {"default": "None (Auto)", "tooltip": "预设方言指令。会自动添加在自定义描述之前。若与自定义文字描述冲突，模型表现将不可预测。"}),
|
||||
"emotion": (["None (Neutral)", "开心 (Happy)", "伤心 (Sad)", "生气 (Angry)", "机器人的方式 (Robotic)", "小猪佩奇风格 (Peppa Pig)"], {"default": "None (Neutral)", "tooltip": "预设情感指令。会自动添加在自定义描述之前。"}),
|
||||
"spk_id": ("STRING", {"default": "", "tooltip": "固定音色 ID (如 pure_1)。对于 0.5B/V3 等 Zero-Shot 模型，此项通常为空，需配合参考音频使用。"}),
|
||||
"speed": ("FLOAT", {"default": 1.0, "min": 0.5, "max": 2.0, "step": 0.1}),
|
||||
"seed": ("INT", {"default": 42, "min": -1, "max": 2147483647}),
|
||||
},
|
||||
|
||||
@@ -0,0 +1,212 @@
|
||||
import torch
|
||||
import numpy as np
|
||||
import os
|
||||
import random
|
||||
import tempfile
|
||||
import soundfile as sf
|
||||
import warnings
|
||||
import sys
|
||||
import subprocess
|
||||
import folder_paths
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
# Suppress annoying warnings
|
||||
warnings.filterwarnings("ignore", category=FutureWarning)
|
||||
warnings.filterwarnings("ignore", category=UserWarning, module="onnxruntime")
|
||||
os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"
|
||||
os.environ["ONNXRUNTIME_QUIET"] = "1"
|
||||
|
||||
# Lazy-loaded global variable
|
||||
CosyVoice = None
|
||||
|
||||
def _install_cosyvoice_if_needed():
|
||||
global CosyVoice
|
||||
if CosyVoice is not None: return
|
||||
try:
|
||||
from cosyvoice.cli.cosyvoice import CosyVoice as CV
|
||||
CosyVoice = CV
|
||||
return
|
||||
except ImportError: pass
|
||||
|
||||
try:
|
||||
libs_dir = os.path.join(os.path.dirname(__file__), "libs")
|
||||
cosyvoice_dir = os.path.join(libs_dir, "CosyVoice")
|
||||
matcha_dir = os.path.join(cosyvoice_dir, "third_party", "Matcha-TTS")
|
||||
if not os.path.exists(libs_dir): os.makedirs(libs_dir, exist_ok=True)
|
||||
if not os.path.exists(cosyvoice_dir):
|
||||
subprocess.check_call(["git", "clone", "--recursive", "https://github.com/FunAudioLLM/CosyVoice.git", cosyvoice_dir])
|
||||
if cosyvoice_dir not in sys.path: sys.path.insert(0, cosyvoice_dir)
|
||||
if matcha_dir not in sys.path: sys.path.insert(0, matcha_dir)
|
||||
from cosyvoice.cli.cosyvoice import CosyVoice as CV
|
||||
CosyVoice = CV
|
||||
except Exception as e:
|
||||
print(f"[AIIA] Failed to install/import CosyVoice: {e}")
|
||||
|
||||
class AIIA_CosyVoice_ModelLoader:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"model_name": ([
|
||||
"FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
|
||||
"FunAudioLLM/CosyVoice2-0.5B",
|
||||
"CosyVoice-300M",
|
||||
"CosyVoice-300M-SFT",
|
||||
"CosyVoice-300M-Instruct"
|
||||
],),
|
||||
"use_fp16": ("BOOLEAN", {"default": True}),
|
||||
}
|
||||
}
|
||||
RETURN_TYPES = ("COSYVOICE_MODEL",)
|
||||
RETURN_NAMES = ("model",)
|
||||
FUNCTION = "load_model"
|
||||
CATEGORY = "AIIA/Loaders"
|
||||
|
||||
def load_model(self, model_name, use_fp16):
|
||||
_install_cosyvoice_if_needed()
|
||||
if model_name.startswith("FunAudioLLM/"):
|
||||
model_dir = os.path.join(folder_paths.models_dir, "cosyvoice", model_name.split("/")[-1])
|
||||
if not os.path.exists(model_dir):
|
||||
snapshot_download(repo_id=model_name, local_dir=model_dir)
|
||||
else:
|
||||
model_dir = os.path.join(folder_paths.models_dir, "cosyvoice", model_name)
|
||||
|
||||
from cosyvoice.cli.cosyvoice import AutoModel
|
||||
is_v3 = os.path.exists(os.path.join(model_dir, "cosyvoice3.yaml"))
|
||||
is_v2 = os.path.exists(os.path.join(model_dir, "cosyvoice2.yaml")) or (not is_v3 and os.path.exists(os.path.join(model_dir, "flow.pt")))
|
||||
|
||||
print(f"[AIIA] Loading {'V3' if is_v3 else ('V2' if is_v2 else 'V1')} model from {model_dir}")
|
||||
model_instance = AutoModel(model_dir=model_dir, fp16=use_fp16)
|
||||
|
||||
# Identity detection
|
||||
available_spks = []
|
||||
spk2info_path = os.path.join(model_dir, "spk2info.pt")
|
||||
if os.path.exists(spk2info_path):
|
||||
try: available_spks = list(torch.load(spk2info_path, map_location='cpu').keys())
|
||||
except: pass
|
||||
if "instruct" in model_dir.lower() and not is_v2 and not is_v3:
|
||||
available_spks = sorted(list(set(available_spks + ["中文男", "中文女", "英文男", "英文女", "日语男", "粤语女", "韩语女"])))
|
||||
|
||||
return ({"model": model_instance, "model_dir": model_dir, "is_v3": is_v3, "is_v2": is_v2, "available_spks": available_spks},)
|
||||
|
||||
class AIIA_CosyVoice_V1_TTS:
|
||||
"""Specialized node for 300M (V1) models with Surgical Fix for Male voices."""
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"model": ("COSYVOICE_MODEL",),
|
||||
"tts_text": ("STRING", {"multiline": True, "default": "你好，这是V1专号节点的测试。"}),
|
||||
"instruct_text": ("STRING", {"multiline": True, "default": "Theo 'Crimson', is a fiery, passionate rebel leader."}),
|
||||
"spk_id": ("STRING", {"default": "中文男"}),
|
||||
"speed": ("FLOAT", {"default": 1.0, "min": 0.5, "max": 2.0, "step": 0.1}),
|
||||
"seed": ("INT", {"default": 42, "min": -1, "max": 2147483647}),
|
||||
},
|
||||
"optional": {
|
||||
"reference_audio": ("AUDIO",),
|
||||
"prompt_text": ("STRING", {"multiline": True, "default": ""}),
|
||||
}
|
||||
}
|
||||
RETURN_TYPES = ("AUDIO",)
|
||||
FUNCTION = "generate"
|
||||
CATEGORY = "AIIA/Synthesis"
|
||||
|
||||
def generate(self, model, tts_text, instruct_text, spk_id, speed, seed, reference_audio=None, prompt_text=""):
|
||||
cosyvoice_model = model["model"]
|
||||
if seed >= 0:
|
||||
torch.manual_seed(seed)
|
||||
if torch.cuda.is_available(): torch.cuda.manual_seed_all(seed)
|
||||
|
||||
# 1. Surgical Fix Logic for V1
|
||||
# Check if it's actually V1
|
||||
if model.get("is_v2") or model.get("is_v3"):
|
||||
print("[AIIA] Warning: V1 node used with V2/V3 model. Falling back to native wrapper.")
|
||||
output = cosyvoice_model.inference_instruct(tts_text, instruct_text, None, speed=speed)
|
||||
else:
|
||||
# PURE V1 SURGICAL PATH
|
||||
if instruct_text:
|
||||
print(f"[AIIA] V1 Surgical Instruct | Spk: {spk_id}")
|
||||
clean_inst = instruct_text.strip().split("<|")[0].strip() + "<|endofprompt|>"
|
||||
def gen():
|
||||
chunks = cosyvoice_model.frontend.text_normalize(tts_text, split=True)
|
||||
for c in chunks:
|
||||
mi = cosyvoice_model.frontend.frontend_instruct(c, spk_id, clean_inst)
|
||||
if 'llm_embedding' in mi: del mi['llm_embedding']
|
||||
for o in cosyvoice_model.model.tts(**mi, stream=False, speed=speed): yield o
|
||||
output = gen()
|
||||
elif reference_audio is not None and prompt_text:
|
||||
print("[AIIA] V1 Zero-shot")
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".wav") as tmp:
|
||||
wav = reference_audio["waveform"].squeeze().cpu().numpy()
|
||||
if wav.ndim == 2: wav = wav.T
|
||||
sf.write(tmp.name, wav, cosyvoice_model.sample_rate)
|
||||
output = cosyvoice_model.inference_zero_shot(tts_text, prompt_text, tmp.name, speed=speed)
|
||||
os.unlink(tmp.name)
|
||||
else:
|
||||
print(f"[AIIA] V1 SFT | Spk: {spk_id}")
|
||||
output = cosyvoice_model.inference_sft(tts_text, spk_id, speed=speed)
|
||||
|
||||
all_speech = [c['tts_speech'] for c in output]
|
||||
final_wav = torch.cat(all_speech, dim=-1)
|
||||
return ({"waveform": final_wav.unsqueeze(0).cpu(), "sample_rate": cosyvoice_model.sample_rate},)
|
||||
|
||||
class AIIA_CosyVoice_V2V3_TTS:
|
||||
"""Native node for 0.5B (V2/V3) models using official APIs."""
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"model": ("COSYVOICE_MODEL",),
|
||||
"tts_text": ("STRING", {"multiline": True, "default": "你好，这是V2/V3专用节点的测试。"}),
|
||||
"instruct_text": ("STRING", {"multiline": True, "default": ""}),
|
||||
"spk_id": ("STRING", {"default": ""}),
|
||||
"speed": ("FLOAT", {"default": 1.0, "min": 0.5, "max": 2.0, "step": 0.1}),
|
||||
"seed": ("INT", {"default": 42, "min": -1, "max": 2147483647}),
|
||||
},
|
||||
"optional": {
|
||||
"reference_audio": ("AUDIO",),
|
||||
}
|
||||
}
|
||||
RETURN_TYPES = ("AUDIO",)
|
||||
FUNCTION = "generate"
|
||||
CATEGORY = "AIIA/Synthesis"
|
||||
|
||||
def generate(self, model, tts_text, instruct_text, spk_id, speed, seed, reference_audio=None):
|
||||
cosyvoice_model = model["model"]
|
||||
if seed >= 0:
|
||||
torch.manual_seed(seed)
|
||||
if torch.cuda.is_available(): torch.cuda.manual_seed_all(seed)
|
||||
|
||||
ref_path = None
|
||||
if reference_audio:
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".wav") as tmp:
|
||||
ref_path = tmp.name
|
||||
wav = reference_audio["waveform"].squeeze().cpu().numpy()
|
||||
if wav.ndim == 2: wav = wav.T
|
||||
sf.write(ref_path, wav, cosyvoice_model.sample_rate)
|
||||
|
||||
try:
|
||||
if model["is_v3"]:
|
||||
print(f"[AIIA] V3 Native | Spk: {spk_id}")
|
||||
output = cosyvoice_model.inference_instruct2(tts_text, instruct_text, ref_path, zero_shot_spk_id=spk_id, speed=speed)
|
||||
else:
|
||||
print(f"[AIIA] V2 Native | Spk: {spk_id}")
|
||||
output = cosyvoice_model.inference_instruct(tts_text, instruct_text, ref_path, zero_shot_spk_id=spk_id, speed=speed)
|
||||
|
||||
all_speech = [c['tts_speech'] for c in output]
|
||||
final_wav = torch.cat(all_speech, dim=-1)
|
||||
finally:
|
||||
if ref_path and os.path.exists(ref_path): os.unlink(ref_path)
|
||||
|
||||
return ({"waveform": final_wav.unsqueeze(0).cpu(), "sample_rate": cosyvoice_model.sample_rate},)
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"AIIA_CosyVoice_ModelLoader": AIIA_CosyVoice_ModelLoader,
|
||||
"AIIA_CosyVoice_V1_TTS": AIIA_CosyVoice_V1_TTS,
|
||||
"AIIA_CosyVoice_V2V3_TTS": AIIA_CosyVoice_V2V3_TTS
|
||||
}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"AIIA_CosyVoice_ModelLoader": "CosyVoice Model Loader (AIIA)",
|
||||
"AIIA_CosyVoice_V1_TTS": "CosyVoice V1 (300M) TTS",
|
||||
"AIIA_CosyVoice_V2V3_TTS": "CosyVoice V2/V3 (0.5B+) TTS"
|
||||
}
|
||||
Executable → Regular
+5
-30
@@ -181,36 +181,11 @@ class AIIA_E2E_Speaker_Diarization:
|
||||
if not model_path:
|
||||
return (self._assign_speakers_to_chunks(whisper_chunks, [{"start":0, "end":0, "speaker":f"error_model_not_found_{backend_model}"}]),)
|
||||
|
||||
if audio is None:
|
||||
print("错误: [AIIA E2E Diarization] 音频数据为 None")
|
||||
return (self._assign_speakers_to_chunks(whisper_chunks, [{"start":0, "end":0, "speaker":"error_no_audio"}]),)
|
||||
|
||||
# Handle cases where audio might be passed as a single-item list
|
||||
if isinstance(audio, list) and len(audio) > 0:
|
||||
audio = audio[0]
|
||||
|
||||
# Try to treat as a dictionary or object with waveform/sample_rate
|
||||
try:
|
||||
waveform = audio["waveform"]
|
||||
sample_rate = audio["sample_rate"]
|
||||
except (KeyError, TypeError):
|
||||
try:
|
||||
waveform = getattr(audio, "waveform", None)
|
||||
sample_rate = getattr(audio, "sample_rate", None)
|
||||
except:
|
||||
waveform, sample_rate = None, None
|
||||
|
||||
if waveform is None or sample_rate is None:
|
||||
print(f"错误: [AIIA E2E Diarization] 音频数据格式错误: 无法获取 waveform 或 sample_rate (输入类型: {type(audio)})")
|
||||
return (self._assign_speakers_to_chunks(whisper_chunks, [{"start":0, "end":0, "speaker":"error_no_audio"}]),)
|
||||
|
||||
# Ensure waveform is a tensor and sample_rate is a number
|
||||
if not isinstance(waveform, torch.Tensor) or not isinstance(sample_rate, (int, float)):
|
||||
print(f"错误: [AIIA E2E Diarization] 音频数据类型错误: waveform={type(waveform)}, sample_rate={type(sample_rate)}")
|
||||
return (self._assign_speakers_to_chunks(whisper_chunks, [{"start":0, "end":0, "speaker":"error_no_audio"}]),)
|
||||
|
||||
if waveform.ndim < 1:
|
||||
print("错误: [AIIA E2E Diarization] 音频波形维度不足")
|
||||
if audio is None or not isinstance(audio, dict) or \
|
||||
"waveform" not in audio or not isinstance(audio["waveform"], torch.Tensor) or \
|
||||
"sample_rate" not in audio or not isinstance(audio["sample_rate"], int) or \
|
||||
audio["waveform"].ndim < 1:
|
||||
print("错误: [AIIA E2E Diarization] 音频数据缺失、格式不正确或无效。")
|
||||
return (self._assign_speakers_to_chunks(whisper_chunks, [{"start":0, "end":0, "speaker":"error_no_audio"}]),)
|
||||
|
||||
if not isinstance(whisper_chunks, dict) or not isinstance(whisper_chunks.get("chunks"), list) :
|
||||
|
||||
Executable → Regular
+5
-33
@@ -51,14 +51,6 @@ def _patched_decode_for_in_memory_stack(
|
||||
img_t_gpu_raw, _ = self_float_model.motion_autoencoder.dec(s_r_plus_motion, alpha=None, feats=s_r_feats)
|
||||
img_t_gpu_clamped = torch.clamp(img_t_gpu_raw, -1, 1) # 值域 [-1, 1]
|
||||
|
||||
# --- AIIA FIX: Top Edge Cropping ---
|
||||
mask_top_edge = getattr(self_float_model, '_aiia_mask_top_edge', 0)
|
||||
if mask_top_edge > 0:
|
||||
# Crop the top N rows to remove artifacts
|
||||
if img_t_gpu_clamped.shape[-2] > mask_top_edge:
|
||||
img_t_gpu_clamped = img_t_gpu_clamped[..., mask_top_edge:, :]
|
||||
# ----------------------------------
|
||||
|
||||
gpu_frame_buffer.append(img_t_gpu_clamped.squeeze(0) if B == 1 else img_t_gpu_clamped[0])
|
||||
|
||||
if len(gpu_frame_buffer) >= FRAMES_PER_GPU_CHUNK or \
|
||||
@@ -113,14 +105,6 @@ def _patched_decode_and_save_to_disk(
|
||||
s_r_plus_motion = s_r + current_motion_vector
|
||||
img_t_gpu_raw, _ = self_float_model.motion_autoencoder.dec(s_r_plus_motion, alpha=None, feats=s_r_feats)
|
||||
img_t_gpu_clamped = torch.clamp(img_t_gpu_raw, -1, 1)
|
||||
|
||||
# --- AIIA FIX: Top Edge Cropping ---
|
||||
mask_top_edge = getattr(self_float_model, '_aiia_mask_top_edge', 0)
|
||||
if mask_top_edge > 0:
|
||||
if img_t_gpu_clamped.shape[-2] > mask_top_edge:
|
||||
img_t_gpu_clamped = img_t_gpu_clamped[..., mask_top_edge:, :]
|
||||
# ----------------------------------
|
||||
|
||||
gpu_frame_buffer.append(img_t_gpu_clamped.squeeze(0) if B == 1 else img_t_gpu_clamped[0])
|
||||
|
||||
if len(gpu_frame_buffer) >= FRAMES_PER_GPU_CHUNK_FOR_PROCESSING or \
|
||||
@@ -167,7 +151,7 @@ class AIIA_FloatProcess_InMemory:
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {"float_pipe": ("FLOAT_PIPE",),"ref_image": ("IMAGE",),"ref_audio": ("AUDIO",),"a_cfg_scale": ("FLOAT", {"default": 2.0,"min": 0.0, "max": 10.0, "step": 0.1}),"r_cfg_scale": ("FLOAT", {"default": 1.0,"min": 0.0, "max": 10.0, "step": 0.1}),"e_cfg_scale": ("FLOAT", {"default": 1.0,"min": 0.0, "max": 10.0, "step": 0.1}),"fps": ("FLOAT", {"default": 25.0, "min":1.0, "max": 60.0, "step": 0.5}),"emotion": (['none', 'angry', 'disgust', 'fear', 'happy', 'neutral', 'sad', 'surprise'], {"default": "none"}),"crop_input_image": ("BOOLEAN",{"default":False},),"seed": ("INT", {"default": 0, "min": 0, "max": 0xffffffffffffffff}),"nfe": ("INT", {"default": 10, "min": 1, "max": 100, "step": 1}), },"optional": {"device_override": (["default", "cuda", "cpu"], {"default": "default"}), "decode_gpu_chunk_size": ("INT", {"default": 32, "min":1, "max":128, "step":1, "tooltip":"(In-Memory) GPU解码后一次转移多少帧到CPU。影响显存和速度。"}), "mask_top_edge_pixels": ("INT", {"default": 0, "min": 0, "max": 64, "step": 1, "tooltip": "CROPS the top N rows of pixels to remove artifacts. Output height will be smaller."})}}
|
||||
return {"required": {"float_pipe": ("FLOAT_PIPE",),"ref_image": ("IMAGE",),"ref_audio": ("AUDIO",),"a_cfg_scale": ("FLOAT", {"default": 2.0,"min": 0.0, "max": 10.0, "step": 0.1}),"r_cfg_scale": ("FLOAT", {"default": 1.0,"min": 0.0, "max": 10.0, "step": 0.1}),"e_cfg_scale": ("FLOAT", {"default": 1.0,"min": 0.0, "max": 10.0, "step": 0.1}),"fps": ("FLOAT", {"default": 25.0, "min":1.0, "max": 60.0, "step": 0.5}),"emotion": (['none', 'angry', 'disgust', 'fear', 'happy', 'neutral', 'sad', 'surprise'], {"default": "none"}),"crop_input_image": ("BOOLEAN",{"default":False},),"seed": ("INT", {"default": 0, "min": 0, "max": 0xffffffffffffffff}),"nfe": ("INT", {"default": 10, "min": 1, "max": 100, "step": 1}), },"optional": {"device_override": (["default", "cuda", "cpu"], {"default": "default"}), "decode_gpu_chunk_size": ("INT", {"default": 32, "min":1, "max":128, "step":1, "tooltip":"(In-Memory) GPU解码后一次转移多少帧到CPU。影响显存和速度。"}),}}
|
||||
|
||||
def _create_error_image(self, error_message_text: str, log_message: bool = True) -> tuple:
|
||||
if log_message:
|
||||
@@ -178,8 +162,7 @@ class AIIA_FloatProcess_InMemory:
|
||||
a_cfg_scale, r_cfg_scale, e_cfg_scale,
|
||||
fps, emotion, crop_input_image, seed, nfe,
|
||||
device_override: str = "default",
|
||||
decode_gpu_chunk_size: int = 32,
|
||||
mask_top_edge_pixels: int = 0):
|
||||
decode_gpu_chunk_size: int = 32):
|
||||
node_name_log = f"[{self.__class__.NODE_NAME}]"
|
||||
print(f"{node_name_log} 流程开始 (内存输出模式)。")
|
||||
start_time_process = time.time()
|
||||
@@ -225,8 +208,7 @@ class AIIA_FloatProcess_InMemory:
|
||||
float_pipe.opt.rank = processing_device.index if processing_device.type == 'cuda' and processing_device.index is not None else (0 if processing_device.type == 'cuda' else -1)
|
||||
if hasattr(float_pipe.opt, 'fps'): float_pipe.opt.fps = float(fps)
|
||||
float_pipe.opt.decode_gpu_chunk_size = decode_gpu_chunk_size
|
||||
float_pipe.G._aiia_mask_top_edge = mask_top_edge_pixels # Inject param for patch
|
||||
print(f"{node_name_log} opt 更新: rank={getattr(float_pipe.opt, 'rank', 'N/A')}, fps={getattr(float_pipe.opt, 'fps', 'N/A')}, decode_chunk={getattr(float_pipe.opt, 'decode_gpu_chunk_size', 'N/A')}, mask_top={mask_top_edge_pixels}")
|
||||
print(f"{node_name_log} opt 更新: rank={getattr(float_pipe.opt, 'rank', 'N/A')}, fps={getattr(float_pipe.opt, 'fps', 'N/A')}, decode_chunk={getattr(float_pipe.opt, 'decode_gpu_chunk_size', 'N/A')}")
|
||||
|
||||
model_current_device_before_move = next(float_pipe.G.parameters()).device
|
||||
if model_current_device_before_move != processing_device: float_pipe.G.to(processing_device)
|
||||
@@ -269,10 +251,6 @@ class AIIA_FloatProcess_InMemory:
|
||||
del float_pipe.opt.decode_gpu_chunk_size
|
||||
except AttributeError:
|
||||
pass
|
||||
|
||||
if hasattr(float_pipe.G, '_aiia_mask_top_edge'):
|
||||
try: del float_pipe.G._aiia_mask_top_edge
|
||||
except: pass
|
||||
|
||||
current_g_device_after_proc = next(float_pipe.G.parameters()).device
|
||||
if current_g_device_after_proc.type == 'cuda':
|
||||
@@ -315,8 +293,7 @@ class AIIA_FloatProcess_ToDisk:
|
||||
fps, emotion, crop_input_image, seed, nfe,
|
||||
device_override: str = "default",
|
||||
output_subdir_name: str = "float_frames_AIIA",
|
||||
decode_gpu_chunk_size: int = 16,
|
||||
mask_top_edge_pixels: int = 0):
|
||||
decode_gpu_chunk_size: int = 16):
|
||||
|
||||
node_name_log = f"[{self.__class__.NODE_NAME}]"
|
||||
print(f"{node_name_log} 流程开始 (输出到磁盘模式)。")
|
||||
@@ -373,8 +350,7 @@ class AIIA_FloatProcess_ToDisk:
|
||||
float_pipe.opt.rank = processing_device.index if processing_device.type == 'cuda' and processing_device.index is not None else (0 if processing_device.type == 'cuda' else -1)
|
||||
if hasattr(float_pipe.opt, 'fps'): float_pipe.opt.fps = float(fps)
|
||||
float_pipe.opt.frames_per_gpu_chunk_for_processing = decode_gpu_chunk_size
|
||||
float_pipe.G._aiia_mask_top_edge = mask_top_edge_pixels # Inject param
|
||||
print(f"{node_name_log} opt 更新: rank={getattr(float_pipe.opt, 'rank', 'N/A')}, fps={getattr(float_pipe.opt, 'fps', 'N/A')}, frames_chunk_for_processing={getattr(float_pipe.opt, 'frames_per_gpu_chunk_for_processing', 'N/A')}, mask_top={mask_top_edge_pixels}")
|
||||
print(f"{node_name_log} opt 更新: rank={getattr(float_pipe.opt, 'rank', 'N/A')}, fps={getattr(float_pipe.opt, 'fps', 'N/A')}, frames_chunk_for_processing={getattr(float_pipe.opt, 'frames_per_gpu_chunk_for_processing', 'N/A')}")
|
||||
|
||||
model_current_device_before_move = next(float_pipe.G.parameters()).device
|
||||
if model_current_device_before_move != processing_device: float_pipe.G.to(processing_device)
|
||||
@@ -439,10 +415,6 @@ class AIIA_FloatProcess_ToDisk:
|
||||
del float_pipe.opt.frames_per_gpu_chunk_for_processing
|
||||
except AttributeError:
|
||||
pass
|
||||
|
||||
if hasattr(float_pipe.G, '_aiia_mask_top_edge'):
|
||||
try: del float_pipe.G._aiia_mask_top_edge
|
||||
except: pass
|
||||
|
||||
current_g_device_after_proc = next(float_pipe.G.parameters()).device
|
||||
if current_g_device_after_proc.type == 'cuda':
|
||||
|
||||
Executable → Regular
+5
-40
@@ -165,33 +165,11 @@ class AIIA_GenerateSpeakerSegments:
|
||||
if not model_path:
|
||||
return self._create_error_output(f"模型 '{e2e_backend_model}' 文件路径无效")
|
||||
|
||||
if audio is None:
|
||||
return self._create_error_output("音频数据为 None")
|
||||
|
||||
# Handle cases where audio might be passed as a single-item list
|
||||
if isinstance(audio, list) and len(audio) > 0:
|
||||
audio = audio[0]
|
||||
|
||||
# Try to treat as a dictionary or object with waveform/sample_rate
|
||||
try:
|
||||
waveform = audio["waveform"]
|
||||
sample_rate = audio["sample_rate"]
|
||||
except (KeyError, TypeError):
|
||||
try:
|
||||
waveform = getattr(audio, "waveform", None)
|
||||
sample_rate = getattr(audio, "sample_rate", None)
|
||||
except:
|
||||
waveform, sample_rate = None, None
|
||||
|
||||
if waveform is None or sample_rate is None:
|
||||
return self._create_error_output(f"音频数据格式错误: 无法获取 waveform 或 sample_rate (输入类型: {type(audio)})")
|
||||
|
||||
# Ensure waveform is a tensor and sample_rate is a number
|
||||
if not isinstance(waveform, torch.Tensor) or not isinstance(sample_rate, (int, float)):
|
||||
return self._create_error_output(f"音频数据类型错误: waveform={type(waveform)}, sample_rate={type(sample_rate)}")
|
||||
|
||||
if waveform.ndim < 1:
|
||||
return self._create_error_output("音频波形维度不足")
|
||||
if audio is None or not isinstance(audio, dict) or \
|
||||
"waveform" not in audio or not isinstance(audio["waveform"], torch.Tensor) or \
|
||||
"sample_rate" not in audio or not isinstance(audio["sample_rate"], int) or \
|
||||
audio["waveform"].ndim < 1:
|
||||
return self._create_error_output("音频数据缺失或无效")
|
||||
|
||||
# 检查音频长度
|
||||
if audio["waveform"].shape[-1] == 0:
|
||||
@@ -333,19 +311,6 @@ class AIIA_GenerateSpeakerSegments:
|
||||
print(f"警告: [{self.NODE_NAME}] 最终未能获取任何说话人分段。")
|
||||
|
||||
output_data_structure = {"text": "", "chunks": speaker_segments_for_json_chunks, "language": ""}
|
||||
|
||||
# Cleanup: Move model to CPU and delete
|
||||
try:
|
||||
if 'diar_model' in locals() and diar_model is not None:
|
||||
print(f"{node_name_log} Cleaning up NeMo model (Moving to CPU)...")
|
||||
diar_model.to("cpu")
|
||||
if hasattr(diar_model, 'encoder'): diar_model.encoder.to("cpu")
|
||||
if hasattr(diar_model, 'decoder'): diar_model.decoder.to("cpu")
|
||||
del diar_model
|
||||
torch.cuda.empty_cache()
|
||||
except Exception as cleanup_err:
|
||||
print(f"Warning: Cleanup failed: {cleanup_err}")
|
||||
|
||||
print(f"{node_name_log} 流程结束。")
|
||||
return (output_data_structure,)
|
||||
|
||||
|
||||
Executable → Regular
Executable → Regular
+35
-279
@@ -2,50 +2,6 @@
|
||||
import json
|
||||
import re
|
||||
|
||||
AIIA_EMOTION_LIST = [
|
||||
"None",
|
||||
"Happy (开心)", "Sad (悲伤)", "Angry (愤怒)", "Excited (兴奋)",
|
||||
"Gentle (温柔)", "Fearful (恐惧)", "Surprised (惊讶)", "Disappointed (失望)",
|
||||
"Proud (骄傲)", "Anxious (焦虑)", "Calm (冷静)", "Neutral (中性)",
|
||||
"Affectionate (深情)", "Awkward (尴尬)", "Determined (坚定)", "Hesitant (犹豫)",
|
||||
"With a hint of shyness (带点羞涩)",
|
||||
"With a hint of a smile (带有一丝笑意)",
|
||||
"Seductive tone (充满诱惑力)",
|
||||
"Crying tone (带着哭腔)",
|
||||
"Cheerful tone (充满笑意)",
|
||||
"Serious tone (语气严肃)",
|
||||
"Sarcastic tone (冷嘲热讽)",
|
||||
"Arrogant tone (语气傲慢)",
|
||||
"Cold tone (语气冷淡)",
|
||||
"Affectionate tone (充满爱意)",
|
||||
"Whispering (轻声耳语)",
|
||||
"Shouting (大声叫喊)",
|
||||
"Rapid fire (语速较快)",
|
||||
"Slow and deliberate (语速较慢)",
|
||||
"Tired (疲惫不堪)",
|
||||
"Sleepy tone (睡意朦胧)",
|
||||
"Drunken tone (醉意微醺)",
|
||||
"Professional tone (专业播音)",
|
||||
"Magnetic tone (磁性嗓音)",
|
||||
"Breathless (气喘吁吁)",
|
||||
"Terrified (惊恐万分)",
|
||||
"Nervous (紧张不安)",
|
||||
"Mysterious (语气神秘)",
|
||||
"Enthusiastic (热情高涨)",
|
||||
"Lazy tone (语气慵懒)",
|
||||
"Gossip tone (八卦语气)",
|
||||
"Innocent (语气天真)"
|
||||
]
|
||||
|
||||
AIIA_DIALECT_LIST = [
|
||||
"None",
|
||||
"Mandarin (普通话)", "Cantonese (粤语)", "Shanghainese (上海话)",
|
||||
"Sichuanese (四川话)", "Northeastern (东北话)", "Hokkien (闽南话)",
|
||||
"Hakka (客家话)", "Tianjinese (天津话)", "Shandongnese (山东话)",
|
||||
"Henan (河南话)", "Shaanxi (陕西话)", "Hunan (湖南话)", "Jiangxi (江西话)",
|
||||
"Hubei (湖北话)", "Guizhou (贵州话)", "Yunnan (云南话)", "Gansu (甘肃话)", "Ningxia (宁夏话)"
|
||||
]
|
||||
|
||||
class AIIA_Podcast_Script_Parser:
|
||||
def __init__(self):
|
||||
pass
|
||||
@@ -219,40 +175,33 @@ class AIIA_Dialogue_TTS:
|
||||
def INPUT_TYPES(s):
|
||||
return {
|
||||
"required": {
|
||||
"dialogue_json": ("STRING", {"multiline": True}),
|
||||
"tts_engine": (["CosyVoice", "VibeVoice", "Qwen3-TTS"], {"default": "CosyVoice"}),
|
||||
"dialogue_json": ("STRING", {"forceInput": True}),
|
||||
"tts_engine": (["CosyVoice", "VibeVoice"], {"default": "CosyVoice"}),
|
||||
"pause_duration": ("FLOAT", {"default": 0.5, "min": 0.0, "max": 5.0, "step": 0.1}),
|
||||
"speed_global": ("FLOAT", {"default": 1.0, "min": 0.5, "max": 2.0}),
|
||||
"batch_mode": (["Natural (Hybrid)", "Strict (Per-Speaker)", "Whole (Single Batch)"], {"default": "Natural (Hybrid)"}),
|
||||
},
|
||||
"optional": {
|
||||
# Speaker A
|
||||
"speaker_A_ref": ("AUDIO",),
|
||||
"speaker_A_id": ("STRING", {"default": "", "placeholder": "CosyVoice Internal ID (Optional)"}),
|
||||
"speaker_A_emotion": (AIIA_EMOTION_LIST, {"default": "None"}),
|
||||
"speaker_A_dialect": (AIIA_DIALECT_LIST, {"default": "None"}),
|
||||
|
||||
# Speaker B
|
||||
"speaker_B_ref": ("AUDIO",),
|
||||
"speaker_B_id": ("STRING", {"default": "", "placeholder": "CosyVoice Internal ID (Optional)"}),
|
||||
"speaker_B_emotion": (AIIA_EMOTION_LIST, {"default": "None"}),
|
||||
"speaker_B_dialect": (AIIA_DIALECT_LIST, {"default": "None"}),
|
||||
|
||||
# Speaker C
|
||||
"speaker_C_ref": ("AUDIO",),
|
||||
"speaker_C_id": ("STRING", {"default": "", "placeholder": "CosyVoice Internal ID (Optional)"}),
|
||||
"speaker_C_emotion": (AIIA_EMOTION_LIST, {"default": "None"}),
|
||||
"speaker_C_dialect": (AIIA_DIALECT_LIST, {"default": "None"}),
|
||||
|
||||
# Model Slots and Params (Appended to prevent shift)
|
||||
"cosyvoice_model": ("COSYVOICE_MODEL",),
|
||||
"vibevoice_model": ("VIBEVOICE_MODEL",),
|
||||
"qwen_model": ("QWEN_MODEL",),
|
||||
"max_batch_char": ("INT", {"default": 1000, "min": 100, "max": 32768}),
|
||||
# VibeVoice Specific Params
|
||||
"cfg_scale": ("FLOAT", {"default": 1.5, "min": 1.0, "max": 10.0, "step": 0.1}),
|
||||
"temperature": ("FLOAT", {"default": 0.8, "min": 0.1, "max": 2.0}),
|
||||
"top_k": ("INT", {"default": 20, "min": 0, "max": 100}),
|
||||
"top_p": ("FLOAT", {"default": 0.95, "min": 0.0, "max": 1.0, "step": 0.05}),
|
||||
"top_p": ("FLOAT", {"default": 0.95, "min": 0.0, "max": 1.0}),
|
||||
},
|
||||
"optional": {
|
||||
"cosyvoice_model": ("COSYVOICE_MODEL",),
|
||||
"vibevoice_model": ("VIBEVOICE_MODEL",),
|
||||
|
||||
# Speaker A
|
||||
"speaker_A_ref": ("AUDIO",),
|
||||
"speaker_A_id": ("STRING", {"default": "", "placeholder": "CosyVoice 内部音色ID (可选)"}),
|
||||
|
||||
# Speaker B
|
||||
"speaker_B_ref": ("AUDIO",),
|
||||
"speaker_B_id": ("STRING", {"default": "", "placeholder": "CosyVoice 内部音色ID (可选)"}),
|
||||
|
||||
# Speaker C
|
||||
"speaker_C_ref": ("AUDIO",),
|
||||
"speaker_C_id": ("STRING", {"default": "", "placeholder": "CosyVoice 内部音色ID (可选)"}),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -298,103 +247,9 @@ class AIIA_Dialogue_TTS:
|
||||
print(f"[AIIA Error] Failed to load fallback audio: {e}")
|
||||
return None
|
||||
|
||||
def _generate_qwen_batch(self, batch_data, qwen_gen, current_full_wav, sr_ptr, segments_info, time_ptr, speed_global, cfg_scale, temperature, top_k, top_p):
|
||||
# This helper processes a batch of Qwen items that are compatible (same routed model, etc.)
|
||||
# Qwen's `generate` method takes a single text, so we iterate through the batch.
|
||||
for i, item_params in enumerate(batch_data):
|
||||
target_model = item_params["tm"]
|
||||
text = item_params["tx"]
|
||||
spk_id = item_params["sid"]
|
||||
ref_audio = item_params["ref"]
|
||||
instruct = item_params["ins"]
|
||||
spk_name = item_params["original_speaker"] # Added this to item_params in get_qwen_params
|
||||
original_item = item_params["original_item"] # Added this to item_params in get_qwen_params
|
||||
|
||||
print(f" [Qwen Batch] {spk_name}: {text[:30]}... ({target_model['type']})")
|
||||
if instruct:
|
||||
print(f" [Qwen Instruct] {instruct}")
|
||||
|
||||
try:
|
||||
# Call Qwen TTS with routed model
|
||||
res = qwen_gen.generate(
|
||||
qwen_model=target_model,
|
||||
text=text,
|
||||
language="Auto",
|
||||
speaker=spk_id,
|
||||
instruct=instruct,
|
||||
reference_audio=ref_audio,
|
||||
dialect=item_params.get("dialect", "None"),
|
||||
seed=42+i, # Use a seed for reproducibility within the batch
|
||||
speed=speed_global,
|
||||
cfg_scale=cfg_scale,
|
||||
temperature=temperature,
|
||||
top_k=top_k,
|
||||
top_p=top_p
|
||||
)
|
||||
|
||||
generated = res[0]
|
||||
wav = generated["waveform"]
|
||||
sr = generated["sample_rate"]
|
||||
|
||||
if sr_ptr[0] != sr:
|
||||
if current_full_wav:
|
||||
wav = torchaudio.transforms.Resample(sr, sr_ptr[0])(wav)
|
||||
else:
|
||||
sr_ptr[0] = sr
|
||||
|
||||
if wav.ndim == 3: wav = wav.squeeze(0)
|
||||
if wav.ndim == 1: wav = wav.unsqueeze(0)
|
||||
|
||||
# AIIA Fix: Apply tiny fade-in/out to prevent clicks at boundaries
|
||||
fade_len = int(sr * 0.05) # 50ms fade
|
||||
if wav.shape[-1] > fade_len * 2:
|
||||
fade_in = torch.linspace(0, 1, fade_len, device=wav.device)
|
||||
fade_out = torch.linspace(1, 0, fade_len, device=wav.device)
|
||||
wav[..., :fade_len] *= fade_in
|
||||
wav[..., -fade_len:] *= fade_out
|
||||
|
||||
current_full_wav.append(wav)
|
||||
|
||||
# --- Timestamp Tracking ---
|
||||
seg_duration = wav.shape[-1] / sr
|
||||
seg_start = time_ptr[0]
|
||||
seg_end = seg_start + seg_duration
|
||||
|
||||
segments_info.append({
|
||||
"start": round(seg_start, 3),
|
||||
"end": round(seg_end, 3),
|
||||
"text": text,
|
||||
"speaker": spk_name,
|
||||
"visual": original_item.get("visual")
|
||||
})
|
||||
time_ptr[0] += seg_duration
|
||||
|
||||
# Add a small gap between segments within a Qwen batch
|
||||
gap = 0.2
|
||||
gap_samples = int(gap * sr_ptr[0])
|
||||
current_full_wav.append(torch.zeros(1, gap_samples))
|
||||
time_ptr[0] += gap
|
||||
|
||||
except Exception as e:
|
||||
print(f"[Error] Qwen item generation failed: {e}")
|
||||
current_full_wav.append(torch.zeros(1, 24000))
|
||||
time_ptr[0] += 1.0
|
||||
|
||||
|
||||
def process_dialogue(self, dialogue_json, tts_engine, pause_duration, speed_global, batch_mode, **kwargs):
|
||||
# Extract optional and model-specific params from kwargs
|
||||
max_batch_char = kwargs.get("max_batch_char", 1000)
|
||||
cfg_scale = kwargs.get("cfg_scale", 1.5)
|
||||
temperature = kwargs.get("temperature", 0.8)
|
||||
top_k = kwargs.get("top_k", 20)
|
||||
top_p = kwargs.get("top_p", 0.95)
|
||||
|
||||
cosyvoice_model = kwargs.get("cosyvoice_model")
|
||||
vibevoice_model = kwargs.get("vibevoice_model")
|
||||
qwen_model = kwargs.get("qwen_model")
|
||||
|
||||
# Robustness: ensure max_batch_char is correctly picked up even if shifted or provided as kwarg
|
||||
max_batch_char = kwargs.get("max_batch_char", max_batch_char)
|
||||
def process_dialogue(self, dialogue_json, tts_engine, pause_duration, speed_global,
|
||||
cosyvoice_model=None, vibevoice_model=None,
|
||||
cfg_scale=1.5, temperature=0.8, top_k=20, top_p=0.95, **kwargs):
|
||||
import json
|
||||
import torch
|
||||
import os
|
||||
@@ -405,9 +260,6 @@ class AIIA_Dialogue_TTS:
|
||||
raise ValueError("选择 CosyVoice 引擎时，必须连接 'cosyvoice_model'！")
|
||||
if tts_engine == "VibeVoice" and vibevoice_model is None:
|
||||
raise ValueError("选择 VibeVoice 引擎时，必须连接 'vibevoice_model'！")
|
||||
if tts_engine == "Qwen3-TTS":
|
||||
if qwen_model is None:
|
||||
raise ValueError("选择 Qwen3-TTS 引擎时，必须连接 'qwen_model'！(如果需要多个模型，请使用 Router 节点打包)")
|
||||
|
||||
dialogue = json.loads(dialogue_json)
|
||||
full_waveform = []
|
||||
@@ -415,11 +267,9 @@ class AIIA_Dialogue_TTS:
|
||||
|
||||
from .aiia_cosyvoice_nodes import AIIA_CosyVoice_TTS
|
||||
from .aiia_vibevoice_nodes import AIIA_VibeVoice_TTS
|
||||
from .aiia_qwen_nodes import AIIA_Qwen_TTS
|
||||
|
||||
cosy_gen = AIIA_CosyVoice_TTS()
|
||||
vibe_gen = AIIA_VibeVoice_TTS()
|
||||
qwen_gen = AIIA_Qwen_TTS()
|
||||
|
||||
print(f"[AIIA Podcast] 开始处理对话，共 {len(dialogue)} 个片段。引擎: {tts_engine}")
|
||||
|
||||
@@ -476,13 +326,13 @@ class AIIA_Dialogue_TTS:
|
||||
internal_id = unique_speakers[spk_key]
|
||||
text = item["text"]
|
||||
|
||||
# VibeVoice does not support emotion macro text tags.
|
||||
# We send only pure text to prevent the model from reading tags aloud.
|
||||
char_len = len(text) if text else 1
|
||||
# Clean text for length calc (approx)
|
||||
clean_text = re.sub(r'\[.*?\]', '', text).strip()
|
||||
char_len = len(clean_text) if clean_text else 1
|
||||
total_char_len += char_len
|
||||
item_lengths.append(char_len)
|
||||
|
||||
final_text_lines.append(f"Speaker {internal_id}: {text}")
|
||||
final_text_lines.append(f"[{internal_id}]: {text}")
|
||||
|
||||
full_text = "\n".join(final_text_lines)
|
||||
print(f" [Batch Process] Processing {len(batch_items)} segments using {len(unique_speakers)} speakers.")
|
||||
@@ -552,114 +402,20 @@ class AIIA_Dialogue_TTS:
|
||||
})
|
||||
time_ptr[0] += 1.0
|
||||
|
||||
elif tts_engine == "Qwen3-TTS":
|
||||
# --- Qwen3-TTS Batch Maximization ---
|
||||
current_batch = []
|
||||
current_batch_char = 0
|
||||
current_hash = None
|
||||
|
||||
# Batching items by "compatibility"
|
||||
# Compatibility = Same routed model, speaker_id, and reference_audio
|
||||
def get_qwen_params(it):
|
||||
sk = get_speaker_key(it["speaker"])
|
||||
tx = it["text"]
|
||||
em = it.get("emotion", "None")
|
||||
sid = kwargs.get(f"speaker_{sk}_id", "Vivian") # Default to Vivian if empty
|
||||
if not sid.strip(): sid = "Vivian"
|
||||
ref = get_ref_audio(sk)
|
||||
pemf = kwargs.get(f"speaker_{sk}_emotion", "None")
|
||||
dia = kwargs.get(f"speaker_{sk}_dialect", "None")
|
||||
|
||||
me = em if em and em != "None" else ""
|
||||
if pemf and pemf != "None":
|
||||
el = pemf.split(" (")[0] if " (" in pemf else pemf
|
||||
me = f"{me}，{el}" if me else el
|
||||
ins = f"{me}。" if me else ""
|
||||
|
||||
# Routing: Use bundle if available, else check direct slots
|
||||
tm = qwen_model
|
||||
if qwen_model and qwen_model.get("is_bundle"):
|
||||
if ref is not None: tm = qwen_model.get("base") or qwen_model.get("default")
|
||||
elif ins: tm = qwen_model.get("design") or qwen_model.get("default")
|
||||
else: tm = qwen_model.get("custom") or qwen_model.get("default")
|
||||
elif tm is None:
|
||||
# Fallback for deprecated single-slot inputs
|
||||
if ref is not None: tm = qwen_base_model or qwen_custom_model
|
||||
elif ins: tm = qwen_design_model or qwen_custom_model
|
||||
else: tm = qwen_custom_model or qwen_base_model or qwen_design_model
|
||||
|
||||
# Dialect is part of compatibility
|
||||
return {
|
||||
"tm": tm, "tx": tx, "sid": sid, "ref": ref, "ins": ins, "me": me, "sk": sk,
|
||||
"dialect": dia,
|
||||
"h": (id(tm), dia), # Gouping key
|
||||
"original_speaker": it["speaker"],
|
||||
"original_item": it # Keep original item for visual tag
|
||||
}
|
||||
|
||||
for it in batch_items:
|
||||
p = get_qwen_params(it)
|
||||
|
||||
# Check if the current item is compatible with the current batch
|
||||
# Compatibility: same routed model (via hash), and total char count within limit
|
||||
can_m = (current_hash is not None and p["h"] == current_hash and (current_batch_char + len(p["tx"]) < max_batch_char))
|
||||
|
||||
if not can_m:
|
||||
# If not compatible, or if it's the first item, flush the previous batch (if any)
|
||||
if current_batch:
|
||||
self._generate_qwen_batch(current_batch, qwen_gen, current_full_wav, sr_ptr, segments_info, time_ptr, speed_global, cfg_scale, temperature, top_k, top_p)
|
||||
|
||||
# Start a new batch
|
||||
current_batch = [p]
|
||||
current_batch_char = len(p["tx"])
|
||||
current_hash = p["h"]
|
||||
else:
|
||||
# Add to current batch
|
||||
current_batch.append(p)
|
||||
current_batch_char += len(p["tx"])
|
||||
|
||||
# Flush any remaining items in the last batch
|
||||
if current_batch:
|
||||
self._generate_qwen_batch(current_batch, qwen_gen, current_full_wav, sr_ptr, segments_info, time_ptr, speed_global, cfg_scale, temperature, top_k, top_p)
|
||||
|
||||
else:
|
||||
# CosyVoice (Iterative)
|
||||
for i, item in enumerate(batch_items):
|
||||
spk_name = item["speaker"]
|
||||
spk_key = get_speaker_key(spk_name)
|
||||
text = item["text"]
|
||||
emotion = item.get("emotion") # In CosyVoice, we put it in [] in text
|
||||
emotion = item.get("emotion", "None")
|
||||
|
||||
# Emotion compatibility check
|
||||
is_expressive = False
|
||||
if cosyvoice_model:
|
||||
is_expressive = cosyvoice_model.get("is_instruct") or cosyvoice_model.get("is_v2") or cosyvoice_model.get("is_v3")
|
||||
|
||||
if is_expressive:
|
||||
# Merge preset emotion
|
||||
preset_emo_full = kwargs.get(f"speaker_{spk_key}_emotion", "None")
|
||||
if preset_emo_full and preset_emo_full != "None":
|
||||
emo_label = preset_emo_full.split(" (")[0] if " (" in preset_emo_full else preset_emo_full
|
||||
if emotion: text = f"[{emotion}, {emo_label}] {text}"
|
||||
else: text = f"[{emo_label}] {text}"
|
||||
elif emotion:
|
||||
text = f"[{emotion}] {text}"
|
||||
|
||||
spk_id = kwargs.get(f"speaker_{spk_key}_id", "")
|
||||
ref_audio = get_ref_audio(spk_key)
|
||||
|
||||
# CosyVoice uses instruct_text for emotion, so we use the merged emotion for it
|
||||
merged_emo_for_instruct = ""
|
||||
if is_expressive:
|
||||
if preset_emo_full and preset_emo_full != "None":
|
||||
merged_emo_for_instruct = preset_emo_full.split(" (")[0] if " (" in preset_emo_full else preset_emo_full
|
||||
elif item.get("emotion") and item.get("emotion") != "None": # Use script emotion if no preset
|
||||
merged_emo_for_instruct = item.get("emotion")
|
||||
|
||||
instruct = f"{merged_emo_for_instruct}." if merged_emo_for_instruct else ""
|
||||
instruct = f"{emotion}." if emotion and emotion != "None" else ""
|
||||
|
||||
print(f" [CosyVoice Text] {spk_name}: {text}")
|
||||
if instruct:
|
||||
print(f" [CosyVoice Instruct] {instruct}")
|
||||
print(f" [Processing] {spk_name}: {text[:15]}...")
|
||||
try:
|
||||
res = cosy_gen.generate(
|
||||
model=cosyvoice_model,
|
||||
@@ -668,7 +424,7 @@ class AIIA_Dialogue_TTS:
|
||||
spk_id=spk_id,
|
||||
speed=speed_global,
|
||||
seed=42+i,
|
||||
dialect=kwargs.get(f"speaker_{spk_key}_dialect", "None (Auto)"),
|
||||
dialect="None (Auto)",
|
||||
emotion="None (Neutral)",
|
||||
reference_audio=ref_audio
|
||||
)
|
||||
@@ -779,7 +535,7 @@ NODE_CLASS_MAPPINGS = {
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"AIIA_Podcast_Script_Parser": "AIIA Podcast Script Parser",
|
||||
"AIIA_Dialogue_TTS": "AIIA Dialogue TTS (Multi-Role)",
|
||||
"AIIA_Segment_Merge": "AIIA Segment Merge (Visual)"
|
||||
"AIIA_Podcast_Script_Parser": "📜 AIIA Podcast Script Parser",
|
||||
"AIIA_Dialogue_TTS": "🎧 AIIA Dialogue TTS (Multi-Role)",
|
||||
"AIIA_Segment_Merge": "🔗 AIIA Segment Merge (Visual)"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
import json
|
||||
|
||||
|
||||
class AIIA_Podcast_Splitter:
|
||||
"""
|
||||
将多角色对话脚本按说话人拆分为独立文本。
|
||||
|
||||
输入: Script Parser 输出的 dialogue_json
|
||||
输出: 每个说话人的拼接文本（用于独立 TTS）+ 重组映射表
|
||||
"""
|
||||
|
||||
NODE_NAME = "AIIA Podcast Splitter"
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"dialogue_json": ("STRING", {"forceInput": True}),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("STRING", "STRING", "STRING",)
|
||||
RETURN_NAMES = ("speaker_A_text", "speaker_B_text", "split_map",)
|
||||
FUNCTION = "split_dialogue"
|
||||
CATEGORY = "AIIA/Podcast"
|
||||
|
||||
def split_dialogue(self, dialogue_json):
|
||||
log = f"[{self.NODE_NAME}]"
|
||||
|
||||
# 解析 dialogue_json
|
||||
try:
|
||||
dialogue = json.loads(dialogue_json)
|
||||
except json.JSONDecodeError as e:
|
||||
print(f"{log} JSON 解析失败: {e}")
|
||||
empty_map = json.dumps([], ensure_ascii=False)
|
||||
return ("", "", empty_map)
|
||||
|
||||
if not isinstance(dialogue, list):
|
||||
print(f"{log} 错误: dialogue_json 不是列表")
|
||||
empty_map = json.dumps([], ensure_ascii=False)
|
||||
return ("", "", empty_map)
|
||||
|
||||
# 收集所有说话人
|
||||
speakers_seen = []
|
||||
for item in dialogue:
|
||||
if item.get("type") == "speech":
|
||||
spk = item["speaker"]
|
||||
if spk not in speakers_seen:
|
||||
speakers_seen.append(spk)
|
||||
|
||||
if len(speakers_seen) == 0:
|
||||
print(f"{log} 警告: 没有找到任何说话人")
|
||||
empty_map = json.dumps([], ensure_ascii=False)
|
||||
return ("", "", empty_map)
|
||||
|
||||
if len(speakers_seen) > 2:
|
||||
print(f"{log} 警告: 发现 {len(speakers_seen)} 个说话人 ({speakers_seen})，仅使用前两个")
|
||||
|
||||
speaker_A = speakers_seen[0] if len(speakers_seen) > 0 else None
|
||||
speaker_B = speakers_seen[1] if len(speakers_seen) > 1 else None
|
||||
|
||||
print(f"{log} Speaker A: {speaker_A}, Speaker B: {speaker_B}")
|
||||
|
||||
# 按说话人分组，同时记录顺序映射
|
||||
texts_A = [] # Speaker A 的所有台词
|
||||
texts_B = [] # Speaker B 的所有台词
|
||||
split_map = [] # 原始顺序映射
|
||||
|
||||
for item in dialogue:
|
||||
if item.get("type") != "speech":
|
||||
# 暂停等非语音条目也记录到 split_map
|
||||
if item.get("type") == "pause":
|
||||
split_map.append({
|
||||
"type": "pause",
|
||||
"duration": item.get("duration", 0.3),
|
||||
})
|
||||
continue
|
||||
|
||||
text = item["text"]
|
||||
speaker = item["speaker"]
|
||||
|
||||
if speaker == speaker_A:
|
||||
split_map.append({
|
||||
"type": "speech",
|
||||
"speaker": "A",
|
||||
"index": len(texts_A),
|
||||
"text": text,
|
||||
"original_speaker": speaker,
|
||||
})
|
||||
texts_A.append(text)
|
||||
elif speaker == speaker_B:
|
||||
split_map.append({
|
||||
"type": "speech",
|
||||
"speaker": "B",
|
||||
"index": len(texts_B),
|
||||
"text": text,
|
||||
"original_speaker": speaker,
|
||||
})
|
||||
texts_B.append(text)
|
||||
else:
|
||||
print(f"{log} 跳过第三个说话人 '{speaker}' 的台词: {text[:30]}...")
|
||||
|
||||
# 拼接每个说话人的文本
|
||||
# 每句之间用换行分隔（TTS 会在换行处产生自然停顿）
|
||||
speaker_A_text = "\n".join(texts_A)
|
||||
speaker_B_text = "\n".join(texts_B)
|
||||
|
||||
split_map_json = json.dumps(split_map, ensure_ascii=False, indent=2)
|
||||
|
||||
print(f"{log} 拆分完成:")
|
||||
print(f" Speaker A ({speaker_A}): {len(texts_A)} 句, {len(speaker_A_text)} 字符")
|
||||
print(f" Speaker B ({speaker_B}): {len(texts_B)} 句, {len(speaker_B_text)} 字符")
|
||||
print(f" split_map: {len(split_map)} 条目")
|
||||
|
||||
return (speaker_A_text, speaker_B_text, split_map_json)
|
||||
|
||||
|
||||
# --- ComfyUI 节点注册 ---
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"AIIA_Podcast_Splitter": AIIA_Podcast_Splitter,
|
||||
}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"AIIA_Podcast_Splitter": "✂️ AIIA Podcast Splitter",
|
||||
}
|
||||
@@ -0,0 +1,481 @@
|
||||
import json
|
||||
import torch
|
||||
import numpy as np
|
||||
|
||||
|
||||
class AIIA_Podcast_Stitcher:
|
||||
"""
|
||||
将分轨生成的多角色音频按原始对话顺序精确拼接。
|
||||
|
||||
利用 ASR 词级时间戳找到每句话在音频中的边界，切分后交错拼接。
|
||||
"""
|
||||
|
||||
NODE_NAME = "AIIA Podcast Stitcher"
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"split_map": ("STRING", {"forceInput": True}),
|
||||
"audio_A": ("AUDIO",),
|
||||
"audio_B": ("AUDIO",),
|
||||
"asr_A": ("ASR_RESULT",),
|
||||
"asr_B": ("ASR_RESULT",),
|
||||
},
|
||||
"optional": {
|
||||
"gap_duration": ("FLOAT", {
|
||||
"default": 0.3, "min": 0.0, "max": 2.0, "step": 0.05,
|
||||
"tooltip": "说话人交替时插入的静音时长（秒）"
|
||||
}),
|
||||
"padding": ("FLOAT", {
|
||||
"default": 0.05, "min": 0.0, "max": 0.5, "step": 0.01,
|
||||
"tooltip": "每个切片前后保留的呼吸/尾音余量（秒）"
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("AUDIO", "STRING",)
|
||||
RETURN_NAMES = ("audio", "segments_info",)
|
||||
FUNCTION = "stitch"
|
||||
CATEGORY = "AIIA/Podcast"
|
||||
|
||||
def _audio_to_numpy(self, audio: dict) -> tuple:
|
||||
"""将 ComfyUI AUDIO 转为 numpy 数组和采样率。"""
|
||||
waveform = audio["waveform"]
|
||||
sr = audio["sample_rate"]
|
||||
|
||||
if waveform.ndim == 3:
|
||||
wav = waveform[0]
|
||||
else:
|
||||
wav = waveform
|
||||
|
||||
if wav.ndim == 2 and wav.shape[0] > 1:
|
||||
wav = wav.mean(dim=0)
|
||||
elif wav.ndim == 2:
|
||||
wav = wav.squeeze(0)
|
||||
|
||||
return wav.cpu().numpy().astype(np.float32), sr
|
||||
|
||||
def _find_sentence_boundaries(self, asr_words: list, sentences: list, total_duration: float) -> list:
|
||||
"""
|
||||
将 ASR 词级时间戳与原始句子列表对齐，找到每句话在音频中的时间范围。
|
||||
|
||||
三层匹配策略：
|
||||
1. 精确子串匹配（去标点后）
|
||||
2. 编辑距离模糊匹配（滑动窗口，容忍 ASR 错字/漏字）
|
||||
3. 间隙填补 / 等分回退
|
||||
"""
|
||||
log = f"[{self.NODE_NAME}]"
|
||||
|
||||
if not asr_words:
|
||||
print(f"{log} ASR 结果为空，使用等分策略")
|
||||
return self._fallback_equal_split(sentences, total_duration)
|
||||
|
||||
if not sentences:
|
||||
return []
|
||||
|
||||
# 构建 ASR 文本和字符到词索引的映射
|
||||
asr_full_text = ""
|
||||
char_to_word_idx = [] # char_to_word_idx[i] = 该字符属于哪个 word
|
||||
for word_idx, w in enumerate(asr_words):
|
||||
word_text = w["word"]
|
||||
for ch in word_text:
|
||||
char_to_word_idx.append(word_idx)
|
||||
asr_full_text += word_text
|
||||
|
||||
print(f"{log} ASR 全文 ({len(asr_full_text)} 字): {asr_full_text[:100]}...")
|
||||
|
||||
# 为每句话找到在 ASR 文本中的匹配位置
|
||||
boundaries = []
|
||||
search_start = 0 # 保证顺序匹配
|
||||
|
||||
for sent_idx, sentence in enumerate(sentences):
|
||||
# 清理句子文本（去除标点符号和空格，与 ASR 输出对齐）
|
||||
clean_sent = self._clean_text_for_matching(sentence)
|
||||
|
||||
if not clean_sent:
|
||||
print(f"{log} 句子 {sent_idx} 清理后为空: '{sentence}'")
|
||||
boundaries.append(None)
|
||||
continue
|
||||
|
||||
# === 第 1 层：精确子串匹配 ===
|
||||
match_pos = asr_full_text.find(clean_sent, search_start)
|
||||
|
||||
if match_pos != -1:
|
||||
match_end = match_pos + len(clean_sent) - 1
|
||||
match_quality = "精确"
|
||||
else:
|
||||
# === 第 2 层：编辑距离模糊匹配 ===
|
||||
match_pos, match_end, edit_dist = self._fuzzy_find(
|
||||
asr_full_text, clean_sent, search_start
|
||||
)
|
||||
|
||||
if match_pos != -1:
|
||||
match_quality = f"模糊(ed={edit_dist})"
|
||||
else:
|
||||
print(f"{log} 句子 {sent_idx} 无法匹配: '{clean_sent[:30]}...'")
|
||||
boundaries.append(None)
|
||||
continue
|
||||
|
||||
# 映射字符位置到词索引
|
||||
start_word_idx = char_to_word_idx[match_pos] if match_pos < len(char_to_word_idx) else len(asr_words) - 1
|
||||
end_word_idx = char_to_word_idx[min(match_end, len(char_to_word_idx) - 1)]
|
||||
|
||||
start_time = asr_words[start_word_idx]["start"]
|
||||
end_time = asr_words[end_word_idx]["end"]
|
||||
|
||||
print(f"{log} 句子 {sent_idx} [{match_quality}]: "
|
||||
f"'{clean_sent[:15]}' → pos={match_pos}-{match_end}, "
|
||||
f"time={start_time:.2f}-{end_time:.2f}s")
|
||||
|
||||
boundaries.append({
|
||||
"start": start_time,
|
||||
"end": end_time,
|
||||
"start_word_idx": start_word_idx,
|
||||
"end_word_idx": end_word_idx,
|
||||
})
|
||||
|
||||
# 更新搜索起点
|
||||
search_start = match_end + 1
|
||||
|
||||
# 填补未匹配的句子（使用前后句子的时间插值）
|
||||
boundaries = self._fill_missing_boundaries(boundaries, asr_words, total_duration)
|
||||
|
||||
# 扩展边界到句间间隙的中点（避免截断尾音）
|
||||
boundaries = self._expand_to_midpoints(boundaries, total_duration)
|
||||
|
||||
return boundaries
|
||||
|
||||
@staticmethod
|
||||
def _edit_distance(s1: str, s2: str) -> int:
|
||||
"""计算两个字符串的编辑距离（Levenshtein distance），使用空间优化的 DP。"""
|
||||
m, n = len(s1), len(s2)
|
||||
if m == 0:
|
||||
return n
|
||||
if n == 0:
|
||||
return m
|
||||
|
||||
# 只需两行
|
||||
prev = list(range(n + 1))
|
||||
curr = [0] * (n + 1)
|
||||
|
||||
for i in range(1, m + 1):
|
||||
curr[0] = i
|
||||
for j in range(1, n + 1):
|
||||
if s1[i - 1] == s2[j - 1]:
|
||||
curr[j] = prev[j - 1]
|
||||
else:
|
||||
curr[j] = 1 + min(prev[j], curr[j - 1], prev[j - 1])
|
||||
prev, curr = curr, prev
|
||||
|
||||
return prev[n]
|
||||
|
||||
def _fuzzy_find(self, haystack: str, needle: str, search_start: int = 0,
|
||||
max_error_ratio: float = 0.4) -> tuple:
|
||||
"""
|
||||
在 haystack 中从 search_start 开始，用滑动窗口+编辑距离找到与 needle 最相似的子串。
|
||||
|
||||
参数:
|
||||
haystack: ASR 全文
|
||||
needle: 待匹配的原始句子（已去标点）
|
||||
search_start: 搜索起始位置
|
||||
max_error_ratio: 允许的最大错误率（编辑距离 / needle 长度）
|
||||
|
||||
返回:
|
||||
(match_pos, match_end, edit_distance) 或 (-1, -1, -1) 表示失败
|
||||
"""
|
||||
needle_len = len(needle)
|
||||
if needle_len == 0:
|
||||
return (-1, -1, -1)
|
||||
|
||||
max_errors = int(needle_len * max_error_ratio)
|
||||
remaining = haystack[search_start:]
|
||||
remaining_len = len(remaining)
|
||||
|
||||
if remaining_len == 0:
|
||||
return (-1, -1, -1)
|
||||
|
||||
best_pos = -1
|
||||
best_end = -1
|
||||
best_dist = needle_len + 1 # 初始化为一个大值
|
||||
|
||||
# 尝试多种窗口大小（needle 长度的 ±30%），处理 ASR 漏字/多字的情况
|
||||
window_sizes = set()
|
||||
for ratio in [1.0, 0.85, 0.9, 0.95, 1.05, 1.1, 1.15, 1.2]:
|
||||
ws = max(1, int(needle_len * ratio))
|
||||
if ws <= remaining_len:
|
||||
window_sizes.add(ws)
|
||||
|
||||
# 限制搜索范围以避免 O(n²) 爆炸
|
||||
# 在合理的搜索范围内：从 search_start 开始，最多搜到 needle 长度的 3 倍
|
||||
max_search_len = min(remaining_len, needle_len * 3 + 20)
|
||||
|
||||
for window_size in sorted(window_sizes):
|
||||
for i in range(0, max_search_len - window_size + 1):
|
||||
candidate = remaining[i:i + window_size]
|
||||
dist = self._edit_distance(needle, candidate)
|
||||
|
||||
if dist < best_dist:
|
||||
best_dist = dist
|
||||
best_pos = search_start + i
|
||||
best_end = search_start + i + window_size - 1
|
||||
|
||||
# 如果编辑距离为 0 或 1，可以提前退出
|
||||
if dist <= 1:
|
||||
break
|
||||
|
||||
if best_dist <= 1:
|
||||
break
|
||||
|
||||
# 只接受错误率在阈值内的匹配
|
||||
if best_dist <= max_errors:
|
||||
return (best_pos, best_end, best_dist)
|
||||
else:
|
||||
return (-1, -1, -1)
|
||||
|
||||
def _clean_text_for_matching(self, text: str) -> str:
|
||||
"""清理文本用于与 ASR 输出匹配：去除标点、空格、英文转小写。"""
|
||||
import re
|
||||
# 去除常见中英文标点和空格
|
||||
cleaned = re.sub(r'[，。！？、；：""''「」【】（）《》\s,\.!?\-\;\:\"\'\(\)\[\]\{\}…—~～·]', '', text)
|
||||
# 英文转小写（ASR 可能输出不同大小写）
|
||||
cleaned = cleaned.lower()
|
||||
return cleaned
|
||||
|
||||
def _fallback_equal_split(self, sentences: list, total_duration: float) -> list:
|
||||
"""回退策略：按句子字符数等比例分配时间。"""
|
||||
if not sentences:
|
||||
return []
|
||||
|
||||
total_chars = sum(len(s) for s in sentences)
|
||||
if total_chars == 0:
|
||||
segment_duration = total_duration / len(sentences)
|
||||
return [{"start": i * segment_duration, "end": (i + 1) * segment_duration}
|
||||
for i in range(len(sentences))]
|
||||
|
||||
boundaries = []
|
||||
current_time = 0.0
|
||||
for sent in sentences:
|
||||
ratio = len(sent) / total_chars
|
||||
duration = ratio * total_duration
|
||||
boundaries.append({
|
||||
"start": round(current_time, 3),
|
||||
"end": round(current_time + duration, 3),
|
||||
})
|
||||
current_time += duration
|
||||
|
||||
return boundaries
|
||||
|
||||
def _fill_missing_boundaries(self, boundaries: list, asr_words: list, total_duration: float) -> list:
|
||||
"""填补未能匹配的句子边界。"""
|
||||
filled = list(boundaries)
|
||||
|
||||
for i in range(len(filled)):
|
||||
if filled[i] is not None:
|
||||
continue
|
||||
|
||||
# 找前一个已知边界
|
||||
prev_end = 0.0
|
||||
for j in range(i - 1, -1, -1):
|
||||
if filled[j] is not None:
|
||||
prev_end = filled[j]["end"]
|
||||
break
|
||||
|
||||
# 找后一个已知边界
|
||||
next_start = total_duration
|
||||
for j in range(i + 1, len(filled)):
|
||||
if filled[j] is not None:
|
||||
next_start = filled[j]["start"]
|
||||
break
|
||||
|
||||
# 在空隙中均匀分配
|
||||
gap_count = 0
|
||||
gap_start_idx = i
|
||||
for j in range(i, len(filled)):
|
||||
if filled[j] is None:
|
||||
gap_count += 1
|
||||
else:
|
||||
break
|
||||
|
||||
gap_duration = (next_start - prev_end) / gap_count
|
||||
for k in range(gap_count):
|
||||
filled[gap_start_idx + k] = {
|
||||
"start": round(prev_end + k * gap_duration, 3),
|
||||
"end": round(prev_end + (k + 1) * gap_duration, 3),
|
||||
}
|
||||
|
||||
return filled
|
||||
|
||||
def _expand_to_midpoints(self, boundaries: list, total_duration: float) -> list:
|
||||
"""将切割点扩展到相邻句子间隙的中点，避免截断尾音/吸气声。"""
|
||||
if len(boundaries) <= 1:
|
||||
if boundaries:
|
||||
boundaries[0]["cut_start"] = 0.0
|
||||
boundaries[0]["cut_end"] = total_duration
|
||||
return boundaries
|
||||
|
||||
for i in range(len(boundaries)):
|
||||
if i == 0:
|
||||
boundaries[i]["cut_start"] = 0.0
|
||||
else:
|
||||
# 与前一句的间隙中点
|
||||
gap_mid = (boundaries[i - 1]["end"] + boundaries[i]["start"]) / 2
|
||||
boundaries[i]["cut_start"] = round(gap_mid, 3)
|
||||
|
||||
if i == len(boundaries) - 1:
|
||||
boundaries[i]["cut_end"] = total_duration
|
||||
else:
|
||||
# 与后一句的间隙中点
|
||||
gap_mid = (boundaries[i]["end"] + boundaries[i + 1]["start"]) / 2
|
||||
boundaries[i]["cut_end"] = round(gap_mid, 3)
|
||||
|
||||
return boundaries
|
||||
|
||||
def stitch(self, split_map, audio_A, audio_B, asr_A, asr_B,
|
||||
gap_duration=0.3, padding=0.05):
|
||||
log = f"[{self.NODE_NAME}]"
|
||||
|
||||
# 解析 split_map
|
||||
try:
|
||||
map_items = json.loads(split_map)
|
||||
except json.JSONDecodeError as e:
|
||||
print(f"{log} split_map JSON 解析失败: {e}")
|
||||
return (audio_A, "[]")
|
||||
|
||||
# 提取音频数据
|
||||
wav_A, sr_A = self._audio_to_numpy(audio_A)
|
||||
wav_B, sr_B = self._audio_to_numpy(audio_B)
|
||||
duration_A = len(wav_A) / sr_A
|
||||
duration_B = len(wav_B) / sr_B
|
||||
|
||||
# 使用统一采样率
|
||||
sr = sr_A
|
||||
if sr_A != sr_B:
|
||||
print(f"{log} 警告: sr_A={sr_A} != sr_B={sr_B}, 使用 sr_A")
|
||||
|
||||
print(f"{log} Audio A: {duration_A:.2f}s, Audio B: {duration_B:.2f}s, SR: {sr}")
|
||||
|
||||
# 收集每个说话人的句子列表
|
||||
sentences_A = [item["text"] for item in map_items if item.get("type") == "speech" and item.get("speaker") == "A"]
|
||||
sentences_B = [item["text"] for item in map_items if item.get("type") == "speech" and item.get("speaker") == "B"]
|
||||
|
||||
print(f"{log} 句子数 - A: {len(sentences_A)}, B: {len(sentences_B)}")
|
||||
|
||||
# ASR 对齐切分
|
||||
words_A = asr_A.get("words", []) if isinstance(asr_A, dict) else []
|
||||
words_B = asr_B.get("words", []) if isinstance(asr_B, dict) else []
|
||||
|
||||
print(f"{log} ASR 词数 - A: {len(words_A)}, B: {len(words_B)}")
|
||||
|
||||
boundaries_A = self._find_sentence_boundaries(words_A, sentences_A, duration_A)
|
||||
boundaries_B = self._find_sentence_boundaries(words_B, sentences_B, duration_B)
|
||||
|
||||
print(f"{log} 边界数 - A: {len(boundaries_A)}, B: {len(boundaries_B)}")
|
||||
|
||||
# 按 split_map 顺序拼接
|
||||
audio_segments = []
|
||||
segments_info = []
|
||||
current_time = 0.0
|
||||
idx_A = 0
|
||||
idx_B = 0
|
||||
prev_speaker = None
|
||||
|
||||
for item in map_items:
|
||||
if item.get("type") == "pause":
|
||||
# 显式暂停
|
||||
pause_dur = item.get("duration", 0.3)
|
||||
pause_samples = int(pause_dur * sr)
|
||||
audio_segments.append(np.zeros(pause_samples, dtype=np.float32))
|
||||
current_time += pause_dur
|
||||
continue
|
||||
|
||||
if item.get("type") != "speech":
|
||||
continue
|
||||
|
||||
speaker = item["speaker"]
|
||||
|
||||
# 说话人切换时插入间隙
|
||||
if prev_speaker is not None and speaker != prev_speaker:
|
||||
gap_samples = int(gap_duration * sr)
|
||||
audio_segments.append(np.zeros(gap_samples, dtype=np.float32))
|
||||
current_time += gap_duration
|
||||
|
||||
# 获取对应的边界和音频
|
||||
if speaker == "A":
|
||||
if idx_A >= len(boundaries_A):
|
||||
print(f"{log} 警告: A 的句子索引 {idx_A} 超出边界数 {len(boundaries_A)}")
|
||||
idx_A += 1
|
||||
continue
|
||||
boundary = boundaries_A[idx_A]
|
||||
wav = wav_A
|
||||
idx_A += 1
|
||||
elif speaker == "B":
|
||||
if idx_B >= len(boundaries_B):
|
||||
print(f"{log} 警告: B 的句子索引 {idx_B} 超出边界数 {len(boundaries_B)}")
|
||||
idx_B += 1
|
||||
continue
|
||||
boundary = boundaries_B[idx_B]
|
||||
wav = wav_B
|
||||
idx_B += 1
|
||||
else:
|
||||
continue
|
||||
|
||||
# 切割音频片段（使用 cut_start/cut_end，带 padding）
|
||||
cut_start = boundary.get("cut_start", boundary["start"])
|
||||
cut_end = boundary.get("cut_end", boundary["end"])
|
||||
|
||||
# 应用 padding
|
||||
cut_start = max(0, cut_start - padding)
|
||||
cut_end = min(len(wav) / sr, cut_end + padding)
|
||||
|
||||
start_sample = int(cut_start * sr)
|
||||
end_sample = int(cut_end * sr)
|
||||
end_sample = min(end_sample, len(wav))
|
||||
|
||||
segment = wav[start_sample:end_sample]
|
||||
|
||||
if len(segment) == 0:
|
||||
print(f"{log} 警告: 空片段 at {cut_start:.3f}-{cut_end:.3f}s")
|
||||
continue
|
||||
|
||||
seg_duration = len(segment) / sr
|
||||
|
||||
audio_segments.append(segment)
|
||||
|
||||
# 记录 segment info
|
||||
original_speaker = item.get("original_speaker", speaker)
|
||||
segments_info.append({
|
||||
"start": round(current_time, 3),
|
||||
"end": round(current_time + seg_duration, 3),
|
||||
"text": item["text"],
|
||||
"speaker": original_speaker,
|
||||
})
|
||||
|
||||
current_time += seg_duration
|
||||
prev_speaker = speaker
|
||||
|
||||
# 拼接所有片段
|
||||
if not audio_segments:
|
||||
print(f"{log} 错误: 没有任何音频片段")
|
||||
return (audio_A, "[]")
|
||||
|
||||
final_audio = np.concatenate(audio_segments)
|
||||
total_duration = len(final_audio) / sr
|
||||
print(f"{log} 拼接完成: {total_duration:.2f}s, {len(segments_info)} 个语音段")
|
||||
|
||||
# 转为 ComfyUI AUDIO 格式
|
||||
audio_tensor = torch.from_numpy(final_audio).unsqueeze(0).unsqueeze(0) # (1, 1, samples)
|
||||
audio_output = {"waveform": audio_tensor, "sample_rate": sr}
|
||||
|
||||
segments_info_json = json.dumps(segments_info, ensure_ascii=False, indent=2)
|
||||
|
||||
return (audio_output, segments_info_json)
|
||||
|
||||
|
||||
# --- ComfyUI 节点注册 ---
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"AIIA_Podcast_Stitcher": AIIA_Podcast_Stitcher,
|
||||
}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"AIIA_Podcast_Stitcher": "🧵 AIIA Podcast Stitcher",
|
||||
}
|
||||
Executable → Regular
+12
-428
@@ -2,7 +2,6 @@ import json
|
||||
import datetime
|
||||
import os
|
||||
import random
|
||||
import re
|
||||
import torchaudio
|
||||
import folder_paths
|
||||
|
||||
@@ -16,12 +15,9 @@ class AIIA_Subtitle_Gen:
|
||||
"required": {
|
||||
"segments_info": ("STRING", {"forceInput": True}),
|
||||
"format": (["SRT", "ASS", "Both"], {"default": "SRT"}),
|
||||
"save_file": ("BOOLEAN", {"default": False, "label_on": "Save to Disk", "label_off": "Memory Only"}),
|
||||
},
|
||||
"optional": {
|
||||
"calibration_info": ("WHISPER_CHUNKS",),
|
||||
"ass_style": ("STRING", {"default": "Default", "multiline": False}),
|
||||
"filename_prefix": ("STRING", {"default": "aiia_subtitle"}),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -29,9 +25,8 @@ class AIIA_Subtitle_Gen:
|
||||
RETURN_NAMES = ("srt_content", "ass_content")
|
||||
FUNCTION = "generate_subtitle"
|
||||
CATEGORY = "AIIA/Subtitle"
|
||||
OUTPUT_NODE = True
|
||||
|
||||
def generate_subtitle(self, segments_info, format="SRT", save_file=False, ass_style="Default", filename_prefix="aiia_subtitle", calibration_info=None):
|
||||
def generate_subtitle(self, segments_info, format="SRT", ass_style="Default"):
|
||||
try:
|
||||
segments = json.loads(segments_info)
|
||||
except Exception as e:
|
||||
@@ -42,15 +37,6 @@ class AIIA_Subtitle_Gen:
|
||||
print("[AIIA Subtitle] Segments info must be a list of dicts.")
|
||||
return ("", "")
|
||||
|
||||
# --- Subtitle Calibration (v1.10.2) ---
|
||||
if calibration_info and "chunks" in calibration_info:
|
||||
print(f"[AIIA Subtitle] Calibrating {len(segments)} segments using {len(calibration_info['chunks'])} high-precision chunks.")
|
||||
segments = self._calibrate_segments(segments, calibration_info["chunks"])
|
||||
|
||||
if not isinstance(segments, list):
|
||||
print("[AIIA Subtitle] Segments info must be a list of dicts.")
|
||||
return ("", "")
|
||||
|
||||
srt_out = ""
|
||||
ass_out = ""
|
||||
|
||||
@@ -59,36 +45,12 @@ class AIIA_Subtitle_Gen:
|
||||
|
||||
if format in ["ASS", "Both"]:
|
||||
ass_out = self._generate_ass(segments, ass_style)
|
||||
|
||||
# File Saving Logic
|
||||
if save_file:
|
||||
output_dir = folder_paths.get_output_directory()
|
||||
|
||||
# Timestamp (uniques)
|
||||
timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
|
||||
if format in ["SRT", "Both"]:
|
||||
srt_name = f"{filename_prefix}_{timestamp}.srt"
|
||||
srt_path = os.path.join(output_dir, srt_name)
|
||||
with open(srt_path, "w", encoding="utf-8") as f:
|
||||
f.write(srt_out)
|
||||
print(f"[AIIA Subtitle] Saved SRT to: {srt_path}")
|
||||
|
||||
if format in ["ASS", "Both"]:
|
||||
ass_name = f"{filename_prefix}_{timestamp}.ass"
|
||||
ass_path = os.path.join(output_dir, ass_name)
|
||||
with open(ass_path, "w", encoding="utf-8") as f:
|
||||
f.write(ass_out)
|
||||
print(f"[AIIA Subtitle] Saved ASS to: {ass_path}")
|
||||
|
||||
return (srt_out, ass_out)
|
||||
|
||||
def _generate_srt(self, segments):
|
||||
output = []
|
||||
# Filter out zero-duration (silenced) segments
|
||||
valid_segments = [s for s in segments if s["end"] - s["start"] >= 0.01]
|
||||
|
||||
for i, seg in enumerate(valid_segments):
|
||||
for i, seg in enumerate(segments):
|
||||
start = self._format_srt_time(seg["start"])
|
||||
end = self._format_srt_time(seg["end"])
|
||||
text = seg["text"]
|
||||
@@ -100,21 +62,18 @@ class AIIA_Subtitle_Gen:
|
||||
return "\n".join(output)
|
||||
|
||||
def _generate_ass(self, segments, style_name="Default"):
|
||||
# Filter out zero-duration (silenced) segments
|
||||
valid_segments = [s for s in segments if s["end"] - s["start"] >= 0.01]
|
||||
|
||||
# 1. Collect unique speakers
|
||||
speakers = set()
|
||||
for seg in valid_segments:
|
||||
for seg in segments:
|
||||
speakers.add(seg.get("speaker", "Unknown"))
|
||||
|
||||
# 2. Assign colors to speakers
|
||||
# Simple palette: White, Yellow, Cyan, Green, Orange, Pink, LightBlue
|
||||
palette = [
|
||||
"&H00FFFFFF", # White (Pure)
|
||||
"&H0000D7FF", # Gold/Yellow (Cinematic)
|
||||
"&H00FFFF00", # Cyan (Standard)
|
||||
"&H0000FF00", # Green (Lime)
|
||||
"&H00FFFFFF", # White
|
||||
"&H0000FFFF", # Yellow (BGR)
|
||||
"&H00FFFF00", # Cyan
|
||||
"&H0000FF00", # Green
|
||||
"&H000080FF", # Orange
|
||||
"&H00FF80FF", # Pink
|
||||
"&H00FFC0C0" # LightBlue
|
||||
@@ -125,10 +84,7 @@ class AIIA_Subtitle_Gen:
|
||||
|
||||
# Base Style String Template
|
||||
# Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, ...
|
||||
# Base Style String Template
|
||||
# Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, ...
|
||||
# Changes: Bold=1, Outline=2, Shadow=1 for professional look
|
||||
base_style = "Arial,40,{primary_color},&H000000FF,&H00000000,&H00000000,0,0,0,0,100,100,0,0,1,2,1,2,10,10,10,1"
|
||||
base_style = "Arial,20,{primary_color},&H000000FF,&H00000000,&H00000000,0,0,0,0,100,100,0,0,1,1,0,2,10,10,10,1"
|
||||
|
||||
sorted_speakers = sorted(list(speakers))
|
||||
for i, spk in enumerate(sorted_speakers):
|
||||
@@ -161,12 +117,10 @@ class AIIA_Subtitle_Gen:
|
||||
]
|
||||
|
||||
events = []
|
||||
for seg in valid_segments:
|
||||
for seg in segments:
|
||||
start = self._format_ass_time(seg["start"])
|
||||
end = self._format_ass_time(seg["end"])
|
||||
text = seg["text"].replace("\n", "\\N")
|
||||
# [v1.10.8] Support Markdown Bold (**text**) -> ASS Bold ({\b1}text{\b0})
|
||||
text = re.sub(r'\*\*(.*?)\*\*', r'{\\b1}\1{\\b0}', text)
|
||||
speaker = seg.get("speaker", "Unknown")
|
||||
# Use the mapped style name
|
||||
style_for_event = speaker_map.get(speaker, "Default")
|
||||
@@ -188,260 +142,6 @@ class AIIA_Subtitle_Gen:
|
||||
|
||||
return f"{hours:02}:{minutes:02}:{secs:02},{millis:03}"
|
||||
|
||||
def _calibrate_segments(self, segments, chunks):
|
||||
"""
|
||||
Calibrate estimated segments using high-precision VAD chunks.
|
||||
Algorithm: Iterative sequence matching with speaker-centric isolation (v1.10.5).
|
||||
"""
|
||||
if not chunks:
|
||||
return segments
|
||||
|
||||
# 1. Ensure chunks are sorted chronologically
|
||||
sorted_chunks = sorted(chunks, key=lambda x: x["timestamp"][0])
|
||||
|
||||
calibrated = []
|
||||
chunk_idx = 0
|
||||
num_chunks = len(sorted_chunks)
|
||||
|
||||
def normalize_spk(s):
|
||||
if not s: return ""
|
||||
return str(s).lower().replace("speaker_", "").replace("speaker ", "").strip()
|
||||
|
||||
# [v1.10.17] Speaker Mapping: Script Identity -> VAD Identity
|
||||
speaker_map = {} # e.g. {"speaker_a": "speaker_00", "speaker_b": "speaker_01"}
|
||||
|
||||
for i, seg in enumerate(segments):
|
||||
seg_start = seg["start"]
|
||||
seg_end = seg["end"]
|
||||
seg_dur = seg_end - seg_start
|
||||
seg_spk = normalize_spk(seg.get("speaker"))
|
||||
|
||||
# --- Speaker-Centric Magic (v1.10.5) ---
|
||||
# 1. Find the "Winner Speaker" for this segment based on maximum overlap duration
|
||||
speaker_overlaps = {}
|
||||
# Large window for initial scan to be robust
|
||||
scan_idx = chunk_idx
|
||||
while scan_idx < num_chunks:
|
||||
c = sorted_chunks[scan_idx]
|
||||
c_start, c_end = c["timestamp"]
|
||||
# Hard break if the chunk is way past our segment (relaxed to 60s for sequential matching)
|
||||
if c_start > seg_end + 60.0: break
|
||||
|
||||
# Calculate overlap duration
|
||||
overlap = min(seg_end, c_end) - max(seg_start, c_start)
|
||||
if overlap > 0:
|
||||
spk = normalize_spk(c.get("speaker", "unknown"))
|
||||
speaker_overlaps[spk] = speaker_overlaps.get(spk, 0.0) + overlap
|
||||
scan_idx += 1
|
||||
|
||||
winner_spk = None
|
||||
|
||||
# [v1.10.17] Consistency Logic
|
||||
# If we already know who this Script Speaker maps to, try to find THAT VAD Speaker first.
|
||||
if seg_spk in speaker_map:
|
||||
mapped_vad_spk = speaker_map[seg_spk]
|
||||
# Trust the map (Sequence Priority). Even if no overlap, we search for this speaker.
|
||||
winner_spk = mapped_vad_spk
|
||||
|
||||
# Fallback (First time seeing this speaker OR mapped speaker missing): Max Overlap
|
||||
if winner_spk is None and speaker_overlaps:
|
||||
# Get speaker with most accumulated overlap duration
|
||||
winner_spk = max(speaker_overlaps, key=speaker_overlaps.get)
|
||||
|
||||
# Update Map
|
||||
if winner_spk and seg_spk:
|
||||
speaker_map[seg_spk] = winner_spk
|
||||
|
||||
# 2. Find chunks belonging to the winner spk to use for snapping
|
||||
# [v1.10.16] Greedy Speaker Turn Collection: collect all consecutive chunks for winner_spk
|
||||
matched_chunks = []
|
||||
find_idx = chunk_idx
|
||||
found_start = False
|
||||
|
||||
while find_idx < num_chunks:
|
||||
chunk = sorted_chunks[find_idx]
|
||||
c_start, c_end = chunk["timestamp"]
|
||||
c_spk = normalize_spk(chunk.get("speaker", "unknown"))
|
||||
|
||||
overlap = min(seg_end, c_end) - max(seg_start, c_start)
|
||||
is_overlap = overlap > 0.05
|
||||
|
||||
# Case: First segment special snapping
|
||||
if not is_overlap and i == 0 and find_idx == 0:
|
||||
if abs(c_start - seg_start) < 0.5 and c_end > seg_start - 0.1:
|
||||
is_overlap = True
|
||||
|
||||
if not found_start:
|
||||
# Looking for the first chunk that belongs to our speaker
|
||||
# Trust Sequence: If we have a winner_spk, find their next chunk regardless of overlap
|
||||
match_condition = is_overlap
|
||||
if winner_spk is not None:
|
||||
match_condition = is_overlap or (c_spk == winner_spk)
|
||||
|
||||
if match_condition and (winner_spk is None or c_spk == winner_spk):
|
||||
matched_chunks.append(find_idx)
|
||||
found_start = True
|
||||
if winner_spk is None: winner_spk = c_spk
|
||||
elif c_start > seg_end + 60.0:
|
||||
# Way past the estimated end, give up
|
||||
break
|
||||
else:
|
||||
# Already started collecting.
|
||||
if c_spk == winner_spk:
|
||||
# Ensure no massive gap (e.g. 8s) within a single turn, unless it's within expected segment time
|
||||
last_matched_end = sorted_chunks[matched_chunks[-1]]["timestamp"][1]
|
||||
if (c_start - last_matched_end < 8.0) or (c_start < seg_end + 1.0):
|
||||
matched_chunks.append(find_idx)
|
||||
else:
|
||||
break
|
||||
else:
|
||||
# [v1.10.18 Fix] Encountered unrelated speaker.
|
||||
# Check if this is just a brief interruption (noise/other speaker)
|
||||
# and if our speaker resumes shortly.
|
||||
# Mismatch!
|
||||
# [v1.10.19] Check if this is a real turn of another speaker.
|
||||
# If a different speaker talks for more than 0.5s, we must yield the turn.
|
||||
interruption_dur = c["timestamp"][1] - c["timestamp"][0]
|
||||
if interruption_dur > 0.5:
|
||||
break
|
||||
|
||||
# Lookahead mechanism
|
||||
resume_idx = -1
|
||||
lookahead_limit = 5 # Check next 5 chunks
|
||||
|
||||
for k in range(1, lookahead_limit + 1):
|
||||
next_idx = find_idx + k
|
||||
if next_idx >= num_chunks: break
|
||||
|
||||
nc = sorted_chunks[next_idx]
|
||||
nc_spk = normalize_spk(nc.get("speaker", "unknown"))
|
||||
nc_start = nc["timestamp"][0]
|
||||
|
||||
|
||||
# If we find our speaker again
|
||||
if nc_spk == winner_spk:
|
||||
# Check if the gap is acceptable:
|
||||
# 1. Short interruption (< 0.8s)
|
||||
# 2. OR the resume chunk starts reasonably close to the EXPECTED end of the segment.
|
||||
# (This handles cases where VAD has a large gap but Script says it should be one segment)
|
||||
last_matched_end = sorted_chunks[matched_chunks[-1]]["timestamp"][1]
|
||||
if (nc_start - last_matched_end < 0.8) or (nc_start < seg_end + 1.0):
|
||||
resume_idx = next_idx
|
||||
break
|
||||
|
||||
# If we hit a substantial chunk of another speaker, stop looking
|
||||
if nc["timestamp"][1] - nc["timestamp"][0] > 0.5:
|
||||
break
|
||||
|
||||
if resume_idx != -1:
|
||||
# Resume found! Skip intermediate chunks.
|
||||
# We do NOT add the intermediate chunks to matched_chunks (they belong to noise/others)
|
||||
# But we continue the loop from the resume point.
|
||||
|
||||
# Note: The intermediate chunks are effectively "skipped" by this segment.
|
||||
# If they were important for another segment, that segment logic needs to handle them.
|
||||
# But typically, if they are "interruptions" inside a sentence, they are noise.
|
||||
|
||||
# Advance find_idx to just before resume_idx (loop will increment)
|
||||
find_idx = resume_idx - 1
|
||||
# We will pick up the resumed chunk in next iteration
|
||||
else:
|
||||
# comprehensive stop
|
||||
break
|
||||
|
||||
find_idx += 1
|
||||
# Limit lookahead for safety (total span)
|
||||
if found_start and (find_idx - matched_chunks[0] > 50): break
|
||||
if not found_start and (find_idx - chunk_idx > 20): break
|
||||
|
||||
if matched_chunks:
|
||||
# Use min/max over all matched chunks
|
||||
actual_starts = [sorted_chunks[idx]["timestamp"][0] for idx in matched_chunks]
|
||||
actual_ends = [sorted_chunks[idx]["timestamp"][1] for idx in matched_chunks]
|
||||
|
||||
min_s = min(actual_starts)
|
||||
max_e = max(actual_ends)
|
||||
|
||||
# [v1.10.7 Fix] Handle Multi-Segment Chunks (Shared Chunk Logic)
|
||||
# If we are reusing a chunk from previous segment, we must start AFTER previous segment
|
||||
new_start = min_s
|
||||
if i > 0:
|
||||
prev_end = calibrated[-1]["end"]
|
||||
if prev_end > new_start and prev_end < max_e:
|
||||
new_start = prev_end
|
||||
|
||||
# Determine if we should consume the chunk or share it
|
||||
# Check if next segment also wants this chunk (overlaps with the tail of this chunk)
|
||||
is_shared = False
|
||||
last_matched_idx = max(matched_chunks)
|
||||
chunk_end_time = sorted_chunks[last_matched_idx]["timestamp"][1]
|
||||
|
||||
# Predicted end for this segment
|
||||
predicted_end = new_start + seg_dur
|
||||
|
||||
# Only check for sharing if there is significant leftover time in the chunk
|
||||
# AND if the next segment belongs to the same speaker (critical fix v1.10.15)
|
||||
if chunk_end_time - predicted_end > 0.5 and i + 1 < len(segments):
|
||||
next_seg = segments[i+1]
|
||||
# If next segment effectively overlaps the remainder of this chunk
|
||||
if next_seg["start"] < chunk_end_time:
|
||||
# [v1.10.15] Ensure speaker match before sharing
|
||||
current_spk = normalize_spk(seg.get("speaker"))
|
||||
next_spk = normalize_spk(next_seg.get("speaker"))
|
||||
if current_spk == next_spk:
|
||||
is_shared = True
|
||||
|
||||
if is_shared:
|
||||
# If shared, we limit our end to our duration (trust TTS relative duration)
|
||||
new_end = predicted_end
|
||||
# And we DO NOT advance past this chunk, so next segment can pick it up
|
||||
chunk_idx = last_matched_idx
|
||||
else:
|
||||
# If not shared, we consume the full VAD chunk (snap to VAD end)
|
||||
new_end = max_e
|
||||
chunk_idx = last_matched_idx + 1
|
||||
|
||||
seg["start"] = round(new_start, 3)
|
||||
seg["end"] = round(new_end, 3)
|
||||
else:
|
||||
# No match found for the specific speaker.
|
||||
# [v1.10.19] Silence-Aware Fallback:
|
||||
# Check if there is ANY speech (any speaker) in the estimated range.
|
||||
has_speech = False
|
||||
fallback_scan_idx = chunk_idx
|
||||
while fallback_scan_idx < num_chunks:
|
||||
fc = sorted_chunks[fallback_scan_idx]
|
||||
fc_start, fc_end = fc["timestamp"]
|
||||
if fc_start > seg["end"]: break # Past our range
|
||||
|
||||
# Check overlap
|
||||
overlap = min(seg["end"], fc_end) - max(seg["start"], fc_start)
|
||||
if overlap > 0.1: # Threshold for "speech exists"
|
||||
has_speech = True
|
||||
break
|
||||
|
||||
fallback_scan_idx += 1
|
||||
|
||||
start_point = calibrated[-1]["end"] if i > 0 else 0.0
|
||||
|
||||
if has_speech:
|
||||
# Someone is speaking, so we keep the text but shift it to avoid overlap
|
||||
duration = seg["end"] - seg["start"]
|
||||
if seg["start"] < start_point:
|
||||
seg["start"] = start_point
|
||||
seg["end"] = start_point + duration
|
||||
else:
|
||||
# [Silence Detected]
|
||||
# User feedback: segments_info is the "Plan". Even if silent, the dialogue must be shown to preserve order.
|
||||
# We place the subtitle in the gap (after previous segment) with its estimated duration.
|
||||
duration = seg["end"] - seg["start"]
|
||||
seg["start"] = start_point
|
||||
seg["end"] = start_point + duration
|
||||
|
||||
calibrated.append(seg)
|
||||
|
||||
return calibrated
|
||||
|
||||
def _format_ass_time(self, seconds):
|
||||
# H:MM:SS.cs (centiseconds)
|
||||
td = datetime.timedelta(seconds=seconds)
|
||||
@@ -497,128 +197,12 @@ class AIIA_Subtitle_Preview:
|
||||
|
||||
return {"ui": {"text": [subtitle_content], "audio": [audio_info] if audio_info else []}}
|
||||
|
||||
class AIIA_Subtitle_To_Segments:
|
||||
"""Convert SRT/ASS text or files into segments_info format."""
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"subtitle_text": ("STRING", {"multiline": True, "default": ""}),
|
||||
},
|
||||
"optional": {
|
||||
"subtitle_path": ("STRING", {"default": ""}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("STRING",)
|
||||
RETURN_NAMES = ("segments_info",)
|
||||
FUNCTION = "convert"
|
||||
CATEGORY = "AIIA/Subtitle"
|
||||
|
||||
def convert(self, subtitle_text, subtitle_path=""):
|
||||
import re
|
||||
content = subtitle_text.strip()
|
||||
|
||||
# If path provided and exists, read it
|
||||
if subtitle_path and os.path.exists(subtitle_path):
|
||||
try:
|
||||
with open(subtitle_path, 'r', encoding='utf-8', errors='ignore') as f:
|
||||
content = f.read().strip()
|
||||
except Exception as e:
|
||||
print(f"[AIIA Subtitle Convert] Error reading file: {e}")
|
||||
|
||||
if not content:
|
||||
return (json.dumps([]),)
|
||||
|
||||
segments = []
|
||||
|
||||
# Detect Format
|
||||
if "Dialogue:" in content:
|
||||
segments = self._parse_ass(content)
|
||||
elif " --> " in content:
|
||||
segments = self._parse_srt(content)
|
||||
else:
|
||||
print("[AIIA Subtitle Convert] Unknown format or empty content.")
|
||||
|
||||
return (json.dumps(segments, ensure_ascii=False, indent=2),)
|
||||
|
||||
def _parse_srt(self, text):
|
||||
import re
|
||||
segments = []
|
||||
# Pattern: Index, Time, Text
|
||||
# Handles \n and \r\n
|
||||
blocks = re.split(r'\n\s*\n', text.strip())
|
||||
for block in blocks:
|
||||
lines = [l.strip() for l in block.split('\n') if l.strip()]
|
||||
if len(lines) < 2: continue
|
||||
|
||||
# Find time line
|
||||
time_match = re.search(r'(\d+:\d+:\d+,\d+) --> (\d+:\d+:\d+,\d+)', lines[0] if "-->" in lines[0] else lines[1])
|
||||
if not time_match: continue
|
||||
|
||||
start_s = self._time_to_seconds(time_match.group(1), "srt")
|
||||
end_s = self._time_to_seconds(time_match.group(2), "srt")
|
||||
|
||||
# Content is everything after the time line
|
||||
idx = 1 if "-->" in lines[0] else 2
|
||||
content = " ".join(lines[idx:])
|
||||
|
||||
segments.append({
|
||||
"start": round(start_s, 3),
|
||||
"end": round(end_s, 3),
|
||||
"text": content,
|
||||
"speaker": "Unknown"
|
||||
})
|
||||
return segments
|
||||
|
||||
def _parse_ass(self, text):
|
||||
import re
|
||||
segments = []
|
||||
# Look for Dialogue: lines
|
||||
for line in text.split('\n'):
|
||||
if line.startswith("Dialogue:"):
|
||||
# Dialogue: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
|
||||
parts = line.split(',', 9)
|
||||
if len(parts) < 10: continue
|
||||
|
||||
start_s = self._time_to_seconds(parts[1].strip(), "ass")
|
||||
end_s = self._time_to_seconds(parts[2].strip(), "ass")
|
||||
speaker = parts[4].strip() or "Unknown"
|
||||
content = parts[9].strip().replace('\\N', ' ').replace('\\n', ' ')
|
||||
# Clean ASS tags like {\pos(x,y)}
|
||||
content = re.sub(r'\{.*?\}', '', content)
|
||||
|
||||
segments.append({
|
||||
"start": round(start_s, 3),
|
||||
"end": round(end_s, 3),
|
||||
"text": content,
|
||||
"speaker": speaker
|
||||
})
|
||||
return segments
|
||||
|
||||
def _time_to_seconds(self, t_str, fmt):
|
||||
try:
|
||||
if fmt == "srt":
|
||||
# HH:MM:SS,mmm
|
||||
h, m, s_ms = t_str.split(':')
|
||||
s, ms = s_ms.split(',')
|
||||
return int(h)*3600 + int(m)*60 + int(s) + int(ms)/1000.0
|
||||
else:
|
||||
# H:MM:SS.cc
|
||||
h, m, s_cs = t_str.split(':')
|
||||
s, cs = s_cs.split('.')
|
||||
return int(h)*3600 + int(m)*60 + int(s) + int(cs)/100.0
|
||||
except:
|
||||
return 0.0
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"AIIA_Subtitle_Gen": AIIA_Subtitle_Gen,
|
||||
"AIIA_Subtitle_Preview": AIIA_Subtitle_Preview,
|
||||
"AIIA_Subtitle_To_Segments": AIIA_Subtitle_To_Segments
|
||||
"AIIA_Subtitle_Preview": AIIA_Subtitle_Preview
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"AIIA_Subtitle_Gen": "AIIA Subtitle Generation",
|
||||
"AIIA_Subtitle_Preview": "AIIA Subtitle Preview",
|
||||
"AIIA_Subtitle_To_Segments": "AIIA Subtitle to Segments"
|
||||
"AIIA_Subtitle_Gen": "📝 AIIA Subtitle Generation",
|
||||
"AIIA_Subtitle_Preview": "🎬 AIIA Subtitle Preview"
|
||||
}
|
||||
|
||||
Executable → Regular
Executable → Regular
+10
-205
@@ -4,9 +4,6 @@ import torch
|
||||
import numpy as np
|
||||
import torchaudio
|
||||
import folder_paths
|
||||
import subprocess
|
||||
import tempfile
|
||||
import soundfile as sf
|
||||
# print(f"\n[AIIA DEBUG] Loaded aiia_vibevoice_nodes.py from: {os.path.abspath(__file__)}\n")
|
||||
from tqdm import tqdm
|
||||
from transformers import AutoConfig, AutoModel, AutoTokenizer, Qwen2TokenizerFast
|
||||
@@ -205,7 +202,7 @@ class AIIA_VibeVoice_TTS:
|
||||
"required": {
|
||||
"vibevoice_model": ("VIBEVOICE_MODEL",),
|
||||
"text": ("STRING", {"multiline": True, "default": "Hello, this is a test of VibeVoice."}),
|
||||
# "reference_audio": ("AUDIO",), <-- Moved to optional
|
||||
"reference_audio": ("AUDIO",),
|
||||
"cfg_scale": ("FLOAT", {"default": 1.3, "min": 1.0, "max": 10.0, "step": 0.1}),
|
||||
"ddpm_steps": ("INT", {"default": 20, "min": 10, "max": 100, "step": 1}),
|
||||
"speed": ("FLOAT", {"default": 1.0, "min": 0.5, "max": 2.0, "step": 0.1}),
|
||||
@@ -214,9 +211,6 @@ class AIIA_VibeVoice_TTS:
|
||||
"temperature": ("FLOAT", {"default": 0.8, "min": 0.1, "max": 2.0}),
|
||||
"top_k": ("INT", {"default": 20, "min": 0, "max": 100}),
|
||||
"top_p": ("FLOAT", {"default": 0.95, "min": 0.0, "max": 1.0}),
|
||||
},
|
||||
"optional": {
|
||||
"reference_audio": ("AUDIO",),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -225,95 +219,12 @@ class AIIA_VibeVoice_TTS:
|
||||
FUNCTION = "generate"
|
||||
CATEGORY = "AIIA/VibeVoice"
|
||||
|
||||
def _load_fallback_audio(self, target_name="Female_HQ"):
|
||||
import torchaudio
|
||||
# 定位 assets 目录 (Shared with Podcast nodes)
|
||||
nodes_path = os.path.dirname(os.path.abspath(__file__))
|
||||
assets_dir = os.path.join(nodes_path, "assets")
|
||||
|
||||
# Consistent mapping with Podcast node
|
||||
filename_map = {
|
||||
"Female_HQ": "seed_female_hq.wav",
|
||||
"Male_HQ": "seed_male_hq.wav",
|
||||
"Female": "seed_female.wav",
|
||||
"Male": "seed_male.wav"
|
||||
}
|
||||
|
||||
filename = filename_map.get(target_name, "seed_female_hq.wav")
|
||||
path = os.path.join(assets_dir, filename)
|
||||
|
||||
if not os.path.exists(path):
|
||||
print(f"[AIIA Warning] Fallback seed not found at {path}")
|
||||
return None
|
||||
|
||||
try:
|
||||
waveform, sample_rate = torchaudio.load(path)
|
||||
if waveform.shape[0] > 1:
|
||||
waveform = torch.mean(waveform, dim=0, keepdim=True)
|
||||
waveform = waveform * 0.8 # Attenuate
|
||||
return {"waveform": waveform, "sample_rate": sample_rate}
|
||||
except Exception as e:
|
||||
print(f"[AIIA Error] Failed to load fallback audio: {e}")
|
||||
return None
|
||||
|
||||
def _normalize_roles(self, text):
|
||||
"""
|
||||
Detects custom roles (e.g. 'Host A:', 'User:') and normalizes them to 'Speaker N:'.
|
||||
Returns: (normalized_text, role_mapping)
|
||||
"""
|
||||
import re
|
||||
lines = text.split('\n')
|
||||
# Matches "Role Name:" at start of line.
|
||||
# Excludes "Speaker N:" which is already valid.
|
||||
# Limit role name to 30 chars to avoid matching long sentences.
|
||||
role_pattern = re.compile(r'^([^\n:]{1,30}):\s+')
|
||||
speaker_pattern = re.compile(r'^Speaker\s*\d+', re.IGNORECASE)
|
||||
|
||||
roles_map = {}
|
||||
next_id = 1
|
||||
normalized_lines = []
|
||||
|
||||
for line in lines:
|
||||
stripped = line.strip()
|
||||
if not stripped:
|
||||
normalized_lines.append(line)
|
||||
continue
|
||||
|
||||
match = role_pattern.match(stripped)
|
||||
if match:
|
||||
role_name = match.group(1).strip()
|
||||
|
||||
# If already standard format, keep it
|
||||
if speaker_pattern.match(role_name):
|
||||
normalized_lines.append(line)
|
||||
continue
|
||||
|
||||
# Map custom role
|
||||
if role_name not in roles_map:
|
||||
roles_map[role_name] = next_id
|
||||
next_id += 1
|
||||
|
||||
spk_id = roles_map[role_name]
|
||||
# Replace prefix with Speaker N
|
||||
# We reconstruct the line to ensure standard formatting
|
||||
content = stripped[match.end():]
|
||||
normalized_lines.append(f"Speaker {spk_id}: {content}")
|
||||
else:
|
||||
normalized_lines.append(line)
|
||||
|
||||
return "\n".join(normalized_lines), roles_map
|
||||
|
||||
def generate(self, vibevoice_model, text, cfg_scale, ddpm_steps, speed, normalize_text,
|
||||
do_sample, temperature, top_k, top_p, reference_audio=None):
|
||||
def generate(self, vibevoice_model, text, reference_audio, cfg_scale, ddpm_steps, speed, normalize_text,
|
||||
do_sample, temperature, top_k, top_p):
|
||||
model = vibevoice_model["model"]
|
||||
tokenizer = vibevoice_model["tokenizer"]
|
||||
processor = vibevoice_model.get("processor")
|
||||
is_streaming = vibevoice_model.get("is_streaming", False)
|
||||
|
||||
# AIIA Fix: Ensure model is on GPU (it might have been offloaded to CPU by previous run)
|
||||
if torch.cuda.is_available():
|
||||
model.to("cuda")
|
||||
|
||||
device = model.device
|
||||
|
||||
if processor is None: raise RuntimeError("Processor is missing.")
|
||||
@@ -330,39 +241,12 @@ class AIIA_VibeVoice_TTS:
|
||||
text = re.sub(r'(\d+年)\s*[-—–]\s*(\d+年)', r'\1至\2', text)
|
||||
text = text.replace('"', '').replace("'", '')
|
||||
|
||||
# [AIIA v1.10.8] Auto-Normalize Roles (e.g. "Host A:" -> "Speaker 1:")
|
||||
text, role_map = self._normalize_roles(text)
|
||||
num_roles = len(role_map) if role_map else 0
|
||||
|
||||
# Determine unique speakers count from text if no role map (e.g. manual Speaker 1, Speaker 2)
|
||||
if not role_map:
|
||||
# Basic regex count of unique "Speaker N"
|
||||
spk_ids = set(re.findall(r'^Speaker\s+(\d+):', text, re.MULTILINE))
|
||||
num_roles = len(spk_ids) if spk_ids else 1
|
||||
# Default Speaker Tag
|
||||
if not re.search(r'^Speaker\s+\d+\s*:', text, re.IGNORECASE | re.MULTILINE):
|
||||
lines = text.split('\n')
|
||||
text = "\n".join([f"Speaker 1: {line.strip()}" for line in lines if line.strip()])
|
||||
|
||||
# Process Reference Audio
|
||||
if reference_audio is None:
|
||||
print(f"[AIIA INFO] No reference audio provided. Auto-loading fallbacks for {num_roles} speakers...")
|
||||
reference_audio = []
|
||||
|
||||
# Simple alternating strategy
|
||||
# Speaker 1 (or Host A) -> Female HQ
|
||||
# Speaker 2 (or Host B) -> Male HQ
|
||||
# Speaker 3 -> Female
|
||||
# Speaker 4 -> Male
|
||||
patterns = ["Female_HQ", "Male_HQ", "Female", "Male"]
|
||||
|
||||
for i in range(max(num_roles, 1)):
|
||||
target = patterns[i % len(patterns)]
|
||||
fb = self._load_fallback_audio(target)
|
||||
if fb: reference_audio.append(fb)
|
||||
else:
|
||||
# Should not happen if assets exist, but fallback to anything
|
||||
if reference_audio: reference_audio.append(reference_audio[0])
|
||||
|
||||
if not reference_audio:
|
||||
raise ValueError("Could not load any fallback audio!")
|
||||
|
||||
voice_samples = []
|
||||
|
||||
# Determine if input is list or single item
|
||||
@@ -370,19 +254,6 @@ class AIIA_VibeVoice_TTS:
|
||||
raw_refs = reference_audio
|
||||
else:
|
||||
raw_refs = [reference_audio]
|
||||
|
||||
# [AIIA v1.10.9] Smart Pad: If we have more roles than refs, recycle or pad?
|
||||
# If user provided 1 ref but text has 2 speakers, previous behavior: Speaker 2 gets nothing?
|
||||
# VibeVoice processor slices refs[:num_speakers].
|
||||
# If we pad, we can give Speaker 2 the SAME voice, or a fallback?
|
||||
# Usually if user gives 1 ref, they might want cloning for Speaker 1, but what for Speaker 2?
|
||||
# Safest is to Repeat, or maybe Fallback?
|
||||
# Let's Repeat the last ref to avoid errors, assuming 'Cloning' context.
|
||||
# But for distinct roles, users SHOULD provide distinct audios.
|
||||
if len(raw_refs) < num_roles:
|
||||
print(f"[AIIA Warning] Text has {num_roles} roles but only {len(raw_refs)} reference audios. Recycling last audio for remaining speakers.")
|
||||
while len(raw_refs) < num_roles:
|
||||
raw_refs.append(raw_refs[-1])
|
||||
|
||||
for ref_item in raw_refs:
|
||||
if ref_item is None:
|
||||
@@ -453,67 +324,9 @@ class AIIA_VibeVoice_TTS:
|
||||
if audio_out.ndim == 1: audio_out = audio_out.unsqueeze(0)
|
||||
if audio_out.ndim == 3: audio_out = audio_out.squeeze(0)
|
||||
|
||||
# Ensure float32 (Vital for downstream nodes like Resemble Enhance which fail on fp16)
|
||||
audio_out = audio_out.float()
|
||||
|
||||
# Speed adj
|
||||
if speed != 1.0:
|
||||
original_device = audio_out.device
|
||||
|
||||
# Try System 'sox' command for time stretching (pitch preservation)
|
||||
# This is more robust than torchaudio.sox_effects which may be missing in some builds
|
||||
try:
|
||||
# Prepare input
|
||||
audio_cpu = audio_out.cpu().numpy()
|
||||
# Ensure [C, T]
|
||||
if audio_cpu.ndim == 1: audio_cpu = audio_cpu[np.newaxis, :] # [1, T]
|
||||
elif audio_cpu.ndim == 3: audio_cpu = audio_cpu.squeeze(0) # [C, T]
|
||||
|
||||
# soundfile writes [T, C]
|
||||
audio_cpu_t = audio_cpu.T
|
||||
|
||||
# Normalize to -1dB (approx 0.9) to prevent clipping during sox processing
|
||||
# Sox 'tempo' effect can increase peak amplitude
|
||||
max_val = np.abs(audio_cpu_t).max()
|
||||
if max_val > 0.9:
|
||||
audio_cpu_t = audio_cpu_t * (0.9 / max_val)
|
||||
|
||||
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as in_f, \
|
||||
tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as out_f:
|
||||
in_path = in_f.name
|
||||
out_path = out_f.name
|
||||
|
||||
try:
|
||||
# Write temp file
|
||||
sf.write(in_path, audio_cpu_t, 24000)
|
||||
|
||||
# Call sox
|
||||
# tempo command: changes speed without pitch
|
||||
# -q: quiet
|
||||
# -s: use sequence search (better quality for speech)
|
||||
cmd = ["sox", "-q", in_path, out_path, "tempo", "-s", str(speed)]
|
||||
subprocess.run(cmd, check=True)
|
||||
print(f"[AIIA] Applied time stretch: speed={speed}x (pitch preserved)")
|
||||
|
||||
# Read back
|
||||
out_wav, out_sr = sf.read(out_path)
|
||||
# sf reads as [T, C] or [T] if mono
|
||||
if out_wav.ndim == 1:
|
||||
out_wav = out_wav[np.newaxis, :] # [1, T]
|
||||
else:
|
||||
out_wav = out_wav.T # [C, T]
|
||||
|
||||
audio_out = torch.from_numpy(out_wav).float().to(original_device)
|
||||
|
||||
finally:
|
||||
if os.path.exists(in_path): os.remove(in_path)
|
||||
if os.path.exists(out_path): os.remove(out_path)
|
||||
|
||||
except Exception as e:
|
||||
# Fallback to Resample (Pitch Shift)
|
||||
print(f"[AIIA WARNING] System 'sox' failed ({e}), using Resample (Pitch Shift).")
|
||||
resampler = torchaudio.transforms.Resample(orig_freq=int(24000*speed), new_freq=24000).to(original_device)
|
||||
audio_out = resampler(audio_out)
|
||||
audio_out = torchaudio.transforms.Resample(orig_freq=int(24000*speed), new_freq=24000)(audio_out)
|
||||
|
||||
if audio_out.ndim == 2: audio_out = audio_out.unsqueeze(0)
|
||||
|
||||
@@ -553,14 +366,6 @@ class AIIA_VibeVoice_TTS:
|
||||
except Exception as e:
|
||||
print(f"[AIIA WARNING] Failed to trim audio: {e}")
|
||||
|
||||
# Cleanup: Move model back to CPU to release VRAM
|
||||
try:
|
||||
model.to("cpu")
|
||||
if hasattr(model, "model") and hasattr(model.model, "to"):
|
||||
model.model.to("cpu")
|
||||
torch.cuda.empty_cache()
|
||||
except: pass
|
||||
|
||||
return ({"waveform": audio_out.cpu(), "sample_rate": 24000},)
|
||||
|
||||
except Exception as e:
|
||||
@@ -575,7 +380,7 @@ NODE_CLASS_MAPPINGS = {
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"AIIA_VibeVoice_Loader": "VibeVoice Loader",
|
||||
"AIIA_VibeVoice_TTS": "VibeVoice TTS (Standard)"
|
||||
"AIIA_VibeVoice_Loader": "🎤 VibeVoice Loader",
|
||||
"AIIA_VibeVoice_TTS": "🗣️ VibeVoice TTS (Standard)"
|
||||
}
|
||||
|
||||
|
||||
Executable → Regular
Executable → Regular
Executable → Regular
+8
-386
@@ -177,7 +177,7 @@ def aiia_apply_video_format_config(format_ui_name: str, user_inputs: dict) -> di
|
||||
return processed
|
||||
|
||||
class AIIA_VideoCombine:
|
||||
NODE_NAME = "AIIA Video Combine (Images or Directory)"
|
||||
NODE_NAME = "AIIA 视频合并 (图像或目录)"
|
||||
CATEGORY = "AIIA/视频"
|
||||
FUNCTION = "combine_video"
|
||||
RETURN_TYPES = ("STRING",)
|
||||
@@ -209,7 +209,7 @@ class AIIA_VideoCombine:
|
||||
},
|
||||
"optional": {
|
||||
"images": ("IMAGE",), "frames_directory": ("STRING", {"default": ""}),
|
||||
"filename_pattern": ("STRING", {"default": "frame_%08d.png"}),
|
||||
"filename_pattern": ("STRING", {"default": "frame_%06d.png"}),
|
||||
"audio_tensor": ("AUDIO",), "audio_file_path": ("STRING", {"default": ""}),
|
||||
# 【逻辑修改】将 'auto' 设为默认值
|
||||
"audio_codec": (audio_codec_options, {"default": "auto"}),
|
||||
@@ -233,88 +233,17 @@ class AIIA_VideoCombine:
|
||||
try:
|
||||
effective_frames_dir, effective_filename_pattern = None, filename_pattern
|
||||
if images is not None:
|
||||
num_frames = images.shape[0]
|
||||
logger.info(f"检测到 {num_frames} 帧的图像张量输入...")
|
||||
from tqdm import tqdm
|
||||
import gc
|
||||
logger.info(f"检测到 {images.shape[0]} 帧的图像张量输入...")
|
||||
temp_image_dir_to_delete = tempfile.mkdtemp(prefix="aiia_frames_")
|
||||
effective_frames_dir, effective_filename_pattern = temp_image_dir_to_delete, "frame_%08d.png"
|
||||
pbar = ProgressBar(num_frames)
|
||||
# Direct tqdm to stdout for console logs
|
||||
console_pbar = tqdm(total=num_frames, desc="[AIIA Video] Saving Frames", unit="frame", file=sys.stdout)
|
||||
|
||||
# Move to CPU if on GPU to free GPU memory
|
||||
if images.device.type == 'cuda':
|
||||
images = images.cpu()
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
# Convert to list so we can delete individual frames to free memory
|
||||
# This is crucial for large frame counts
|
||||
frame_list = [images[i] for i in range(num_frames)]
|
||||
del images # Release the original tensor immediately
|
||||
gc.collect()
|
||||
|
||||
for i in range(num_frames):
|
||||
# Get frame, convert, save, and release immediately
|
||||
frame_np = (frame_list[i].numpy() * 255).astype(np.uint8)
|
||||
frame_list[i] = None # Release this frame from the list
|
||||
img = Image.fromarray(frame_np)
|
||||
img.save(os.path.join(effective_frames_dir, effective_filename_pattern % (i + 1)))
|
||||
del frame_np, img
|
||||
pbar = ProgressBar(images.shape[0])
|
||||
for i, frame_tensor in enumerate(images):
|
||||
Image.fromarray((frame_tensor.cpu().numpy() * 255).astype(np.uint8)).save(os.path.join(effective_frames_dir, effective_filename_pattern % (i + 1)))
|
||||
pbar.update(1)
|
||||
console_pbar.update(1)
|
||||
|
||||
# Periodic garbage collection every 50 frames
|
||||
if i > 0 and i % 50 == 0:
|
||||
gc.collect()
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
console_pbar.close()
|
||||
|
||||
# Final cleanup
|
||||
del frame_list
|
||||
gc.collect()
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
elif frames_directory:
|
||||
effective_frames_dir = strip_path_aiia(frames_directory)
|
||||
if not validate_path_aiia(effective_frames_dir, check_is_dir=True):
|
||||
raise ValueError(f"帧目录验证失败: {effective_frames_dir}")
|
||||
|
||||
# Intelligent Pattern Detection (Fix for v1.9.27 compatibility)
|
||||
# If user workflow has old default (%06d) but files are new (%08d), auto-correct it.
|
||||
try:
|
||||
# Look for frame_*.png files
|
||||
search_glob = os.path.join(effective_frames_dir, "frame_*.png")
|
||||
sample_files = sorted(glob.glob(search_glob))
|
||||
|
||||
# Log detection status
|
||||
if not sample_files:
|
||||
logger.warning(f"[AIIA_VideoCombine] No 'frame_*.png' files found in {effective_frames_dir}. Detection skipped.")
|
||||
else:
|
||||
first_file = os.path.basename(sample_files[0])
|
||||
logger.info(f"[AIIA_VideoCombine] Auto-detecting pattern from first file: {first_file}")
|
||||
|
||||
# Extract the numeric part: frame_00000123.png -> 00000123
|
||||
match = re.search(r"frame_(\d+)\.png", first_file)
|
||||
if match:
|
||||
digit_count = len(match.group(1))
|
||||
auto_pattern = f"frame_%0{digit_count}d.png"
|
||||
|
||||
if auto_pattern != effective_filename_pattern:
|
||||
logger.warning(f"[AIIA_VideoCombine] Pattern Mismatch! User: '{effective_filename_pattern}', Actual: '{auto_pattern}'. Auto-correcting.")
|
||||
effective_filename_pattern = auto_pattern
|
||||
|
||||
# Verification: Check if ffmpeg will find it
|
||||
test_glob = os.path.join(effective_frames_dir, f"frame_{'0'*digit_count}.png")
|
||||
if not os.path.exists(test_glob) and len(sample_files) > 0:
|
||||
# Sometimes start index is not 0?
|
||||
logger.warning(f"[AIIA_VideoCombine] Verify: frame_{'0'*digit_count}.png not found, but we found {len(sample_files)} files.")
|
||||
else:
|
||||
logger.warning(f"[AIIA_VideoCombine] Regex failed on {first_file}")
|
||||
except Exception as e:
|
||||
logger.warning(f"[AIIA_VideoCombine] Auto-pattern detection failed: {e}")
|
||||
else: raise ValueError("错误: 必须提供 'images' 或 'frames_directory' 输入。")
|
||||
|
||||
output_dir = folder_paths.get_output_directory() if save_output else folder_paths.get_temp_directory()
|
||||
@@ -394,312 +323,5 @@ class AIIA_VideoCombine:
|
||||
try: shutil.rmtree(temp_image_dir_to_delete); logger.info(f"已清理临时图像帧目录: {temp_image_dir_to_delete}")
|
||||
except Exception as e_del: logger.error(f"清理临时图像帧目录失败: {e_del}")
|
||||
|
||||
|
||||
class AIIA_BodySway:
|
||||
"""Simulate subtle body movement through crop-based pan and rotation."""
|
||||
|
||||
NODE_NAME = "AIIA Body Sway"
|
||||
CATEGORY = "AIIA/视频"
|
||||
FUNCTION = "apply_sway"
|
||||
RETURN_TYPES = ("IMAGE", "STRING")
|
||||
RETURN_NAMES = ("images", "output_frames_dir")
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"crop_ratio": ("FLOAT", {"default": 0.99, "min": 0.90, "max": 1.0, "step": 0.001,
|
||||
"tooltip": "Output size as ratio of input (0.99 = keep 99%, crop 1%)"}),
|
||||
"rotation_amplitude": ("FLOAT", {"default": 0.1, "min": 0.0, "max": 2.0, "step": 0.1,
|
||||
"tooltip": "Max rotation in degrees"}),
|
||||
"smoothness": ("FLOAT", {"default": 0.02, "min": 0.005, "max": 0.1, "step": 0.005,
|
||||
"tooltip": "Perlin noise smoothness (smaller = slower drift)"}),
|
||||
"seed": ("INT", {"default": 0, "min": 0, "max": 0xffffffffffffffff}),
|
||||
},
|
||||
"optional": {
|
||||
"images": ("IMAGE",), # [B, H, W, C] tensor
|
||||
"frames_directory": ("STRING", {"default": "", "tooltip": "Path to directory containing frames (from Ditto disk mode)"}),
|
||||
}
|
||||
}
|
||||
|
||||
def apply_sway(self, crop_ratio: float, rotation_amplitude: float,
|
||||
smoothness: float, seed: int, images: torch.Tensor = None, frames_directory: str = ""):
|
||||
"""
|
||||
Apply body sway effect using crop + rotation with Perlin noise.
|
||||
|
||||
images: [B, H, W, C] tensor (float32, 0-1 range) - optional
|
||||
frames_directory: Path to directory containing frames - optional (for OOM-safe mode)
|
||||
crop_ratio: Output size as ratio of input (e.g., 0.99)
|
||||
smoothness: Perlin noise smoothness (smaller = slower drift)
|
||||
"""
|
||||
import math
|
||||
from PIL import Image
|
||||
import random
|
||||
import os
|
||||
import glob
|
||||
import tempfile
|
||||
import gc
|
||||
|
||||
# Validate inputs
|
||||
frames_directory = frames_directory.strip() if frames_directory else ""
|
||||
use_disk_mode = bool(frames_directory and os.path.isdir(frames_directory))
|
||||
|
||||
if images is None and not use_disk_mode:
|
||||
raise ValueError("必须提供 images 或 frames_directory 输入")
|
||||
|
||||
# Limit seed to 32-bit for numpy compatibility
|
||||
seed_32 = seed % (2**32)
|
||||
random.seed(seed_32)
|
||||
np.random.seed(seed_32)
|
||||
|
||||
# Determine input dimensions and frame count
|
||||
if use_disk_mode:
|
||||
# Get frame list from directory
|
||||
frame_files = sorted(glob.glob(os.path.join(frames_directory, "*.png")))
|
||||
if not frame_files:
|
||||
frame_files = sorted(glob.glob(os.path.join(frames_directory, "*.jpg")))
|
||||
if not frame_files:
|
||||
raise ValueError(f"目录中未找到帧文件: {frames_directory}")
|
||||
|
||||
batch_size = len(frame_files)
|
||||
first_frame = Image.open(frame_files[0])
|
||||
in_w, in_h = first_frame.size
|
||||
channels = 3
|
||||
first_frame.close()
|
||||
logger.info(f"[BodySway] Disk mode: {batch_size} frames from {frames_directory}")
|
||||
else:
|
||||
batch_size, in_h, in_w, channels = images.shape
|
||||
|
||||
# Auto-calculate target size and sway amplitude from crop_ratio
|
||||
target_width = int(in_w * crop_ratio)
|
||||
target_height = int(in_h * crop_ratio)
|
||||
|
||||
# Ensure target is even (for compatibility with video encoders)
|
||||
target_width = target_width - (target_width % 2)
|
||||
target_height = target_height - (target_height % 2)
|
||||
|
||||
# Calculate total available margin
|
||||
margin_x = (in_w - target_width) / 2
|
||||
margin_y = (in_h - target_height) / 2
|
||||
total_margin = min(margin_x, margin_y)
|
||||
|
||||
# Reserve margin for rotation (rotation introduces black corners)
|
||||
diagonal = math.sqrt(in_w**2 + in_h**2)
|
||||
rotation_margin = (diagonal / 2) * math.sin(math.radians(rotation_amplitude)) if rotation_amplitude > 0 else 0
|
||||
|
||||
# Safe margin for sway = total margin - rotation margin - 20% safety buffer
|
||||
safe_margin = max(0, total_margin - rotation_margin)
|
||||
sway_amplitude = safe_margin * 0.8 # Use 80% of remaining safe margin
|
||||
|
||||
# Clamp rotation to prevent exceeding available margin
|
||||
max_safe_rotation = math.degrees(math.asin(min(1.0, total_margin * 0.5 / (diagonal / 2 + 0.001))))
|
||||
actual_rotation = min(rotation_amplitude, max_safe_rotation)
|
||||
if actual_rotation < rotation_amplitude:
|
||||
logger.warning(f"[BodySway] Rotation clamped from {rotation_amplitude}° to {actual_rotation:.2f}° to prevent black corners")
|
||||
|
||||
logger.info(f"[BodySway] Input: {in_w}x{in_h}, Output: {target_width}x{target_height}, "
|
||||
f"Margin: {total_margin:.1f}px, RotMargin: {rotation_margin:.1f}px, Sway: {sway_amplitude:.1f}px")
|
||||
|
||||
# Generate Perlin-like noise trajectory (1D)
|
||||
def generate_perlin_trajectory(n_frames: int, amplitude: float, scale: float):
|
||||
"""Generate smooth organic trajectory using 1D Perlin-like noise."""
|
||||
# Use cumulative random walk with smoothing
|
||||
trajectory = np.zeros(n_frames)
|
||||
|
||||
# Generate multi-octave noise
|
||||
octaves = 4
|
||||
persistence = 0.5
|
||||
|
||||
for octave in range(octaves):
|
||||
freq = scale * (2 ** octave)
|
||||
amp = amplitude * (persistence ** octave)
|
||||
phase = random.random() * 1000
|
||||
|
||||
for t in range(n_frames):
|
||||
# Smooth interpolated noise using sine-based sampling
|
||||
x = t * freq + phase
|
||||
# Interpolate between random values
|
||||
i = int(x)
|
||||
f = x - i
|
||||
# Smooth interpolation (cosine)
|
||||
f = (1 - math.cos(f * math.pi)) / 2
|
||||
|
||||
# Get random values seeded by position
|
||||
random.seed(seed_32 + i + octave * 10000)
|
||||
v0 = random.random() * 2 - 1
|
||||
random.seed(seed_32 + i + 1 + octave * 10000)
|
||||
v1 = random.random() * 2 - 1
|
||||
|
||||
trajectory[t] += (v0 * (1 - f) + v1 * f) * amp
|
||||
|
||||
# Normalize to amplitude range
|
||||
if np.max(np.abs(trajectory)) > 0:
|
||||
trajectory = trajectory / np.max(np.abs(trajectory)) * amplitude
|
||||
|
||||
return trajectory
|
||||
|
||||
# Generate X and Rotation trajectories (NO Y translation to reduce dizziness)
|
||||
traj_x = generate_perlin_trajectory(batch_size, sway_amplitude, smoothness)
|
||||
traj_y = np.zeros(batch_size) # No vertical movement
|
||||
traj_rot = generate_perlin_trajectory(batch_size, actual_rotation, smoothness * 0.7)
|
||||
|
||||
# Calculate GLOBAL valid region based on MAX rotation angle
|
||||
if actual_rotation > 0:
|
||||
max_rot_rad = math.radians(actual_rotation)
|
||||
global_rot_margin_x = int(math.ceil((in_h * math.sin(max_rot_rad) + in_w * (1 - math.cos(max_rot_rad))) / 2))
|
||||
global_rot_margin_y = int(math.ceil((in_w * math.sin(max_rot_rad) + in_h * (1 - math.cos(max_rot_rad))) / 2))
|
||||
valid_left = global_rot_margin_x
|
||||
valid_top = global_rot_margin_y
|
||||
valid_right = in_w - global_rot_margin_x
|
||||
valid_bottom = in_h - global_rot_margin_y
|
||||
logger.info(f"[BodySway] Global valid region: ({valid_left}, {valid_top}) to ({valid_right}, {valid_bottom})")
|
||||
else:
|
||||
valid_left, valid_top = 0, 0
|
||||
valid_right, valid_bottom = in_w, in_h
|
||||
|
||||
# Check if target fits within valid region
|
||||
if target_width > (valid_right - valid_left) or target_height > (valid_bottom - valid_top):
|
||||
logger.warning(f"[BodySway] Target size ({target_width}x{target_height}) exceeds valid region. Consider reducing rotation_amplitude or crop_ratio.")
|
||||
|
||||
# Create output directory for disk mode
|
||||
output_frames_dir = ""
|
||||
if use_disk_mode:
|
||||
import folder_paths
|
||||
output_frames_dir = tempfile.mkdtemp(prefix="bodysway_frames_", dir=folder_paths.get_temp_directory())
|
||||
logger.info(f"[BodySway] Output frames will be saved to: {output_frames_dir}")
|
||||
|
||||
# Process frames: disk mode or memory mode
|
||||
if use_disk_mode:
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import threading
|
||||
|
||||
# Disk mode: parallel processing and saving
|
||||
# Use 4 workers to balance CPU usage (cropping/rotating is fast, saving is slow)
|
||||
process_executor = ThreadPoolExecutor(max_workers=4)
|
||||
write_sem = threading.Semaphore(50) # Limit pending writes
|
||||
|
||||
def process_and_save_task(idx, in_path, out_path, sem):
|
||||
try:
|
||||
img = Image.open(in_path)
|
||||
|
||||
# Apply rotation
|
||||
if actual_rotation > 0:
|
||||
rotated = img.rotate(traj_rot[idx], resample=Image.BILINEAR, expand=False)
|
||||
img.close()
|
||||
img = rotated
|
||||
|
||||
# Calculate crop box
|
||||
center_x = in_w / 2 + traj_x[idx]
|
||||
center_y = in_h / 2 + traj_y[idx] # traj_y is actually 0 but for consistency
|
||||
|
||||
left = int(center_x - target_width / 2)
|
||||
top = int(center_y - target_height / 2)
|
||||
right = left + target_width
|
||||
bottom = top + target_height
|
||||
|
||||
# Clamp
|
||||
left = max(valid_left, min(left, valid_right - target_width))
|
||||
top = max(valid_top, min(top, valid_bottom - target_height))
|
||||
right = left + target_width
|
||||
bottom = top + target_height
|
||||
|
||||
# Crop and save
|
||||
cropped = img.crop((left, top, right, bottom))
|
||||
img.close()
|
||||
|
||||
# Fast save
|
||||
cropped.save(out_path, format="PNG", compress_level=0)
|
||||
cropped.close()
|
||||
except Exception as e:
|
||||
logger.error(f"[BodySway] Error processing frame {idx}: {e}")
|
||||
finally:
|
||||
sem.release()
|
||||
|
||||
logger.info("[BodySway] Starting parallel disk processing...")
|
||||
|
||||
for i, frame_path in enumerate(frame_files):
|
||||
output_path = os.path.join(output_frames_dir, f"frame_{i:08d}.png")
|
||||
|
||||
# Wait for slot
|
||||
write_sem.acquire()
|
||||
|
||||
# Submit task
|
||||
process_executor.submit(process_and_save_task, i, frame_path, output_path, write_sem)
|
||||
|
||||
# Periodic logging
|
||||
if i > 0 and i % 100 == 0:
|
||||
if batch_size > 200:
|
||||
logger.info(f"[BodySway] Submitted: {i}/{batch_size} frames")
|
||||
|
||||
# Wait for all tasks to complete
|
||||
process_executor.shutdown(wait=True)
|
||||
|
||||
# Return placeholder tensor for disk mode
|
||||
output_tensor = torch.zeros((1, 1, 1, 3), dtype=torch.float32)
|
||||
logger.info(f"[BodySway] Disk mode: Processed {batch_size} frames, saved to {output_frames_dir}")
|
||||
|
||||
else:
|
||||
# Memory mode: original logic with tensor input/output
|
||||
output_tensor = torch.zeros((batch_size, target_height, target_width, channels),
|
||||
dtype=torch.float32, device='cpu')
|
||||
|
||||
if images.device.type == 'cuda':
|
||||
images = images.cpu()
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
batch_chunk_size = 50
|
||||
|
||||
for batch_start in range(0, batch_size, batch_chunk_size):
|
||||
batch_end = min(batch_start + batch_chunk_size, batch_size)
|
||||
|
||||
for i in range(batch_start, batch_end):
|
||||
frame_np = (images[i].numpy() * 255).astype(np.uint8)
|
||||
img = Image.fromarray(frame_np)
|
||||
del frame_np
|
||||
|
||||
if actual_rotation > 0:
|
||||
rotated = img.rotate(traj_rot[i], resample=Image.BILINEAR, expand=False)
|
||||
del img
|
||||
img = rotated
|
||||
|
||||
center_x = in_w / 2 + traj_x[i]
|
||||
center_y = in_h / 2 + traj_y[i]
|
||||
|
||||
left = int(center_x - target_width / 2)
|
||||
top = int(center_y - target_height / 2)
|
||||
right = left + target_width
|
||||
bottom = top + target_height
|
||||
|
||||
left = max(valid_left, min(left, valid_right - target_width))
|
||||
top = max(valid_top, min(top, valid_bottom - target_height))
|
||||
right = left + target_width
|
||||
bottom = top + target_height
|
||||
|
||||
cropped = img.crop((left, top, right, bottom))
|
||||
del img
|
||||
|
||||
cropped_arr = np.array(cropped, dtype=np.float32)
|
||||
del cropped
|
||||
output_tensor[i] = torch.from_numpy(cropped_arr / 255.0)
|
||||
del cropped_arr
|
||||
|
||||
gc.collect()
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
if batch_size > 200:
|
||||
logger.info(f"[BodySway] Progress: {batch_end}/{batch_size} frames processed")
|
||||
|
||||
logger.info(f"[BodySway] Processed {batch_size} frames with Perlin noise (smoothness={smoothness})")
|
||||
|
||||
return (output_tensor, output_frames_dir)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"AIIA_VideoCombine": AIIA_VideoCombine,
|
||||
"AIIA_BodySway": AIIA_BodySway,
|
||||
}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"AIIA_VideoCombine": "AIIA Video Combine (Images or Dir)",
|
||||
"AIIA_BodySway": "AIIA Body Sway",
|
||||
}
|
||||
NODE_CLASS_MAPPINGS = { "AIIA_VideoCombine": AIIA_VideoCombine }
|
||||
NODE_DISPLAY_NAME_MAPPINGS = { "AIIA_VideoCombine": "视频合并 (AIIA, 图像或目录)" }
|
||||
Executable → Regular
+2
-2
@@ -254,6 +254,6 @@ NODE_CLASS_MAPPINGS = {
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"AIIA_VoxCPM_Loader": "VoxCPM Loader",
|
||||
"AIIA_VoxCPM_TTS": "VoxCPM 1.5 TTS"
|
||||
"AIIA_VoxCPM_Loader": "🎤 VoxCPM Loader",
|
||||
"AIIA_VoxCPM_TTS": "🗣️ VoxCPM 1.5 TTS"
|
||||
}
|
||||
|
||||
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
+4
-69
@@ -77,8 +77,6 @@ class AIIABrowserDialog extends ComfyDialog {
|
||||
this.applyFocusRaf = null;
|
||||
this.iconLoadTimers = new Map();
|
||||
|
||||
this.outsideClickListener = null;
|
||||
|
||||
this.tooltipImage = $el("img.aiia-tooltip-image");
|
||||
this.tooltipVideo = $el("video.aiia-tooltip-video", { autoplay: true, muted: true, loop: true, controls: false, volume: 0.8 });
|
||||
this.tooltipAudio = $el("audio", { autoplay: true });
|
||||
@@ -93,10 +91,7 @@ class AIIABrowserDialog extends ComfyDialog {
|
||||
|
||||
const splitViewContainer = $el("div.aiia-split-view-container", [this.directoryPanel, this.contentPanel, this.tooltipElement]);
|
||||
|
||||
this.iconViewObserver = new IntersectionObserver(this.handleIconIntersection.bind(this), {
|
||||
root: null, // Use root: null to observe relative to browser viewport
|
||||
rootMargin: '500px 0px 500px 0px'
|
||||
});
|
||||
this.iconViewObserver = new IntersectionObserver(this.handleIconIntersection.bind(this), { root: this.contentArea, rootMargin: '300px 0px 300px 0px' });
|
||||
|
||||
this.titleElement = $el("span", { textContent: "AIIA Media Browser" });
|
||||
this.closeButton = $el("button.close", { textContent: "✖", title: "Close" });
|
||||
@@ -303,25 +298,6 @@ class AIIABrowserDialog extends ComfyDialog {
|
||||
}
|
||||
}
|
||||
}
|
||||
async deleteItem(itemName) {
|
||||
if (!itemName) return;
|
||||
if (!confirm(`Are you sure you want to delete "${itemName}"? This action cannot be undone.`)) return;
|
||||
try {
|
||||
const res = await api.fetchApi('/aiia/v1/browser/delete_item', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ path: this.currentPath, filename: itemName })
|
||||
});
|
||||
if (!res.ok) throw new Error(`HTTP ${res.status}: ${await res.text()}`);
|
||||
const result = await res.json();
|
||||
if (result.status === 'success') {
|
||||
this.setFocus(null, null);
|
||||
this.refreshCurrentDirectory();
|
||||
}
|
||||
} catch (e) {
|
||||
alert(`Error deleting item: ${e.message}`);
|
||||
console.error(e);
|
||||
}
|
||||
}
|
||||
updateTooltipSizeLimits() { const rect = this.contentPanel.getBoundingClientRect(); const maxSize = Math.min(rect.width, rect.height) * 0.30; this.tooltipElement.style.setProperty('--aiia-tooltip-media-max-size', `${maxSize}px`); }
|
||||
updateSortControls() { this.sortKeySelect.value = this.sortKey; this.sortDirButton.textContent = this.sortDir === 'asc' ? '▲' : '▼'; this.sortDirButton.dataset.tooltipText = `Sort ${this.sortDir === 'asc' ? 'Descending' : 'Ascending'}`; }
|
||||
showPathInput() { this.breadcrumbs.style.display = 'none'; this.pathInputContainer.style.display = 'flex'; this.pathInput.value = this.currentPath; this.pathInput.focus(); this.pathInput.select(); }
|
||||
@@ -1335,15 +1311,11 @@ class AIIABrowserDialog extends ComfyDialog {
|
||||
|
||||
if ((target.tagName === 'INPUT' && target !== this.pathInput) || target.tagName === 'SELECT') return;
|
||||
|
||||
const keyMap = { 'ArrowUp': 'moveFocus', 'ArrowDown': 'moveFocus', 'ArrowLeft': 'moveFocus', 'ArrowRight': 'moveFocus', 'Enter': 'activateFocusedItem', ' ': 'toggleTooltipForFocusedItem', 'Escape': 'close', 'Delete': 'deleteFocusedItem' };
|
||||
const keyMap = { 'ArrowUp': 'moveFocus', 'ArrowDown': 'moveFocus', 'ArrowLeft': 'moveFocus', 'ArrowRight': 'moveFocus', 'Enter': 'activateFocusedItem', ' ': 'toggleTooltipForFocusedItem' };
|
||||
if (keyMap[e.key]) {
|
||||
e.preventDefault(); e.stopPropagation();
|
||||
this.isKeyboardNavigating = true;
|
||||
if (keyMap[e.key] === 'moveFocus') this.moveFocus(e.key);
|
||||
else if (keyMap[e.key] === 'deleteFocusedItem') {
|
||||
const item = this.getAllNavigableItems()[this.currentFocusIndex];
|
||||
if (item) this.deleteItem(item.name);
|
||||
}
|
||||
else this[keyMap[e.key]]();
|
||||
}
|
||||
}
|
||||
@@ -1355,35 +1327,6 @@ class AIIABrowserDialog extends ComfyDialog {
|
||||
super.show();
|
||||
this.updateTooltipSizeLimits();
|
||||
this.element.focus({ preventScroll: true });
|
||||
|
||||
// [v1.9.184] Click-outside to close - targeting the native ComfyUI modal backdrop
|
||||
if (!this.outsideClickListener) {
|
||||
this.outsideClickListener = (e) => {
|
||||
if (this.element.style.display !== "none") {
|
||||
// If the click is on the parent container (the backdrop) and NOT on the element itself
|
||||
if (e.target === this.element.parentElement || e.target.classList.contains('comfy-modal')) {
|
||||
// Safety check: Don't close if we are interacting with another aiia modal
|
||||
if (this.fullscreenViewer && this.fullscreenViewer.element.style.display !== 'none') return;
|
||||
this.close();
|
||||
}
|
||||
}
|
||||
};
|
||||
setTimeout(() => document.addEventListener("mousedown", this.outsideClickListener), 10);
|
||||
}
|
||||
|
||||
// Force a resize/scroll event to trigger initial icon rendering
|
||||
setTimeout(() => { this.render(); }, 100);
|
||||
}
|
||||
|
||||
close() {
|
||||
if (this.outsideClickListener) {
|
||||
document.removeEventListener("mousedown", this.outsideClickListener);
|
||||
this.outsideClickListener = null;
|
||||
}
|
||||
this.element.style.display = "none";
|
||||
this.hideTooltip();
|
||||
if (typeof super.close === "function") super.close();
|
||||
else if (typeof super.hide === "function") super.hide();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1418,16 +1361,8 @@ app.registerExtension({
|
||||
let browserDialog = null;
|
||||
const createDialog = () => { if (!browserDialog) browserDialog = new AIIABrowserDialog(); browserDialog.show(); };
|
||||
document.addEventListener('keydown', (e) => {
|
||||
if (!browserDialog || (browserDialog.element && browserDialog.element.style.display === 'none')) return;
|
||||
if (browserDialog.fullscreenViewer && browserDialog.fullscreenViewer.element && browserDialog.fullscreenViewer.element.style.display !== 'none') return;
|
||||
|
||||
if (e.key === 'Escape') {
|
||||
e.preventDefault();
|
||||
e.stopPropagation();
|
||||
browserDialog.close();
|
||||
return;
|
||||
}
|
||||
|
||||
if (!browserDialog || browserDialog.element.style.display === 'none') return;
|
||||
if (browserDialog.fullscreenViewer && browserDialog.fullscreenViewer.element.style.display !== 'none') return;
|
||||
if (!browserDialog.element.contains(document.activeElement)) {
|
||||
const isNavKey = ['ArrowUp', 'ArrowDown', 'ArrowLeft', 'ArrowRight', 'Enter', ' '].includes(e.key);
|
||||
if (isNavKey) {
|
||||
|
||||
Executable → Regular
+1
-1
@@ -2,7 +2,7 @@
|
||||
export const browserStyles = `
|
||||
/* Main Browser Styles */
|
||||
#aiia-browser-menu-button { margin-left: 10px; }
|
||||
.comfy-modal.aiia-browser-dialog-root { top: calc(50% + 40px); left: 50%; transform: translate(-50%, -50%); width: 80vw; height: 85vh; max-width: 1400px; max-height: 1000px; min-width: 800px; min-height: 500px; display: flex; flex-direction: column; padding: 0; border-radius: 8px; box-shadow: 0 10px 30px rgba(0,0,0,0.2); resize: both; overflow: hidden; }
|
||||
.comfy-modal.aiia-browser-dialog-root { top: 50%; left: 50%; transform: translate(-50%, -50%); width: 80vw; height: 85vh; max-width: 1400px; max-height: 1000px; min-width: 800px; min-height: 500px; display: flex; flex-direction: column; padding: 0; border-radius: 8px; box-shadow: 0 10px 30px rgba(0,0,0,0.2); resize: both; overflow: hidden; }
|
||||
.aiia-browser-titlebar { background: var(--comfy-box-bg); padding: 4px 8px; font-weight: bold; display: flex; justify-content: space-between; align-items: center; border-bottom: 1px solid var(--border-color); flex-shrink: 0; color: #F9FAFB; cursor: default; }
|
||||
.aiia-browser-main-container { display: flex; flex-direction: column; flex-grow: 1; padding: 8px; overflow: hidden; }
|
||||
.aiia-browser-header-controls { display: flex; justify-content: space-between; align-items: center; margin-bottom: 8px; flex-shrink: 0; gap: 8px; flex-wrap: wrap; }
|
||||
|
||||
Executable → Regular
@@ -0,0 +1,55 @@
|
||||
|
||||
import { app } from "../../../scripts/app.js";
|
||||
|
||||
// Extension to handle static labels (identifying by 'is_label' flag in metadata)
|
||||
app.registerExtension({
|
||||
name: "AIIA.Labels",
|
||||
async beforeRegisterNodeDef(nodeType, nodeData, app) {
|
||||
// Iterate through required inputs to find ones marked as is_label
|
||||
const requiredInputs = nodeData.input?.required || {};
|
||||
|
||||
let labelWidgetNames = [];
|
||||
for (const [name, inputDef] of Object.entries(requiredInputs)) {
|
||||
// inputDef is [type, metadata_dict]
|
||||
if (inputDef[1] && inputDef[1].is_label === true) {
|
||||
labelWidgetNames.push(name);
|
||||
}
|
||||
}
|
||||
|
||||
if (labelWidgetNames.length > 0) {
|
||||
const onNodeCreated = nodeType.prototype.onNodeCreated;
|
||||
nodeType.prototype.onNodeCreated = function () {
|
||||
const r = onNodeCreated ? onNodeCreated.apply(this, arguments) : undefined;
|
||||
|
||||
for (const w of this.widgets) {
|
||||
if (labelWidgetNames.includes(w.name)) {
|
||||
// Change type to avoid standard text box rendering
|
||||
w.type = "AIIA_STATIC_TEXT";
|
||||
|
||||
w.draw = function (ctx, node, widget_width, y, widget_height) {
|
||||
ctx.save();
|
||||
// Background or subtle underline if needed
|
||||
// ctx.fillStyle = "#222222";
|
||||
// ctx.fillRect(0, y, widget_width, widget_height);
|
||||
|
||||
ctx.fillStyle = "#AAAAAA"; // Label color
|
||||
ctx.font = "italic 12px Arial";
|
||||
// Draw the text
|
||||
ctx.fillText(this.value, 15, y + widget_height * 0.7);
|
||||
ctx.restore();
|
||||
};
|
||||
|
||||
// Disable interaction
|
||||
w.mouse = () => { };
|
||||
w.computeSize = () => [200, 20];
|
||||
|
||||
// Prevent this widget from being converted to a socket (though it's already a Primitive STRING)
|
||||
w.inputKey = null;
|
||||
w.serializeValue = async () => ""; // Don't send label text back to Python
|
||||
}
|
||||
}
|
||||
return r;
|
||||
};
|
||||
}
|
||||
}
|
||||
});
|
||||
Executable → Regular
Executable → Regular
+16
-30
@@ -12,15 +12,15 @@ function toggleWidget(node, widget, show = false) {
|
||||
// console.log(`AIIA Debug (toggleWidget): Toggling '${widget.name}'. Should show: ${show}`);
|
||||
// --- Debug End ---
|
||||
|
||||
if (!widget) return;
|
||||
if (!origProps[widget.name]) {
|
||||
origProps[widget.name] = {
|
||||
origType: widget.type,
|
||||
origComputeSize: widget.computeSize
|
||||
};
|
||||
}
|
||||
widget.type = show ? origProps[widget.name].origType : "AIIA_HIDDEN";
|
||||
widget.computeSize = show ? origProps[widget.name].origComputeSize : () => [0, -4];
|
||||
if (!widget) return;
|
||||
if (!origProps[widget.name]) {
|
||||
origProps[widget.name] = {
|
||||
origType: widget.type,
|
||||
origComputeSize: widget.computeSize
|
||||
};
|
||||
}
|
||||
widget.type = show ? origProps[widget.name].origType : "AIIA_HIDDEN";
|
||||
widget.computeSize = show ? origProps[widget.name].origComputeSize : () => [0, -4];
|
||||
node.setDirtyCanvas(true);
|
||||
}
|
||||
|
||||
@@ -28,7 +28,7 @@ function toggleWidget(node, widget, show = false) {
|
||||
function chainCallback(object, property, callback) {
|
||||
if (object[property]) {
|
||||
const original = object[property];
|
||||
object[property] = function () {
|
||||
object[property] = function() {
|
||||
original.apply(this, arguments);
|
||||
callback.apply(this, arguments);
|
||||
};
|
||||
@@ -41,15 +41,15 @@ app.registerExtension({
|
||||
name: "AIIA.VideoNodes.DynamicWidgets.Final",
|
||||
async beforeRegisterNodeDef(nodeType, nodeData, app) {
|
||||
if (nodeData.name === "AIIA_VideoCombine") {
|
||||
|
||||
|
||||
const widgetsByFormat = nodeData.input.required.format[1].formats;
|
||||
if (!widgetsByFormat) return;
|
||||
|
||||
chainCallback(nodeType.prototype, "onNodeCreated", function () {
|
||||
chainCallback(nodeType.prototype, "onNodeCreated", function() {
|
||||
const node = this;
|
||||
const formatWidget = findWidgetByName(node, "format");
|
||||
if (!formatWidget) return;
|
||||
|
||||
|
||||
const allDynamicWidgetNames = new Set(Object.values(widgetsByFormat).flat().map(p => p[0]));
|
||||
|
||||
const updateWidgetsVisibility = (formatValue) => {
|
||||
@@ -57,37 +57,23 @@ app.registerExtension({
|
||||
const visibleWidgetNames = new Set(
|
||||
(widgetsByFormat[formatValue] || []).map(p => p[0])
|
||||
);
|
||||
|
||||
|
||||
for (const widgetName of allDynamicWidgetNames) {
|
||||
const widget = findWidgetByName(node, widgetName);
|
||||
if (widget) {
|
||||
toggleWidget(node, widget, visibleWidgetNames.has(widgetName));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
node.setSize([node.size[0], node.computeSize()[1]]);
|
||||
};
|
||||
|
||||
// 为format widget的callback链接上更新函数
|
||||
chainCallback(formatWidget, "callback", updateWidgetsVisibility);
|
||||
|
||||
// Expose function for onConfigure
|
||||
node.aiiaUpdateVideoWidgets = updateWidgetsVisibility;
|
||||
|
||||
|
||||
// 初始加载时触发
|
||||
updateWidgetsVisibility(formatWidget.value);
|
||||
});
|
||||
|
||||
chainCallback(nodeType.prototype, "onConfigure", function () {
|
||||
const node = this;
|
||||
// 使用 requestAnimationFrame 确保在所有widget值被写入后再执行更新
|
||||
requestAnimationFrame(() => {
|
||||
const formatWidget = findWidgetByName(node, "format");
|
||||
if (formatWidget && node.aiiaUpdateVideoWidgets) {
|
||||
node.aiiaUpdateVideoWidgets(formatWidget.value);
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
}
|
||||
});
|
||||
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
|
Before Width: | Height: | Size: 120 KiB After Width: | Height: | Size: 120 KiB |
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user