diff --git a/README.md b/README.md new file mode 100644 index 0000000..f0617e5 --- /dev/null +++ b/README.md @@ -0,0 +1,246 @@ +
+ +# Audio Separation 🎤🎛️ + +![Audio Separation Logo](example_workflows/audioseparation_logo.jpg) + +
+ +## Overview + +**AudioSeparation** is both, a ComfyUI group of nodes and a command line tool, to do +audio demixing, also known as audio separation. + +From an audio the objective is to separate the vocals, instruments, drums, bass, etc. +from the rest of the sounds. + +To achieve this we use [MDX Net](https://arxiv.org/abs/2111.12203) neural networks (models). +**AudioSeparation** currently supports [39 models](https://huggingface.co/set-soft/audio_separation) +from the [UVR5](https://github.com/Anjok07/ultimatevocalremovergui) project. + +The models are small (from 21 MB to 65 MB), but really efficient. +Models specialized on different stems are provided. +We support more than one model for each task because some times a model will perform better +for a song and other worst. + +We support two *karaoke* models. They are slightly different to regular models because they try +to keep the secondary vocals along with the instruments. + +The objectives for these nodes are: + +- Multiple stems (Vocals, Instruments, Drums, Bass, etc.) +- Easy of use +- Clear download (with progress and known destination) +- Support for all possible input audio (mono/stereo, any sample rate, any batch size) +- Good quality vs size +- Reduced dependencies + +--- + +## 📜 Table of Contents + +* [Overview](#overview) +* 🚀 [Installation](#-installation) + * [ComfyUI Nodes](#comfyui-nodes) + * [Command Line Tool](#command-line-tool) + * [Models](#models) +* 📦 [Dependencies](#-dependencies) +* 🖼️ [Usage](#️-usage) + * [ComfyUI](#comfyui) + * [Command Line](#command-line) +* ✨ [Nodes](#-nodes) + * [Vocals using MDX](#vocals-using-mdx) +* 📝 [Usage Notes](#-usage-notes) +* ⚖️ [License](#️-license) +* 🙏 [Attributions](#-attributions) + +___ + +## 🚀 Installation + +### ComfyUI Nodes + +You can install the nodes from the ComfyUI nodes manager, the name is *Audio Separation*, +or just do it manually: + +1. Clone this repository into your `ComfyUI/custom_nodes/` directory: + ```bash + cd ComfyUI/custom_nodes/ + git clone https://github.com/set-soft/AudioSeparation + ``` +2. Restart ComfyUI. + +The nodes should then appear under the "audio/separation" category in the "Add Node" menu. +You don't need to install extra dependencies. + +### Command Line Tool + +1. Clone this repository + +2. Change to its directory (`cd AudioSeparation`) + +3. Install the dependencies, only needed if ComfyUI isn't installed: + +```bash +pip3 install -r requirements.txt +``` + +or + +```bash +pip install -r requirements.txt +``` + +4. Run the scripts like this: + +``` +python3 tool/demix.py AUDIO_FILE +```` + +or + +``` +python tool/demix.py AUDIO_FILE +```` + +You don't need to install it, you could even add a symlink in `/usr/bin`. + +A list of all the available tools can be found [here](tool/README.md). + +### Models + +Models are automatically downloaded. + +When using ComfyUI they are downloaded to `ComfyUI/models/audio/MDX`. + +When using the command line the default is `../models` relative to the script, but you can specify another dir. + +If you want to download the models manually you can get the safetensors files from +[here](https://huggingface.co/set-soft/audio_separation/). + +For the command line you can also download the ONNX files from other repos. + + +## 📦 Dependencies + +These nodes just uses `torchaudio` (part of PyTorch), `numpy` for math and `tqdm` for progress bars. +All of them are used by ComfyUI, so you don't need to install any additional dependency on a ComfyUI setup. + +The following are optional dependencies: + +- onnxruntime: if you want to use the ONNX version of the models you'll need the ONNX runtime, but I don't see any advantage. Note that only the command line can handle them. +- requests: this is a robust internet connection module. If not installed we use the Python's `urllib` which is fine. +- colorama: it helps to generate colored messages (for debug, warning and errors). If not installed we use the most common ANSI escape sequences, supported by most terminals. You might need it for Windows. + + +## 🖼️ Usage + +### ComfyUI + +You can start using template workflows, ComfyUI *Workflow* menu and then choose *Browse Templates*, +look for "Audio Separation" + +If you want to do it manually you'll find the nodes in the *audio/separation* category. +Or you can use the search for them double clicking and then typing **MDX**: + +![ComfyUI Search](doc/node_search.png) + +Choose a node to extract what you want, i.e. *Vocals*. The complement output for it will +be the instruments, but using a node for *Instrumental* separation you'll usually get a better result +than using the *Complement* output. + +Then simply connect your audio input to the node (i.e. **LoadAudio** node from Comfy core) and +connect its output to some audio node (i.e. **PreviewAudio** or **SaveAudio** nodes from Comfy core). +You'll get something like this: + +![ComfyUI Search](doc/node_structure.png) + +Now choose a model in the MDX node. On a fresh install all models will have an arrow (⬇️ ) +indicating it must be downloaded. + +Now just run the workflow. That's all. + +Note that after downloading a model its name will change and will show a disk (💾). +If for some reason the list of models gets out of sync just press `R` to refresh ComfyUI and +select the correct name from the list. + +> [!TIP] +> Models are trained using 44.1 kHz audio, to get optimal results use this sample rate. +> The node will convert any sample rate to this. + + +### Command Line + +Use the `demix.py` tool like this + +``` +$ python3 tool/demix.py AUDIO_FILE +``` + +To get all the available options use: + +``` +$ python3 tool/demix.py --help +``` + +## ✨ Nodes + +`Vocals using MDX`, `Instrumental using MDX`, `Bass using MDX`, `Drums using MDX` and `Various using MDX` +share the same structure, so here is the first: + +### Vocals using MDX + - **Display Name:** `Vocals using MDX` + - **Internal Name:** `AudioSeparateVocals` + - **Category:** `audio/separation` + - **Description:** Takes one audio input (which can be batch) separates the vocals from the rest of the sounds. + - **Inputs:** + - `input_sound` (AUDIO): The audio input. Can be a single audio item or a batch. + - `model` (COMBO): The name of the model to use. Choose one from the list. + - `segments` (INT): How many segments to process at once. More segments needs more VRAM, but the audio might have less discontinuities. + - `taget_device` (COMBO): The device where we will run the neural network. + - **Output:** + - `Vocals` (AUDIO): The separated stem + - `Complement` (AUDIO): The input audio subtracting the `Vocals` + - **Behavior Details:** + - **Sample Rate:** The sample rate of the input is adjusted to 44.1 kHz + - **Channels:** Mono audios are converted to fake stereo (left == right) + - **Input Batch Handling:** If `input_sound` is a batch the outputs will be batches. The process is sequential, not parallel. + - **Missing Models:** They are downloaded and stored under `models/audio/MDX` of the ComfyUI installation + + +## 📝 Usage Notes + +- **AUDIO Type:** These nodes work with ComfyUI's standard "AUDIO" data type, which is a Python dictionary containing: + - `'waveform'`: A `torch.Tensor` of shape `(batch_size, num_channels, num_samples)`. + - `'sample_rate'`: An `int` representing the sample rate in Hz. +- **Logging:** 🔊 The nodes use Python's `logging` module. Debug messages can be helpful for understanding the transformations being applied. + You can control log verbosity through ComfyUI's startup arguments (e.g., `--preview-method auto --verbose DEBUG` for more detailed ComfyUI logs + which might also affect custom node loggers if they are configured to inherit levels). The logger name used is "AudioSeparation". + You can force debugging level for these nodes defining the `AUDIOSEPARATION_NODES_DEBUG` environment variable to `1`. + + +## ⚖️ License + +[GPL-3.0](LICENSE) + +The models are under [MIT license](https://opensource.org/license/mit), but aren't in this repo + +## 🙏 Attributions + +- Main author: [Salvador E. Tropea](https://github.com/set-soft) +- Assisted by Gemini 2.5 Pro, i.e. most of the inference class and ONNX to safetensors conversion + +______ + +- Various ideas from [DeepExtract](https://github.com/abdozmantar/ComfyUI-DeepExtract) by + [Abdullah Ozmantar](https://github.com/comfyui-abdozmantar) +- Models collected by the [UVR5 project](https://github.com/Anjok07/ultimatevocalremovergui) and + found in the [UVR Resources](https://huggingface.co/Politrees/UVR_resources) by + [Artyom Bebroy](https://github.com/Politrees) +- The logo image was created using text generated using [Text Studio](https://www.textstudio.com/) and + resources from [Vecteezy](https://www.vecteezy.com/) by: + - [Titima Ongkantong](https://www.vecteezy.com/members/titima157) + - [Chaiwut Sridara](https://www.vecteezy.com/members/cs21) + - [Kirsty Pargeter](https://www.vecteezy.com/members/kjpargeter2018) + - [Luz Eugenia Velasquez](https://www.vecteezy.com/members/jemastock) + - [Paulo Henrique dos Santos](https://www.vecteezy.com/members/pauloh381992288929) diff --git a/doc/node_search.png b/doc/node_search.png new file mode 100644 index 0000000..edb0cd1 Binary files /dev/null and b/doc/node_search.png differ diff --git a/doc/node_structure.png b/doc/node_structure.png new file mode 100644 index 0000000..ae8f3fd Binary files /dev/null and b/doc/node_structure.png differ diff --git a/example_workflows/00_Vocals.jpg b/example_workflows/00_Vocals.jpg new file mode 120000 index 0000000..a7fdea9 --- /dev/null +++ b/example_workflows/00_Vocals.jpg @@ -0,0 +1 @@ +audioseparation_logo.jpg \ No newline at end of file diff --git a/example_workflows/00_Vocals.json b/example_workflows/00_Vocals.json new file mode 100644 index 0000000..4a6064a --- /dev/null +++ b/example_workflows/00_Vocals.json @@ -0,0 +1 @@ +{"id":"74d99502-e6ea-4f02-a638-8220691cfe9e","revision":0,"last_node_id":8,"last_link_id":5,"nodes":[{"id":3,"type":"PreviewAudio","pos":[1580,1100],"size":[330,90],"flags":{},"order":4,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"AUDIO","link":3},{"localized_name":"audioUI","name":"audioUI","type":"AUDIO_UI","widget":{"name":"audioUI"},"link":null}],"outputs":[],"properties":{"cnr_id":"comfy-core","ver":"0.3.34","Node name for S&R":"PreviewAudio","widget_ue_connectable":{}},"widgets_values":[]},{"id":6,"type":"PreviewAudio","pos":[1580,1260],"size":[330,90],"flags":{},"order":5,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"AUDIO","link":5},{"localized_name":"audioUI","name":"audioUI","type":"AUDIO_UI","widget":{"name":"audioUI"},"link":null}],"outputs":[],"properties":{"cnr_id":"comfy-core","ver":"0.3.34","Node name for S&R":"PreviewAudio","widget_ue_connectable":{}},"widgets_values":[]},{"id":4,"type":"LoadAudio","pos":[850,1160],"size":[300,140],"flags":{},"order":0,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"COMBO","widget":{"name":"audio"},"link":null},{"localized_name":"audioUI","name":"audioUI","type":"AUDIO_UI","widget":{"name":"audioUI"},"link":null},{"localized_name":"choose file to upload","name":"upload","type":"AUDIOUPLOAD","widget":{"name":"upload"},"link":null}],"outputs":[{"localized_name":"AUDIO","name":"AUDIO","type":"AUDIO","links":[4]}],"properties":{"cnr_id":"comfy-core","ver":"0.3.34","Node name for S&R":"LoadAudio","widget_ue_connectable":{}},"widgets_values":["audio_ace-step_input.mp3",null,null]},{"id":5,"type":"AudioSeparateVocals","pos":[1220,1160],"size":[270,126],"flags":{},"order":3,"mode":0,"inputs":[{"localized_name":"input_sound","name":"input_sound","type":"AUDIO","link":4},{"localized_name":"model","name":"model","type":"COMBO","widget":{"name":"model"},"link":null},{"localized_name":"segments","name":"segments","type":"INT","widget":{"name":"segments"},"link":null},{"localized_name":"target_device","name":"target_device","type":"COMBO","widget":{"name":"target_device"},"link":null}],"outputs":[{"localized_name":"Vocals","name":"Vocals","type":"AUDIO","links":[3]},{"localized_name":"Complement","name":"Complement","type":"AUDIO","links":[5]}],"properties":{"aux_id":"set-soft/ComfyUI-DeepExtract","ver":"1e80343e5a83ecac2f38b6bb0d3d411f21dd1ca0","Node name for S&R":"AudioSeparateVocals","widget_ue_connectable":{}},"widgets_values":["⬇️ Kim Jensen 2",1,"cuda"]},{"id":7,"type":"MarkdownNote","pos":[840,920],"size":[320,140],"flags":{},"order":1,"mode":0,"inputs":[],"outputs":[],"properties":{},"widgets_values":["# 1 - Select the audio demix\n\nThe MDX models expects 44.1 kHz stereo audio. \nThe node will convert mono audio to a fake stereo audio.\nIf not 44.1 kHz it will be resampled\n"],"color":"#432","bgcolor":"#653"},{"id":8,"type":"MarkdownNote","pos":[1200,870],"size":[330,180],"flags":{},"order":2,"mode":0,"inputs":[],"outputs":[],"properties":{},"widgets_values":["# 2 - Select the model\n\nAn ⬇️ means the model will be downloaded and 💾 that is already on disk.\n\nNote that we have specialized nodes for Vocals, Instruments, Bass, Drums, Reverb and Others (not Vocals, Bass or Drum).\n\nUse a node for the output you want. Vocal node will also generate Instrumental output in the Complement, but you'll get better results using an Instrumental node.\nFor karaoke look for models specialized for it."],"color":"#432","bgcolor":"#653"}],"links":[[3,5,0,3,0,"AUDIO"],[4,4,0,5,0,"AUDIO"],[5,5,1,6,0,"AUDIO"]],"groups":[{"id":1,"title":"Built-in","bounding":[840,1070,320,250],"color":"#3f789e","font_size":24,"flags":{}},{"id":2,"title":"Built-in","bounding":[1570,1000,350,380],"color":"#3f789e","font_size":24,"flags":{}},{"id":3,"title":"AudioSeparation","bounding":[1180,1070,370,230],"color":"#A88","font_size":24,"flags":{}}],"config":{},"extra":{"ue_links":[],"links_added_by_ue":[],"ds":{"scale":1.3580092944295392,"offset":[-855.972073881374,-782.5230145288682]}},"version":0.4} diff --git a/example_workflows/01_Vocals_Drums_Bass.jpg b/example_workflows/01_Vocals_Drums_Bass.jpg new file mode 120000 index 0000000..a7fdea9 --- /dev/null +++ b/example_workflows/01_Vocals_Drums_Bass.jpg @@ -0,0 +1 @@ +audioseparation_logo.jpg \ No newline at end of file diff --git a/example_workflows/01_Vocals_Drums_Bass.json b/example_workflows/01_Vocals_Drums_Bass.json new file mode 100644 index 0000000..9dd218d --- /dev/null +++ b/example_workflows/01_Vocals_Drums_Bass.json @@ -0,0 +1 @@ +{"id":"151379ea-7ee4-4023-b8ec-39b52efbc365","revision":0,"last_node_id":12,"last_link_id":11,"nodes":[{"id":3,"type":"PreviewAudio","pos":[1580,1160],"size":[330,90],"flags":{},"order":6,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"AUDIO","link":3},{"localized_name":"audioUI","name":"audioUI","type":"AUDIO_UI","widget":{"name":"audioUI"},"link":null}],"outputs":[],"properties":{"cnr_id":"comfy-core","ver":"0.3.34","Node name for S&R":"PreviewAudio","widget_ue_connectable":{}},"widgets_values":[]},{"id":5,"type":"AudioSeparateVocals","pos":[1220,1160],"size":[270,126],"flags":{},"order":2,"mode":0,"inputs":[{"localized_name":"input_sound","name":"input_sound","type":"AUDIO","link":4},{"localized_name":"model","name":"model","type":"COMBO","widget":{"name":"model"},"link":null},{"localized_name":"segments","name":"segments","type":"INT","widget":{"name":"segments"},"link":null},{"localized_name":"target_device","name":"target_device","type":"COMBO","widget":{"name":"target_device"},"link":null}],"outputs":[{"localized_name":"Vocals","name":"Vocals","type":"AUDIO","links":[3]},{"localized_name":"Complement","name":"Complement","type":"AUDIO","links":[]}],"properties":{"aux_id":"set-soft/ComfyUI-DeepExtract","ver":"1e80343e5a83ecac2f38b6bb0d3d411f21dd1ca0","Node name for S&R":"AudioSeparateVocals","widget_ue_connectable":{}},"widgets_values":["💾 KUIELab B",1,"cuda"]},{"id":7,"type":"AudioSeparateDrums","pos":[1220,1350],"size":[270,126],"flags":{},"order":3,"mode":0,"inputs":[{"localized_name":"input_sound","name":"input_sound","type":"AUDIO","link":6},{"localized_name":"model","name":"model","type":"COMBO","widget":{"name":"model"},"link":null},{"localized_name":"segments","name":"segments","type":"INT","widget":{"name":"segments"},"link":null},{"localized_name":"target_device","name":"target_device","type":"COMBO","widget":{"name":"target_device"},"link":null}],"outputs":[{"localized_name":"Drums","name":"Drums","type":"AUDIO","links":[7]},{"localized_name":"Complement","name":"Complement","type":"AUDIO","links":null}],"properties":{"aux_id":"set-soft/ComfyUI-DeepExtract","ver":"f9b0a02700853168fa4b7324bf362da1c11eae19","Node name for S&R":"AudioSeparateDrums","widget_ue_connectable":{}},"widgets_values":["⬇️ KUIELab B",1,"cuda"]},{"id":6,"type":"PreviewAudio","pos":[1580,1350],"size":[330,90],"flags":{},"order":7,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"AUDIO","link":7},{"localized_name":"audioUI","name":"audioUI","type":"AUDIO_UI","widget":{"name":"audioUI"},"link":null}],"outputs":[],"properties":{"cnr_id":"comfy-core","ver":"0.3.34","Node name for S&R":"PreviewAudio","widget_ue_connectable":{}},"widgets_values":[]},{"id":8,"type":"AudioSeparateBass","pos":[1220,1540],"size":[270,126],"flags":{},"order":4,"mode":0,"inputs":[{"localized_name":"input_sound","name":"input_sound","type":"AUDIO","link":10},{"localized_name":"model","name":"model","type":"COMBO","widget":{"name":"model"},"link":null},{"localized_name":"segments","name":"segments","type":"INT","widget":{"name":"segments"},"link":null},{"localized_name":"target_device","name":"target_device","type":"COMBO","widget":{"name":"target_device"},"link":null}],"outputs":[{"localized_name":"Bass","name":"Bass","type":"AUDIO","links":[8]},{"localized_name":"Complement","name":"Complement","type":"AUDIO","links":null}],"properties":{"aux_id":"set-soft/ComfyUI-DeepExtract","ver":"f9b0a02700853168fa4b7324bf362da1c11eae19","Node name for S&R":"AudioSeparateBass","widget_ue_connectable":{}},"widgets_values":["⬇️ KUIELab B",1,"cuda"]},{"id":9,"type":"AudioSeparateVarious","pos":[1220,1730],"size":[270,126],"flags":{},"order":5,"mode":0,"inputs":[{"localized_name":"input_sound","name":"input_sound","type":"AUDIO","link":11},{"localized_name":"model","name":"model","type":"COMBO","widget":{"name":"model"},"link":null},{"localized_name":"segments","name":"segments","type":"INT","widget":{"name":"segments"},"link":null},{"localized_name":"target_device","name":"target_device","type":"COMBO","widget":{"name":"target_device"},"link":null}],"outputs":[{"localized_name":"Main","name":"Main","type":"AUDIO","links":[9]},{"localized_name":"Complement","name":"Complement","type":"AUDIO","links":null}],"properties":{"aux_id":"set-soft/ComfyUI-DeepExtract","ver":"f9b0a02700853168fa4b7324bf362da1c11eae19","Node name for S&R":"AudioSeparateVarious","widget_ue_connectable":{}},"widgets_values":["⬇️ KUIELab Other B",1,"cuda"]},{"id":11,"type":"PreviewAudio","pos":[1580,1730],"size":[330,90],"flags":{},"order":9,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"AUDIO","link":9},{"localized_name":"audioUI","name":"audioUI","type":"AUDIO_UI","widget":{"name":"audioUI"},"link":null}],"outputs":[],"properties":{"cnr_id":"comfy-core","ver":"0.3.34","Node name for S&R":"PreviewAudio","widget_ue_connectable":{}},"widgets_values":[]},{"id":10,"type":"PreviewAudio","pos":[1580,1540],"size":[330,90],"flags":{},"order":8,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"AUDIO","link":8},{"localized_name":"audioUI","name":"audioUI","type":"AUDIO_UI","widget":{"name":"audioUI"},"link":null}],"outputs":[],"properties":{"cnr_id":"comfy-core","ver":"0.3.34","Node name for S&R":"PreviewAudio","widget_ue_connectable":{}},"widgets_values":[]},{"id":4,"type":"LoadAudio","pos":[810,1460],"size":[300,140],"flags":{},"order":0,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"COMBO","widget":{"name":"audio"},"link":null},{"localized_name":"audioUI","name":"audioUI","type":"AUDIO_UI","widget":{"name":"audioUI"},"link":null},{"localized_name":"choose file to upload","name":"upload","type":"AUDIOUPLOAD","widget":{"name":"upload"},"link":null}],"outputs":[{"localized_name":"AUDIO","name":"AUDIO","type":"AUDIO","links":[4,6,10,11]}],"properties":{"cnr_id":"comfy-core","ver":"0.3.34","Node name for S&R":"LoadAudio","widget_ue_connectable":{}},"widgets_values":["madonna-la-isla-bonita.mp3",null,null]},{"id":12,"type":"MarkdownNote","pos":[810,1240],"size":[310,110],"flags":{},"order":1,"mode":0,"inputs":[],"outputs":[],"properties":{"widget_ue_connectable":{}},"widgets_values":["# 1 - Select the audio demix\n\nThis example shows how to obtain vocals, drums, bass and the rest"],"color":"#432","bgcolor":"#653"}],"links":[[3,5,0,3,0,"AUDIO"],[4,4,0,5,0,"AUDIO"],[6,4,0,7,0,"AUDIO"],[7,7,0,6,0,"AUDIO"],[8,8,0,10,0,"AUDIO"],[9,9,0,11,0,"AUDIO"],[10,4,0,8,0,"AUDIO"],[11,4,0,9,0,"AUDIO"]],"groups":[{"id":1,"title":"Built-in","bounding":[800,1370,320,250],"color":"#3f789e","font_size":24,"flags":{}},{"id":2,"title":"Built-in","bounding":[1570,1060,350,820],"color":"#3f789e","font_size":24,"flags":{}},{"id":3,"title":"AudioSeparation","bounding":[1180,1070,360,810],"color":"#A88","font_size":24,"flags":{}}],"config":{},"extra":{"ue_links":[],"links_added_by_ue":[],"ds":{"scale":1.1223217309335034,"offset":[-642.4277342028577,-967.989755125568]}},"version":0.4} diff --git a/example_workflows/02_Batch.jpg b/example_workflows/02_Batch.jpg new file mode 120000 index 0000000..a7fdea9 --- /dev/null +++ b/example_workflows/02_Batch.jpg @@ -0,0 +1 @@ +audioseparation_logo.jpg \ No newline at end of file diff --git a/example_workflows/02_Batch.json b/example_workflows/02_Batch.json new file mode 100644 index 0000000..9c3575c --- /dev/null +++ b/example_workflows/02_Batch.json @@ -0,0 +1 @@ +{"id":"e425571c-2aac-4ba2-90dd-77389f6df58c","revision":0,"last_node_id":14,"last_link_id":15,"nodes":[{"id":9,"type":"LoadAudio","pos":[600,1080],"size":[350,150],"flags":{},"order":0,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"COMBO","widget":{"name":"audio"},"link":null},{"localized_name":"audioUI","name":"audioUI","type":"AUDIO_UI","widget":{"name":"audioUI"},"link":null},{"localized_name":"choose file to upload","name":"upload","type":"AUDIOUPLOAD","widget":{"name":"upload"},"link":null}],"outputs":[{"localized_name":"AUDIO","name":"AUDIO","type":"AUDIO","links":[6]}],"properties":{"cnr_id":"comfy-core","ver":"0.3.34","Node name for S&R":"LoadAudio","widget_ue_connectable":{}},"widgets_values":["madonna-la-isla-bonita.mp3",null,null],"color":"#233","bgcolor":"#355"},{"id":10,"type":"LoadAudio","pos":[600,1280],"size":[350,150],"flags":{},"order":1,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"COMBO","widget":{"name":"audio"},"link":null},{"localized_name":"audioUI","name":"audioUI","type":"AUDIO_UI","widget":{"name":"audioUI"},"link":null},{"localized_name":"choose file to upload","name":"upload","type":"AUDIOUPLOAD","widget":{"name":"upload"},"link":null}],"outputs":[{"localized_name":"AUDIO","name":"AUDIO","type":"AUDIO","links":[7]}],"properties":{"cnr_id":"comfy-core","ver":"0.3.34","Node name for S&R":"LoadAudio","widget_ue_connectable":{}},"widgets_values":["The_Beatles_-_Yesterday.mp3",null,null],"color":"#233","bgcolor":"#355"},{"id":6,"type":"PreviewAudio","pos":[2110,1270],"size":[330,90],"flags":{},"order":8,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"AUDIO","link":12},{"localized_name":"audioUI","name":"audioUI","type":"AUDIO_UI","widget":{"name":"audioUI"},"link":null}],"outputs":[],"properties":{"cnr_id":"comfy-core","ver":"0.3.34","Node name for S&R":"PreviewAudio","widget_ue_connectable":{}},"widgets_values":[]},{"id":3,"type":"PreviewAudio","pos":[2110,1070],"size":[330,90],"flags":{},"order":7,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"AUDIO","link":10},{"localized_name":"audioUI","name":"audioUI","type":"AUDIO_UI","widget":{"name":"audioUI"},"link":null}],"outputs":[],"properties":{"cnr_id":"comfy-core","ver":"0.3.34","Node name for S&R":"PreviewAudio","widget_ue_connectable":{}},"widgets_values":[]},{"id":13,"type":"SET_SelectAudioFromBatch","pos":[1690,1270],"size":[350,106],"flags":{},"order":6,"mode":0,"inputs":[{"localized_name":"audio_batch","name":"audio_batch","type":"AUDIO","link":15},{"localized_name":"index","name":"index","type":"INT","widget":{"name":"index"},"link":null},{"localized_name":"behavior_out_of_range","name":"behavior_out_of_range","type":"COMBO","widget":{"name":"behavior_out_of_range"},"link":null},{"localized_name":"silence_duration_seconds","name":"silence_duration_seconds","type":"FLOAT","widget":{"name":"silence_duration_seconds"},"link":null}],"outputs":[{"localized_name":"selected_audio","name":"selected_audio","type":"AUDIO","links":[12]}],"properties":{"cnr_id":"audio-batch","ver":"65e3da3da285b8593e74f138d114ec0b576a22bd","Node name for S&R":"SET_SelectAudioFromBatch","widget_ue_connectable":{},"aux_id":"set-soft/ComfyUI-AudioBatch"},"widgets_values":[1,"silence_original_length",1],"color":"#2a363b","bgcolor":"#3f5159"},{"id":12,"type":"SET_SelectAudioFromBatch","pos":[1690,1070],"size":[350,106],"flags":{},"order":5,"mode":0,"inputs":[{"localized_name":"audio_batch","name":"audio_batch","type":"AUDIO","link":14},{"localized_name":"index","name":"index","type":"INT","widget":{"name":"index"},"link":null},{"localized_name":"behavior_out_of_range","name":"behavior_out_of_range","type":"COMBO","widget":{"name":"behavior_out_of_range"},"link":null},{"localized_name":"silence_duration_seconds","name":"silence_duration_seconds","type":"FLOAT","widget":{"name":"silence_duration_seconds"},"link":null}],"outputs":[{"localized_name":"selected_audio","name":"selected_audio","type":"AUDIO","links":[10]}],"properties":{"cnr_id":"audio-batch","ver":"65e3da3da285b8593e74f138d114ec0b576a22bd","Node name for S&R":"SET_SelectAudioFromBatch","widget_ue_connectable":{},"aux_id":"set-soft/ComfyUI-AudioBatch"},"widgets_values":[0,"silence_original_length",1],"color":"#2a363b","bgcolor":"#3f5159"},{"id":11,"type":"SET_AudioBatch","pos":[1040,1160],"size":[146.42578125,46],"flags":{},"order":3,"mode":0,"inputs":[{"localized_name":"audio1","name":"audio1","type":"AUDIO","link":6},{"localized_name":"audio2","name":"audio2","type":"AUDIO","link":7}],"outputs":[{"localized_name":"audio_batch","name":"audio_batch","type":"AUDIO","links":[13]}],"properties":{"cnr_id":"audio-batch","ver":"65e3da3da285b8593e74f138d114ec0b576a22bd","Node name for S&R":"SET_AudioBatch","widget_ue_connectable":{},"aux_id":"set-soft/ComfyUI-AudioBatch"},"widgets_values":[],"color":"#2a363b","bgcolor":"#3f5159"},{"id":7,"type":"MarkdownNote","pos":[590,840],"size":[370,140],"flags":{},"order":2,"mode":0,"inputs":[],"outputs":[],"properties":{"widget_ue_connectable":{}},"widgets_values":["# Audio in batch\n\nThis example shows the nodes can handle audio batches.\n\nCurrently there is no performance advantage."],"color":"#432","bgcolor":"#653"},{"id":14,"type":"AudioSeparateInstrumental","pos":[1290,1160],"size":[270,126],"flags":{},"order":4,"mode":0,"inputs":[{"localized_name":"input_sound","name":"input_sound","type":"AUDIO","link":13},{"localized_name":"model","name":"model","type":"COMBO","widget":{"name":"model"},"link":null},{"localized_name":"segments","name":"segments","type":"INT","widget":{"name":"segments"},"link":null},{"localized_name":"target_device","name":"target_device","type":"COMBO","widget":{"name":"target_device"},"link":null}],"outputs":[{"localized_name":"Instrumental","name":"Instrumental","type":"AUDIO","links":[14,15]},{"localized_name":"Complement","name":"Complement","type":"AUDIO","links":null}],"properties":{"aux_id":"set-soft/ComfyUI-DeepExtract","ver":"d0c0810dde744fd5435ccafb2e470dd4b55b64f9","Node name for S&R":"AudioSeparateInstrumental","widget_ue_connectable":{}},"widgets_values":["⬇️ Kim Jensen",1,"cuda"]}],"links":[[6,9,0,11,0,"AUDIO"],[7,10,0,11,1,"AUDIO"],[10,12,0,3,0,"AUDIO"],[12,13,0,6,0,"AUDIO"],[13,11,0,14,0,"AUDIO"],[14,14,0,12,0,"AUDIO"],[15,14,0,13,0,"AUDIO"]],"groups":[{"id":1,"title":"Built-in","bounding":[590,990,370,450],"color":"#3f789e","font_size":24,"flags":{}},{"id":2,"title":"Built-in","bounding":[2100,980,350,390],"color":"#3f789e","font_size":24,"flags":{}},{"id":3,"title":"AudioSeparation","bounding":[1250,1070,370,230],"color":"#A88","font_size":24,"flags":{}},{"id":4,"title":"AudioBatch","bounding":[1670,980,390,420],"color":"#8A8","font_size":24,"flags":{}},{"id":5,"title":"AudioBatch","bounding":[1020,1070,190,150],"color":"#8A8","font_size":24,"flags":{}}],"config":{},"extra":{"ue_links":[],"links_added_by_ue":[],"ds":{"scale":0.9275386206062011,"offset":[-309.290090332301,-648.4922258980088]}},"version":0.4} diff --git a/example_workflows/03_Instrumental_keep.jpg b/example_workflows/03_Instrumental_keep.jpg new file mode 120000 index 0000000..a7fdea9 --- /dev/null +++ b/example_workflows/03_Instrumental_keep.jpg @@ -0,0 +1 @@ +audioseparation_logo.jpg \ No newline at end of file diff --git a/example_workflows/03_Instrumental_keep.json b/example_workflows/03_Instrumental_keep.json new file mode 100644 index 0000000..3a4fe1c --- /dev/null +++ b/example_workflows/03_Instrumental_keep.json @@ -0,0 +1 @@ +{"id":"ad88a81a-2172-4191-be75-10bd6fc0aaab","revision":0,"last_node_id":21,"last_link_id":32,"nodes":[{"id":9,"type":"SET_AudioInfo","pos":[1370,1390],"size":[149.63046264648438,106],"flags":{},"order":2,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"AUDIO","link":6}],"outputs":[{"localized_name":"audio_bypass","name":"audio_bypass","type":"AUDIO","links":null},{"localized_name":"batch_size","name":"batch_size","type":"INT","links":[]},{"localized_name":"channels","name":"channels","type":"INT","links":[13]},{"localized_name":"num_samples","name":"num_samples","type":"INT","links":[]},{"localized_name":"sample_rate","name":"sample_rate","type":"INT","links":[12]}],"properties":{"cnr_id":"audio-batch","ver":"1df62f7c5184849e0cf51bd3f2cb8f93e6748d06","Node name for S&R":"SET_AudioInfo","widget_ue_connectable":{}},"widgets_values":[]},{"id":16,"type":"SET_AudioForceChannels","pos":[1600,1160],"size":[210,60],"flags":{},"order":4,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"AUDIO","link":32},{"localized_name":"channels","name":"channels","type":"INT","widget":{"name":"channels"},"link":13}],"outputs":[{"localized_name":"audio","name":"audio","type":"AUDIO","links":[28]}],"properties":{"cnr_id":"audio-batch","ver":"1df62f7c5184849e0cf51bd3f2cb8f93e6748d06","widget_ue_connectable":{},"Node name for S&R":"SET_AudioForceChannels"},"widgets_values":[0]},{"id":15,"type":"SET_AudioResampler","pos":[1880,1160],"size":[210,58],"flags":{},"order":5,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"AUDIO","link":28},{"localized_name":"target_sample_rate","name":"target_sample_rate","type":"INT","widget":{"name":"target_sample_rate"},"link":12}],"outputs":[{"localized_name":"audio_out","name":"audio_out","type":"AUDIO","links":[29,30]}],"properties":{"cnr_id":"audio-batch","ver":"1df62f7c5184849e0cf51bd3f2cb8f93e6748d06","Node name for S&R":"SET_AudioResampler","widget_ue_connectable":{}},"widgets_values":[0]},{"id":18,"type":"SET_AudioInfo","pos":[2140,910],"size":[149.63046264648438,106],"flags":{},"order":7,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"AUDIO","link":30}],"outputs":[{"localized_name":"audio_bypass","name":"audio_bypass","type":"AUDIO","links":null},{"localized_name":"batch_size","name":"batch_size","type":"INT","links":null},{"localized_name":"channels","name":"channels","type":"INT","links":[22]},{"localized_name":"num_samples","name":"num_samples","type":"INT","links":null},{"localized_name":"sample_rate","name":"sample_rate","type":"INT","links":[23]}],"properties":{"cnr_id":"audio-batch","ver":"1df62f7c5184849e0cf51bd3f2cb8f93e6748d06","widget_ue_connectable":{},"Node name for S&R":"SET_AudioInfo"}},{"id":20,"type":"easy showAnything","pos":[2360,1010],"size":[140,76.00006103515625],"flags":{},"order":9,"mode":0,"inputs":[{"localized_name":"anything","name":"anything","shape":7,"type":"*","link":23}],"outputs":[{"localized_name":"output","name":"output","type":"*","links":null}],"properties":{"cnr_id":"comfyui-easy-use","ver":"1.2.9","widget_ue_connectable":{},"Node name for S&R":"easy showAnything"},"widgets_values":["44100"]},{"id":19,"type":"easy showAnything","pos":[2360,860],"size":[140,76],"flags":{},"order":8,"mode":0,"inputs":[{"localized_name":"anything","name":"anything","shape":7,"type":"*","link":22}],"outputs":[{"localized_name":"output","name":"output","type":"*","links":null}],"properties":{"cnr_id":"comfyui-easy-use","ver":"1.2.9","widget_ue_connectable":{},"Node name for S&R":"easy showAnything"},"widgets_values":["2"]},{"id":4,"type":"LoadAudio","pos":[800,1160],"size":[300,140],"flags":{},"order":0,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"COMBO","widget":{"name":"audio"},"link":null},{"localized_name":"audioUI","name":"audioUI","type":"AUDIO_UI","widget":{"name":"audioUI"},"link":null},{"localized_name":"choose file to upload","name":"upload","type":"AUDIOUPLOAD","widget":{"name":"upload"},"link":null}],"outputs":[{"localized_name":"AUDIO","name":"AUDIO","type":"AUDIO","links":[6,31]}],"properties":{"cnr_id":"comfy-core","ver":"0.3.34","Node name for S&R":"LoadAudio","widget_ue_connectable":{}},"widgets_values":["audio_ace-step_input.mp3",null,null]},{"id":7,"type":"MarkdownNote","pos":[800,950],"size":[690,140],"flags":{},"order":1,"mode":0,"inputs":[],"outputs":[],"properties":{"widget_ue_connectable":{}},"widgets_values":["# Keep channels and sample rate example\n\nHere the node will convert to 44.1 kHz stereo, but we then change it back to the original\n"],"color":"#432","bgcolor":"#653"},{"id":3,"type":"PreviewAudio","pos":[2180,1160],"size":[330,90],"flags":{},"order":6,"mode":0,"inputs":[{"localized_name":"audio","name":"audio","type":"AUDIO","link":29},{"localized_name":"audioUI","name":"audioUI","type":"AUDIO_UI","widget":{"name":"audioUI"},"link":null}],"outputs":[],"properties":{"cnr_id":"comfy-core","ver":"0.3.34","Node name for S&R":"PreviewAudio","widget_ue_connectable":{}},"widgets_values":[]},{"id":21,"type":"AudioSeparateInstrumental","pos":[1230,1160],"size":[270,126],"flags":{},"order":3,"mode":0,"inputs":[{"localized_name":"input_sound","name":"input_sound","type":"AUDIO","link":31},{"localized_name":"model","name":"model","type":"COMBO","widget":{"name":"model"},"link":null},{"localized_name":"segments","name":"segments","type":"INT","widget":{"name":"segments"},"link":null},{"localized_name":"target_device","name":"target_device","type":"COMBO","widget":{"name":"target_device"},"link":null}],"outputs":[{"localized_name":"Instrumental","name":"Instrumental","type":"AUDIO","links":[32]},{"localized_name":"Complement","name":"Complement","type":"AUDIO","links":null}],"properties":{"aux_id":"set-soft/ComfyUI-DeepExtract","ver":"d0c0810dde744fd5435ccafb2e470dd4b55b64f9","widget_ue_connectable":{},"Node name for S&R":"AudioSeparateInstrumental"},"widgets_values":["⬇️ Kim Jensen",1,"cuda"]}],"links":[[6,4,0,9,0,"AUDIO"],[12,9,4,15,1,"INT"],[13,9,2,16,1,"INT"],[22,18,2,19,0,"*"],[23,18,4,20,0,"*"],[28,16,0,15,0,"AUDIO"],[29,15,0,3,0,"AUDIO"],[30,15,0,18,0,"AUDIO"],[31,4,0,21,0,"AUDIO"],[32,21,0,16,0,"AUDIO"]],"groups":[],"config":{},"extra":{"ue_links":[],"links_added_by_ue":[],"ds":{"scale":0.9275386206062011,"offset":[-647.7401041252758,-700.2851527231417]},"linkExtensions":[{"id":6,"parentId":3}],"reroutes":[{"id":3,"pos":[1200,1390],"linkIds":[6]}]},"version":0.4} diff --git a/example_workflows/audioseparation_logo.jpg b/example_workflows/audioseparation_logo.jpg new file mode 100644 index 0000000..206899b Binary files /dev/null and b/example_workflows/audioseparation_logo.jpg differ diff --git a/tool/README.md b/tool/README.md new file mode 100644 index 0000000..2c8a675 --- /dev/null +++ b/tool/README.md @@ -0,0 +1,77 @@ +# Audio separatio tools + +Some random tools used during development. + +They are intended to be invoked from the root of the nodes. + +# ONNX to safetensors converter + +File: onnx2safetensors.py + +Used to convert MDX-Net models in ONNX format to safetensors. +Might work for other ONNX files using the same operands. +But you'll need a PyTorch class for its architecture that creates the layers +in the same order as the ONNX file. + +# Batch converter + +File: batch_convert.py + +Used to convert all the MDX-Net files in one run + +# Demix + +File: demix.py + +Used to run the inference on audio files performing the demixing process + +# Download Model + +File: download_model.py + +Downloads a model listed in the DB + +# Show Data Base + +File: show_db.py + +Prints the content of the models data base. +This is useful to adjust details in the DB, but you need to modify the code (`apply_process`) + +# Show ONNX + +File: show_onnx.py + +Displays the ONNX input, output and layers. +Is a text representation, not as nice as [Netron](https://netron.app/), but you +can extract a lot of information from it. + +It can also show details of what onnx2pytorch interprets from it. + +And can do an inferece run using random data. We used it to determine some +details of the onnx2pytorch details. + +# Show Class + +File: show_class.py + +Used to display our PyTorch class, also the state_dict keys. +It can optionally export the class structure as an ONNX file that can be loaded by +Netron, but contains too much extra names. + +# Style Fixer + +File: style_fixer.py + +Fixes various common style errors found in Gemini 2.5 Pro code. +The script was created by Gemini ... can create the script, but can't stop making +the same errors over and over. + +# UVR Hash + +File: uvr_hash.py + +Computes the hash used by UVR to identify a model. We use the same hash, as many +tools do. + +This is the key value for aur model database.