From ac4ef0ee9dea0a7c2e15e75f95682547d39e354f Mon Sep 17 00:00:00 2001 From: "Salvador E. Tropea" Date: Wed, 2 Jul 2025 08:58:09 -0300 Subject: [PATCH] [DOCs][Added] Some random notes --- doc/MDX/.gitignore | 1 + doc/MDX/BatchNormalization.md | 12 + doc/MDX/MDX-Net.md | 72 ++++ doc/MDX/MatMul.md | 85 ++++ doc/MDX/maps/Kim_Vocal_2.arch.txt | 343 +++++++++++++++ doc/MDX/maps/Kim_Vocal_2_class.map | 201 +++++++++ doc/MDX/maps/Kim_Vocal_2_onnx.map | 642 +++++++++++++++++++++++++++++ doc/MDX/maps/mapping.txt | 232 +++++++++++ doc/Resources.md | 16 + 9 files changed, 1604 insertions(+) create mode 100644 doc/MDX/.gitignore create mode 100644 doc/MDX/BatchNormalization.md create mode 100644 doc/MDX/MDX-Net.md create mode 100644 doc/MDX/MatMul.md create mode 100644 doc/MDX/maps/Kim_Vocal_2.arch.txt create mode 100644 doc/MDX/maps/Kim_Vocal_2_class.map create mode 100644 doc/MDX/maps/Kim_Vocal_2_onnx.map create mode 100644 doc/MDX/maps/mapping.txt create mode 100644 doc/Resources.md diff --git a/doc/MDX/.gitignore b/doc/MDX/.gitignore new file mode 100644 index 0000000..97bdc6b --- /dev/null +++ b/doc/MDX/.gitignore @@ -0,0 +1 @@ +informal diff --git a/doc/MDX/BatchNormalization.md b/doc/MDX/BatchNormalization.md new file mode 100644 index 0000000..03a89c5 --- /dev/null +++ b/doc/MDX/BatchNormalization.md @@ -0,0 +1,12 @@ +### Takeaway: Findings on `BatchNormalization` Conversion + +Our rigorous, isolated testing of the `BatchNormalization` node has yielded one crucial, definitive finding: + +**A standard PyTorch `torch.nn.BatchNorm2d` layer is a numerically exact equivalent of the ONNX `BatchNormalization` operator when applied to 4D tensors, provided the weights are loaded directly from the ONNX initializers.** + +This was proven by the `isolate_and_verify_bn.py` script, where our manual replication (`my_bn`) perfectly matched the output of the ONNX Runtime ground truth. + +This key finding has two critical implications: + +1. **Our Core Understanding is Correct:** We can be 100% confident that `nn.BatchNorm2d` is the correct PyTorch module to use for this operation in our final standalone model. +2. **The `onnx2pytorch` Discrepancy:** The fact that the `onnx2pytorch` `BatchNormWrapper` produced a different result in the same test proves that it has a non-standard implementation. While this non-standard behavior works within the context of the full `onnx2pytorch` converted model, it is not a faithful reproduction of the ONNX specification for that isolated layer. Therefore, we should **not** use the `onnx2pytorch` `state_dict` as a source for `BatchNorm` weights, as they may be part of a flawed layer; we must use the raw ONNX initializers. diff --git a/doc/MDX/MDX-Net.md b/doc/MDX/MDX-Net.md new file mode 100644 index 0000000..af54719 --- /dev/null +++ b/doc/MDX/MDX-Net.md @@ -0,0 +1,72 @@ +## **Architectural Analysis of a Hybrid Time-Frequency Domain U-Net for Music Source Separation** + +### Abstract + +This document provides a detailed architectural analysis of a sophisticated deep learning model used for state-of-the-art music source separation. The architecture, a common and high-performing topology found in models submitted to the Music Demixing (MDX) Challenge and used by prominent community trainers like Kim Jensen, is a heavily modified U-Net. Its core innovation is a unique building block that processes audio information in both the time and frequency domains sequentially within a residual connection. This hybrid approach allows the network to effectively learn both the local timbral characteristics and the global harmonic structure of audio, leading to high-fidelity separation of stems like vocals, drums, and bass. + +--- + +### 1. Overall Architecture: A Modified U-Net + +At the highest level, the model is a variant of the classic **U-Net architecture**, widely used in both image segmentation and audio processing. The network consists of an encoder path, a bottleneck, and a decoder path, with a total of five resolution levels. + +* **Encoder:** A series of five processing stages (`enc_stage_n`) that progressively extract features while five downsampling layers (`enc_down_n`) reduce the spatial dimensions of the feature maps by a factor of two at each step. +* **Bottleneck:** The deepest point in the network, where the feature representation is most compressed spatially but has the highest channel count. +* **Decoder:** A symmetric series of five upsampling layers (`dec_up_n`) and five processing stages (`dec_stage_n`) that reconstruct the feature map back to its original dimensions. +* **Multiplicative Skip Connections:** A key feature of this U-Net is its use of skip connections. The output of each encoder stage (`s_n`) is passed directly to the corresponding decoder stage. However, instead of being concatenated (as in a standard U-Net), it is combined via **element-wise multiplication**. This acts as a "gate," allowing the detailed features from the encoder to modulate and refine the feature maps being reconstructed by the decoder. + +### 2. The Core Component: The Sequential-Residual TDF Block + +The primary innovation of this architecture lies in its core building block, which we have identified as a **Time-Domain Frequency (TDF) Block** with a **sequential-residual** data flow. This block replaces the simple double-convolution block found in traditional U-Nets. + +The data flow within this block is crucial and was a key finding of our analysis: +1. An input tensor `x` first passes entirely through a **Time-Domain Branch**. Let the result be `time_out`. +2. The output of the time branch, `time_out`, is then fed as the input to a **Frequency-Domain Branch**. Let the result be `freq_out`. +3. The final output of the TDF Block is the sum of these two results: `result = time_out + freq_out`. This forms a residual connection where the frequency branch is learning a residual transformation on top of the time branch's output. + + + +#### 2.1. Time-Domain Branch + +* **Structure:** This branch consists of a simple sequence of three `Conv2d -> ReLU` blocks. +* **Implementation:** `nn.Sequential(Conv2d(C,C), ReLU(), Conv2d(C,C), ReLU(), Conv2d(C,C), ReLU())` +* **Purpose:** The time-domain branch uses standard 2D convolutions with a `3x3` kernel. After the initial `Transpose` operation (see Section 3.1), the input tensor's dimensions are `(Batch, Channels, Time, Freq)`. The convolutions operate across the `Time` and `Freq` dimensions, allowing them to learn localized patterns related to timbre, transients, and the short-term temporal evolution of sounds. + +#### 2.2. Frequency-Domain Branch + +* **Structure:** This branch is responsible for capturing the global harmonic structure of the audio. It consists of a `Linear -> BatchNorm2d -> ReLU -> Linear -> BatchNorm2d -> ReLU` sequence. +* **Implementation Detail - The `nn.Linear` Layer:** The key to this branch is the use of `nn.Linear` layers on a 4D tensor. + * A layer defined as `nn.Linear(in_features, out_features)` receives a 4D tensor of shape `(B, C, T, F)`, where `F` matches `in_features`. + * PyTorch's broadcasting capability means the linear transformation is applied to the last dimension only, producing an output of shape `(B, C, T, H)`, where `H` is `out_features`. No manual reshaping is needed in the `forward` pass. + * **Crucial Insight:** The weight matrix for this layer, as stored in the ONNX file, must be **transposed** before being loaded into the PyTorch `nn.Linear` layer's `.weight` parameter. +* **Implementation Detail - The `nn.BatchNorm2d` Layer:** + * This layer receives the 4D output of the `nn.Linear` layer. + * A standard `nn.BatchNorm2d(C)` operates on the channel dimension (`C`), normalizing across the `B`, `T`, and `H` dimensions. This is the correct and verified behavior. + +### 3. Key Architectural Details + +#### 3.1. Initial `Transpose` Operation + +Immediately after the initial 1x1 convolution, a `Transpose` operation permutes the tensor dimensions from the standard audio-processing format of `(Batch, Channels, Frequency, Time)` to `(Batch, Channels, Time, Frequency)`. This is a critical transformation that enables the rest of the architecture: +* It places the `Frequency` dimension last, making it compatible with the broadcasting behavior of the `nn.Linear` layers in the frequency branch. +* It makes `Time` one of the last two dimensions, allowing the `3x3 Conv2d` layers in the time branch to effectively convolve over local time steps. +A corresponding `Transpose` exists at the very end of the network to revert the tensor to its original dimension order. + +### 4. Scientific Context and Substantiation + +This architecture is a prime example of the hybrid models that have come to dominate the field of music source separation. It combines the strengths of Convolutional Neural Networks (CNNs) and, conceptually, Transformers. + +* **Hybrid Transformer Networks:** The structure of the TDF block is heavily inspired by and directly comparable to the **KUIELab-MDX-Net**, detailed in the paper "Hybrid Transformer for Music Source Separation". The paper describes a U-Net where each block has a local CNN-based module and a global Transformer-based module. Our `TimeBranch` (with its local convolutional receptive field) is the "local" module, and our `FrequencyBranch` (where `nn.Linear` layers operate across the entire frequency spectrum) is the "global" module, acting as a simplified attention mechanism. + +* **Time-Frequency Domain Processing:** The concept of using distinct operations for the time and frequency axes is a cornerstone of modern audio processing. Papers on models like Demucs and the work on Time-Frequency Convolutions (TFC-TDF) established the principle that CNNs are excellent for learning local features (like timbre), while other structures are needed to capture long-range correlations (like musical harmony), which is exactly what our `FrequencyBranch` achieves. + +**References:** + _Choi, W., Kim, J., & Chung, J. (2021). KUIELab-MDX-Net: A Two-Stream Neural Network for Music Demixing. arXiv preprint arXiv:2109.05418._ + _Défossez, A., Usunier, N., Bottou, L., & Bach, F. (2019). Music Source Separation in the Waveform Domain. arXiv preprint arXiv:1911.13254._ + _Choi, W., et al. (2020). LaSAFT: Latent-Source-Attentive Frequency-Time-Transformer for Speech Enhancement. arXiv preprint arXiv:2005.07139._ (While for speech, it details the TFC-TDF concept). + +--- + +### Conclusion + +The analyzed model is a highly sophisticated and effective U-Net architecture. Its strength lies in its **sequential-residual TDF block**, which processes the input first through a convolutional time-domain branch and then uses that output as the input for a frequency-domain branch, finally combining them with a residual connection. This, combined with multiplicative skip connections and a crucial initial `Transpose` operation, represents a state-of-the-art design pattern that has been empirically validated by top results in public leaderboards like the MDX Challenge. diff --git a/doc/MDX/MatMul.md b/doc/MDX/MatMul.md new file mode 100644 index 0000000..489e5a6 --- /dev/null +++ b/doc/MDX/MatMul.md @@ -0,0 +1,85 @@ +## Technical Brief: Converting ONNX `MatMul` to PyTorch + +### Objective + +This document outlines the definitive, verified methodology for converting a `MatMul` operation from an ONNX graph into an equivalent PyTorch `nn.Module`. This process was determined through a rigorous, iterative testing process comparing the ONNX Runtime, the `onnx2pytorch` conversion library, and manual PyTorch implementations. + +### Background + +During the reverse-engineering of an audio separation model, we encountered significant and persistent discrepancies between the output of the ONNX Runtime and our PyTorch implementations for nodes of type `MatMul`. Simple replications using `torch.matmul` or standard `nn.Linear` layers failed, even though the `onnx2pytorch` library produced a functionally correct end-to-end model. This prompted a deep, first-principles investigation to uncover the correct conversion logic. + +### Key Findings & Correct Implementation + +Through a series of targeted tests, we have proven that the conversion of an ONNX `MatMul` node, where one input is a dynamic activation tensor and the other is a static weight initializer, must be handled as follows: + +**The operation is equivalent to a `torch.nn.Linear` layer, with one critical and non-obvious caveat regarding the weight format.** + +#### 1. The `MatMul` to `nn.Linear` Equivalence + +The ONNX `MatMul` operation, when used with a weight tensor (e.g., in a typical fully-connected layer), is functionally identical to a PyTorch `nn.Linear` layer. Both perform the operation `Y = X @ W.T + b`, where `X` is the input activation and `W` is the weight matrix. PyTorch's `nn.Linear` layer correctly handles broadcasting over higher dimensions (e.g., `(Batch, Channels, Time, InFeatures)`) by applying the matrix multiplication only to the last dimension. + +#### 2. The Crucial Insight: The Weight Transposition + +The core of the conversion problem lies in how the weight tensor is stored in the ONNX file versus how it is stored internally by a PyTorch `nn.Linear` layer. + +* A PyTorch `nn.Linear(in_features, out_features)` layer stores its weight parameter as a matrix of shape `(out_features, in_features)`. +* The `onnx2pytorch` converter, and our subsequent verification tests, have definitively proven that the weight initializers associated with the `MatMul` nodes in our target ONNX file are stored in the shape `(in_features, out_features)`. + +Therefore, the **definitive rule for converting an ONNX `MatMul` node to a PyTorch `nn.Linear` layer is:** + +> 1. Create an `nn.Linear` layer with the correct `in_features` and `out_features`. +> 2. Load the corresponding raw weight tensor from the ONNX initializer. +> 3. **You MUST transpose (`.t()`) this raw weight tensor before copying it into the `.weight.data` parameter of the `nn.Linear` layer.** + +This is demonstrated by the following verified code snippet: + +```python +# Assume raw_weight_W is a numpy array from the ONNX initializer +# with shape (in_features, out_features), e.g., (3072, 384) + +in_features = raw_weight_W.shape[0] +out_features = raw_weight_W.shape[1] + +# 1. Create the PyTorch layer +my_linear_layer = nn.Linear(in_features, out_features, bias=False) + +# 2. Load the raw weight and convert to a tensor +raw_weight_tensor = torch.from_numpy(raw_weight_W) + +# 3. Transpose the raw weight and copy it into the layer +# raw_weight_tensor.t() has shape (out_features, in_features), +# which matches the internal shape of my_linear_layer.weight +my_linear_layer.weight.data.copy_(raw_weight_tensor.t()) +``` + +### Conclusions for Future Conversions + +1. **Trust but Verify:** The `__str__` representation of a converted model (e.g., from `onnx2pytorch`) can be misleading. While it may print ``, this does not reveal the crucial weight transformations that happened during its initialization. +2. **The Transpose is Key:** The primary source of error when converting `MatMul` to `nn.Linear` is failing to account for the potential mismatch in weight storage conventions. Always assume the raw ONNX weight needs to be transposed. +3. **Broadcasting Works:** Do not manually `reshape` or `flatten` a multi-dimensional tensor (e.g., `(B, C, T, F)`) before passing it to an `nn.Linear` layer. The layer's `forward` pass correctly handles broadcasting and operates on the last dimension only. Manually reshaping is a common source of bugs. +4. **Isolate and Test:** When a complex model fails, the most effective debugging strategy is to create a minimal, self-contained ONNX file for the single problematic layer and test it against the ONNX Runtime, which serves as the ultimate ground truth. + +By adhering to these principles, particularly the mandatory transposition of the raw weight initializer, a successful and numerically accurate conversion from an ONNX `MatMul` node to a PyTorch `nn.Linear` layer can be reliably achieved. + + +### Summary: Converting ONNX `MatMul` to PyTorch + +After a comprehensive, iterative debugging process, we have definitively determined the correct methodology for converting an ONNX `MatMul` node (used as a dense layer) into its equivalent PyTorch `nn.Module`. + +**The Key Finding:** + +The core of the conversion lies in resolving a mismatch between how weights are stored. The ONNX `MatMul` operation uses a weight tensor with a shape of `(in_features, out_features)`. In contrast, PyTorch's `nn.Linear` layer stores its internal weight parameter with the shape `(out_features, in_features)`. + +Therefore, a direct copy is impossible. The solution, proven through our tests, is that **the raw weight tensor from the ONNX file must be transposed before being loaded into the PyTorch `nn.Linear` layer.** + +**The Correct, Verified Procedure:** + +1. **Identify the Node:** For an ONNX `MatMul` node with an activation input `A` and a weight initializer `W`. +2. **Determine Layer Shape:** Inspect the shape of the raw weight tensor `W`. Its shape `(K, M)` corresponds to `(in_features, out_features)`. +3. **Create PyTorch Layer:** Instantiate a standard `torch.nn.Linear` layer: `layer = nn.Linear(in_features=K, out_features=M, bias=False)`. +4. **Load and Transpose Weight:** Load the raw weight tensor `W` and copy its transpose into the layer's parameter: `layer.weight.data.copy_(W.t())`. +5. **Forward Pass:** Pass the multi-dimensional activation tensor `A` directly to the layer. PyTorch's broadcasting will correctly apply the linear transformation to the last dimension without needing any manual `reshape` operations. + +This procedure creates a PyTorch layer that is a numerically identical replica of the ONNX Runtime's `MatMul` operation, accounting for the subtle but critical difference in weight storage conventions. + +Recommendation: store transposed values in the safetensors file. diff --git a/doc/MDX/maps/Kim_Vocal_2.arch.txt b/doc/MDX/maps/Kim_Vocal_2.arch.txt new file mode 100644 index 0000000..461eccd --- /dev/null +++ b/doc/MDX/maps/Kim_Vocal_2.arch.txt @@ -0,0 +1,343 @@ +ConvertModel( + (Conv_684): Conv2d(4, 48, kernel_size=(1, 1), stride=(1, 1)) + (Relu_447): ReLU(inplace=True) + (Transpose_448): Transpose() + +-------------------------------------------------- +Stage 1 TDF_Block +-------------------------------------------------- + + -- TDF_Block s1 + + (Conv_687): Conv2d(48, 48, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_451): ReLU(inplace=True) + (Conv_690): Conv2d(48, 48, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_454): ReLU(inplace=True) + (Conv_693): Conv2d(48, 48, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_457): ReLU(inplace=True) + (MatMul_459): Linear(in_features=3072, out_features=384, bias=False) + (BatchNormalization_460): BatchNormWrapper( + (bnu): BatchNormUnsafe(48, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_461): ReLU(inplace=True) + (MatMul_463): Linear(in_features=384, out_features=3072, bias=False) + (BatchNormalization_464): BatchNormWrapper( + (bnu): BatchNormUnsafe(48, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_465): ReLU(inplace=True) + (Add_466): Add() + + --- Down for s1 (d1) + + (Conv_696): Conv2d(48, 96, kernel_size=(2, 2), stride=(2, 2)) + (Relu_469): ReLU(inplace=True) + +-------------------------------------------------- +Stage 2 +-------------------------------------------------- + + -- TDF_Block s2 + + (Conv_699): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_472): ReLU(inplace=True) + (Conv_702): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_475): ReLU(inplace=True) + (Conv_705): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_478): ReLU(inplace=True) + (MatMul_480): Linear(in_features=1536, out_features=192, bias=False) + (BatchNormalization_481): BatchNormWrapper( + (bnu): BatchNormUnsafe(96, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_482): ReLU(inplace=True) + (MatMul_484): Linear(in_features=192, out_features=1536, bias=False) + (BatchNormalization_485): BatchNormWrapper( + (bnu): BatchNormUnsafe(96, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_486): ReLU(inplace=True) + (Add_487): Add() + + --- Down for s2 (d2) + + (Conv_708): Conv2d(96, 144, kernel_size=(2, 2), stride=(2, 2)) + (Relu_490): ReLU(inplace=True) + +-------------------------------------------------- +Stage 3 +-------------------------------------------------- + + -- TDF_Block s3 + + (Conv_711): Conv2d(144, 144, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_493): ReLU(inplace=True) + (Conv_714): Conv2d(144, 144, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_496): ReLU(inplace=True) + (Conv_717): Conv2d(144, 144, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_499): ReLU(inplace=True) + (MatMul_501): Linear(in_features=768, out_features=96, bias=False) + (BatchNormalization_502): BatchNormWrapper( + (bnu): BatchNormUnsafe(144, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_503): ReLU(inplace=True) + (MatMul_505): Linear(in_features=96, out_features=768, bias=False) + (BatchNormalization_506): BatchNormWrapper( + (bnu): BatchNormUnsafe(144, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_507): ReLU(inplace=True) + (Add_508): Add() + + --- Down for s3 (d3) + + (Conv_720): Conv2d(144, 192, kernel_size=(2, 2), stride=(2, 2)) + (Relu_511): ReLU(inplace=True) + +-------------------------------------------------- +Stage 4 +-------------------------------------------------- + + -- TDF_Block s4 + + (Conv_723): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_514): ReLU(inplace=True) + (Conv_726): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_517): ReLU(inplace=True) + (Conv_729): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_520): ReLU(inplace=True) + (MatMul_522): Linear(in_features=384, out_features=48, bias=False) + (BatchNormalization_523): BatchNormWrapper( + (bnu): BatchNormUnsafe(192, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_524): ReLU(inplace=True) + (MatMul_526): Linear(in_features=48, out_features=384, bias=False) + (BatchNormalization_527): BatchNormWrapper( + (bnu): BatchNormUnsafe(192, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_528): ReLU(inplace=True) + (Add_529): Add() + + --- Down for s4 (d4) + + (Conv_732): Conv2d(192, 240, kernel_size=(2, 2), stride=(2, 2)) + (Relu_532): ReLU(inplace=True) + +-------------------------------------------------- +Stage 5 +-------------------------------------------------- + + -- TDF_Block s5 + + (Conv_735): Conv2d(240, 240, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_535): ReLU(inplace=True) + (Conv_738): Conv2d(240, 240, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_538): ReLU(inplace=True) + (Conv_741): Conv2d(240, 240, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_541): ReLU(inplace=True) + (MatMul_543): Linear(in_features=192, out_features=24, bias=False) + (BatchNormalization_544): BatchNormWrapper( + (bnu): BatchNormUnsafe(240, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_545): ReLU(inplace=True) + (MatMul_547): Linear(in_features=24, out_features=192, bias=False) + (BatchNormalization_548): BatchNormWrapper( + (bnu): BatchNormUnsafe(240, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_549): ReLU(inplace=True) + (Add_550): Add() + + --- Down for s5 (d5) + + (Conv_744): Conv2d(240, 288, kernel_size=(2, 2), stride=(2, 2)) + (Relu_553): ReLU(inplace=True) + +-------------------------------------------------- +Bottleneck +-------------------------------------------------- + + (Conv_747): Conv2d(288, 288, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_556): ReLU(inplace=True) + (Conv_750): Conv2d(288, 288, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_559): ReLU(inplace=True) + (Conv_753): Conv2d(288, 288, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_562): ReLU(inplace=True) + (MatMul_564): Linear(in_features=96, out_features=12, bias=False) + (BatchNormalization_565): BatchNormWrapper( + (bnu): BatchNormUnsafe(288, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_566): ReLU(inplace=True) + (MatMul_568): Linear(in_features=12, out_features=96, bias=False) + (BatchNormalization_569): BatchNormWrapper( + (bnu): BatchNormUnsafe(288, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_570): ReLU(inplace=True) + (Add_571): Add() + +-------------------------------------------------- +Stage 5 +-------------------------------------------------- + + -- Up 5 + + (ConvTranspose_572): ConvTranspose2d(288, 240, kernel_size=(2, 2), stride=(2, 2)) + (BatchNormalization_573): BatchNormWrapper( + (bnu): BatchNormUnsafe(240, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_574): ReLU(inplace=True) + (Mul_575): mul() + + -- TDF_Block 5 + + (Conv_756): Conv2d(240, 240, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_578): ReLU(inplace=True) + (Conv_759): Conv2d(240, 240, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_581): ReLU(inplace=True) + (Conv_762): Conv2d(240, 240, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_584): ReLU(inplace=True) + (MatMul_586): Linear(in_features=192, out_features=24, bias=False) + (BatchNormalization_587): BatchNormWrapper( + (bnu): BatchNormUnsafe(240, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_588): ReLU(inplace=True) + (MatMul_590): Linear(in_features=24, out_features=192, bias=False) + (BatchNormalization_591): BatchNormWrapper( + (bnu): BatchNormUnsafe(240, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_592): ReLU(inplace=True) + (Add_593): Add() + +-------------------------------------------------- +Stage 4 +-------------------------------------------------- + + -- Up 4 + + (ConvTranspose_594): ConvTranspose2d(240, 192, kernel_size=(2, 2), stride=(2, 2)) + (BatchNormalization_595): BatchNormWrapper( + (bnu): BatchNormUnsafe(192, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_596): ReLU(inplace=True) + (Mul_597): mul() + + -- TDF_Block 4 + + (Conv_765): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_600): ReLU(inplace=True) + (Conv_768): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_603): ReLU(inplace=True) + (Conv_771): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_606): ReLU(inplace=True) + (MatMul_608): Linear(in_features=384, out_features=48, bias=False) + (BatchNormalization_609): BatchNormWrapper( + (bnu): BatchNormUnsafe(192, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_610): ReLU(inplace=True) + (MatMul_612): Linear(in_features=48, out_features=384, bias=False) + (BatchNormalization_613): BatchNormWrapper( + (bnu): BatchNormUnsafe(192, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_614): ReLU(inplace=True) + (Add_615): Add() + +-------------------------------------------------- +Stage 3 +-------------------------------------------------- + + -- Up 3 + + (ConvTranspose_616): ConvTranspose2d(192, 144, kernel_size=(2, 2), stride=(2, 2)) + (BatchNormalization_617): BatchNormWrapper( + (bnu): BatchNormUnsafe(144, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_618): ReLU(inplace=True) + (Mul_619): mul() + + -- TDF_Block 3 + + (Conv_774): Conv2d(144, 144, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_622): ReLU(inplace=True) + (Conv_777): Conv2d(144, 144, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_625): ReLU(inplace=True) + (Conv_780): Conv2d(144, 144, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_628): ReLU(inplace=True) + (MatMul_630): Linear(in_features=768, out_features=96, bias=False) + (BatchNormalization_631): BatchNormWrapper( + (bnu): BatchNormUnsafe(144, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_632): ReLU(inplace=True) + (MatMul_634): Linear(in_features=96, out_features=768, bias=False) + (BatchNormalization_635): BatchNormWrapper( + (bnu): BatchNormUnsafe(144, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_636): ReLU(inplace=True) + (Add_637): Add() + +-------------------------------------------------- +Stage 2 +-------------------------------------------------- + + -- Up 2 + + (ConvTranspose_638): ConvTranspose2d(144, 96, kernel_size=(2, 2), stride=(2, 2)) + (BatchNormalization_639): BatchNormWrapper( + (bnu): BatchNormUnsafe(96, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_640): ReLU(inplace=True) + (Mul_641): mul() + + -- TDF_Block 2 + + (Conv_783): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_644): ReLU(inplace=True) + (Conv_786): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_647): ReLU(inplace=True) + (Conv_789): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_650): ReLU(inplace=True) + (MatMul_652): Linear(in_features=1536, out_features=192, bias=False) + (BatchNormalization_653): BatchNormWrapper( + (bnu): BatchNormUnsafe(96, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_654): ReLU(inplace=True) + (MatMul_656): Linear(in_features=192, out_features=1536, bias=False) + (BatchNormalization_657): BatchNormWrapper( + (bnu): BatchNormUnsafe(96, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_658): ReLU(inplace=True) + (Add_659): Add() + +-------------------------------------------------- +Stage 1 +-------------------------------------------------- + + -- Up 1 + + (ConvTranspose_660): ConvTranspose2d(96, 48, kernel_size=(2, 2), stride=(2, 2)) + (BatchNormalization_661): BatchNormWrapper( + (bnu): BatchNormUnsafe(48, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_662): ReLU(inplace=True) + (Mul_663): mul() + (Conv_792): Conv2d(48, 48, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_666): ReLU(inplace=True) + (Conv_795): Conv2d(48, 48, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_669): ReLU(inplace=True) + (Conv_798): Conv2d(48, 48, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) + (Relu_672): ReLU(inplace=True) + (MatMul_674): Linear(in_features=3072, out_features=384, bias=False) + (BatchNormalization_675): BatchNormWrapper( + (bnu): BatchNormUnsafe(48, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_676): ReLU(inplace=True) + (MatMul_678): Linear(in_features=384, out_features=3072, bias=False) + (BatchNormalization_679): BatchNormWrapper( + (bnu): BatchNormUnsafe(48, eps=9.999999747378752e-06, momentum=0.8999999761581421, affine=True, track_running_stats=True) + ) + (Relu_680): ReLU(inplace=True) + (Add_681): Add() + + -- TDF_Block 2 + +-------------------------------------------------- +Final +-------------------------------------------------- + + (Transpose_682): Transpose() + (Conv_output): Conv2d(48, 4, kernel_size=(1, 1), stride=(1, 1)) +) diff --git a/doc/MDX/maps/Kim_Vocal_2_class.map b/doc/MDX/maps/Kim_Vocal_2_class.map new file mode 100644 index 0000000..120863e --- /dev/null +++ b/doc/MDX/maps/Kim_Vocal_2_class.map @@ -0,0 +1,201 @@ +Importing module 'MDX_Net' from 'modules'... +Successfully imported class 'MDX_Net'. +--- PyTorch Model Structure --- +model.MDX_Net: MDX_Net + model.initial_conv: Conv2d + model.initial_relu: ReLU + model.initial_transpose: Transpose + model.enc_stages.ModuleList: ModuleList + model.enc_stages.0.TDF_Block: TDF_Block + model.enc_stages.0.time_branch.TimeBranch: TimeBranch + model.enc_stages.0.time_branch.sequence.0: Conv2d + model.enc_stages.0.time_branch.sequence.1: ReLU + model.enc_stages.0.time_branch.sequence.2: Conv2d + model.enc_stages.0.time_branch.sequence.3: ReLU + model.enc_stages.0.time_branch.sequence.4: Conv2d + model.enc_stages.0.time_branch.sequence.5: ReLU + model.enc_stages.0.freq_branch.FrequencyBranch: FrequencyBranch + model.enc_stages.0.freq_branch.sequence.0: Linear + model.enc_stages.0.freq_branch.sequence.1: BatchNorm2d + model.enc_stages.0.freq_branch.sequence.2: ReLU + model.enc_stages.0.freq_branch.sequence.3: Linear + model.enc_stages.0.freq_branch.sequence.4: BatchNorm2d + model.enc_stages.0.freq_branch.sequence.5: ReLU + model.enc_stages.1.0: Conv2d + model.enc_stages.1.1: ReLU + model.enc_stages.2.TDF_Block: TDF_Block + model.enc_stages.2.time_branch.TimeBranch: TimeBranch + model.enc_stages.2.time_branch.sequence.0: Conv2d + model.enc_stages.2.time_branch.sequence.1: ReLU + model.enc_stages.2.time_branch.sequence.2: Conv2d + model.enc_stages.2.time_branch.sequence.3: ReLU + model.enc_stages.2.time_branch.sequence.4: Conv2d + model.enc_stages.2.time_branch.sequence.5: ReLU + model.enc_stages.2.freq_branch.FrequencyBranch: FrequencyBranch + model.enc_stages.2.freq_branch.sequence.0: Linear + model.enc_stages.2.freq_branch.sequence.1: BatchNorm2d + model.enc_stages.2.freq_branch.sequence.2: ReLU + model.enc_stages.2.freq_branch.sequence.3: Linear + model.enc_stages.2.freq_branch.sequence.4: BatchNorm2d + model.enc_stages.2.freq_branch.sequence.5: ReLU + model.enc_stages.3.0: Conv2d + model.enc_stages.3.1: ReLU + model.enc_stages.4.TDF_Block: TDF_Block + model.enc_stages.4.time_branch.TimeBranch: TimeBranch + model.enc_stages.4.time_branch.sequence.0: Conv2d + model.enc_stages.4.time_branch.sequence.1: ReLU + model.enc_stages.4.time_branch.sequence.2: Conv2d + model.enc_stages.4.time_branch.sequence.3: ReLU + model.enc_stages.4.time_branch.sequence.4: Conv2d + model.enc_stages.4.time_branch.sequence.5: ReLU + model.enc_stages.4.freq_branch.FrequencyBranch: FrequencyBranch + model.enc_stages.4.freq_branch.sequence.0: Linear + model.enc_stages.4.freq_branch.sequence.1: BatchNorm2d + model.enc_stages.4.freq_branch.sequence.2: ReLU + model.enc_stages.4.freq_branch.sequence.3: Linear + model.enc_stages.4.freq_branch.sequence.4: BatchNorm2d + model.enc_stages.4.freq_branch.sequence.5: ReLU + model.enc_stages.5.0: Conv2d + model.enc_stages.5.1: ReLU + model.enc_stages.6.TDF_Block: TDF_Block + model.enc_stages.6.time_branch.TimeBranch: TimeBranch + model.enc_stages.6.time_branch.sequence.0: Conv2d + model.enc_stages.6.time_branch.sequence.1: ReLU + model.enc_stages.6.time_branch.sequence.2: Conv2d + model.enc_stages.6.time_branch.sequence.3: ReLU + model.enc_stages.6.time_branch.sequence.4: Conv2d + model.enc_stages.6.time_branch.sequence.5: ReLU + model.enc_stages.6.freq_branch.FrequencyBranch: FrequencyBranch + model.enc_stages.6.freq_branch.sequence.0: Linear + model.enc_stages.6.freq_branch.sequence.1: BatchNorm2d + model.enc_stages.6.freq_branch.sequence.2: ReLU + model.enc_stages.6.freq_branch.sequence.3: Linear + model.enc_stages.6.freq_branch.sequence.4: BatchNorm2d + model.enc_stages.6.freq_branch.sequence.5: ReLU + model.enc_stages.7.0: Conv2d + model.enc_stages.7.1: ReLU + model.enc_stages.8.TDF_Block: TDF_Block + model.enc_stages.8.time_branch.TimeBranch: TimeBranch + model.enc_stages.8.time_branch.sequence.0: Conv2d + model.enc_stages.8.time_branch.sequence.1: ReLU + model.enc_stages.8.time_branch.sequence.2: Conv2d + model.enc_stages.8.time_branch.sequence.3: ReLU + model.enc_stages.8.time_branch.sequence.4: Conv2d + model.enc_stages.8.time_branch.sequence.5: ReLU + model.enc_stages.8.freq_branch.FrequencyBranch: FrequencyBranch + model.enc_stages.8.freq_branch.sequence.0: Linear + model.enc_stages.8.freq_branch.sequence.1: BatchNorm2d + model.enc_stages.8.freq_branch.sequence.2: ReLU + model.enc_stages.8.freq_branch.sequence.3: Linear + model.enc_stages.8.freq_branch.sequence.4: BatchNorm2d + model.enc_stages.8.freq_branch.sequence.5: ReLU + model.enc_stages.9.0: Conv2d + model.enc_stages.9.1: ReLU + model.bottleneck.TDF_Block: TDF_Block + model.bottleneck.time_branch.TimeBranch: TimeBranch + model.bottleneck.time_branch.sequence.0: Conv2d + model.bottleneck.time_branch.sequence.1: ReLU + model.bottleneck.time_branch.sequence.2: Conv2d + model.bottleneck.time_branch.sequence.3: ReLU + model.bottleneck.time_branch.sequence.4: Conv2d + model.bottleneck.time_branch.sequence.5: ReLU + model.bottleneck.freq_branch.FrequencyBranch: FrequencyBranch + model.bottleneck.freq_branch.sequence.0: Linear + model.bottleneck.freq_branch.sequence.1: BatchNorm2d + model.bottleneck.freq_branch.sequence.2: ReLU + model.bottleneck.freq_branch.sequence.3: Linear + model.bottleneck.freq_branch.sequence.4: BatchNorm2d + model.bottleneck.freq_branch.sequence.5: ReLU + model.dec_stages.ModuleList: ModuleList + model.dec_stages.0.0: ConvTranspose2d + model.dec_stages.0.1: BatchNorm2d + model.dec_stages.0.2: ReLU + model.dec_stages.1.TDF_Block: TDF_Block + model.dec_stages.1.time_branch.TimeBranch: TimeBranch + model.dec_stages.1.time_branch.sequence.0: Conv2d + model.dec_stages.1.time_branch.sequence.1: ReLU + model.dec_stages.1.time_branch.sequence.2: Conv2d + model.dec_stages.1.time_branch.sequence.3: ReLU + model.dec_stages.1.time_branch.sequence.4: Conv2d + model.dec_stages.1.time_branch.sequence.5: ReLU + model.dec_stages.1.freq_branch.FrequencyBranch: FrequencyBranch + model.dec_stages.1.freq_branch.sequence.0: Linear + model.dec_stages.1.freq_branch.sequence.1: BatchNorm2d + model.dec_stages.1.freq_branch.sequence.2: ReLU + model.dec_stages.1.freq_branch.sequence.3: Linear + model.dec_stages.1.freq_branch.sequence.4: BatchNorm2d + model.dec_stages.1.freq_branch.sequence.5: ReLU + model.dec_stages.2.0: ConvTranspose2d + model.dec_stages.2.1: BatchNorm2d + model.dec_stages.2.2: ReLU + model.dec_stages.3.TDF_Block: TDF_Block + model.dec_stages.3.time_branch.TimeBranch: TimeBranch + model.dec_stages.3.time_branch.sequence.0: Conv2d + model.dec_stages.3.time_branch.sequence.1: ReLU + model.dec_stages.3.time_branch.sequence.2: Conv2d + model.dec_stages.3.time_branch.sequence.3: ReLU + model.dec_stages.3.time_branch.sequence.4: Conv2d + model.dec_stages.3.time_branch.sequence.5: ReLU + model.dec_stages.3.freq_branch.FrequencyBranch: FrequencyBranch + model.dec_stages.3.freq_branch.sequence.0: Linear + model.dec_stages.3.freq_branch.sequence.1: BatchNorm2d + model.dec_stages.3.freq_branch.sequence.2: ReLU + model.dec_stages.3.freq_branch.sequence.3: Linear + model.dec_stages.3.freq_branch.sequence.4: BatchNorm2d + model.dec_stages.3.freq_branch.sequence.5: ReLU + model.dec_stages.4.0: ConvTranspose2d + model.dec_stages.4.1: BatchNorm2d + model.dec_stages.4.2: ReLU + model.dec_stages.5.TDF_Block: TDF_Block + model.dec_stages.5.time_branch.TimeBranch: TimeBranch + model.dec_stages.5.time_branch.sequence.0: Conv2d + model.dec_stages.5.time_branch.sequence.1: ReLU + model.dec_stages.5.time_branch.sequence.2: Conv2d + model.dec_stages.5.time_branch.sequence.3: ReLU + model.dec_stages.5.time_branch.sequence.4: Conv2d + model.dec_stages.5.time_branch.sequence.5: ReLU + model.dec_stages.5.freq_branch.FrequencyBranch: FrequencyBranch + model.dec_stages.5.freq_branch.sequence.0: Linear + model.dec_stages.5.freq_branch.sequence.1: BatchNorm2d + model.dec_stages.5.freq_branch.sequence.2: ReLU + model.dec_stages.5.freq_branch.sequence.3: Linear + model.dec_stages.5.freq_branch.sequence.4: BatchNorm2d + model.dec_stages.5.freq_branch.sequence.5: ReLU + model.dec_stages.6.0: ConvTranspose2d + model.dec_stages.6.1: BatchNorm2d + model.dec_stages.6.2: ReLU + model.dec_stages.7.TDF_Block: TDF_Block + model.dec_stages.7.time_branch.TimeBranch: TimeBranch + model.dec_stages.7.time_branch.sequence.0: Conv2d + model.dec_stages.7.time_branch.sequence.1: ReLU + model.dec_stages.7.time_branch.sequence.2: Conv2d + model.dec_stages.7.time_branch.sequence.3: ReLU + model.dec_stages.7.time_branch.sequence.4: Conv2d + model.dec_stages.7.time_branch.sequence.5: ReLU + model.dec_stages.7.freq_branch.FrequencyBranch: FrequencyBranch + model.dec_stages.7.freq_branch.sequence.0: Linear + model.dec_stages.7.freq_branch.sequence.1: BatchNorm2d + model.dec_stages.7.freq_branch.sequence.2: ReLU + model.dec_stages.7.freq_branch.sequence.3: Linear + model.dec_stages.7.freq_branch.sequence.4: BatchNorm2d + model.dec_stages.7.freq_branch.sequence.5: ReLU + model.dec_stages.8.0: ConvTranspose2d + model.dec_stages.8.1: BatchNorm2d + model.dec_stages.8.2: ReLU + model.dec_stages.9.TDF_Block: TDF_Block + model.dec_stages.9.time_branch.TimeBranch: TimeBranch + model.dec_stages.9.time_branch.sequence.0: Conv2d + model.dec_stages.9.time_branch.sequence.1: ReLU + model.dec_stages.9.time_branch.sequence.2: Conv2d + model.dec_stages.9.time_branch.sequence.3: ReLU + model.dec_stages.9.time_branch.sequence.4: Conv2d + model.dec_stages.9.time_branch.sequence.5: ReLU + model.dec_stages.9.freq_branch.FrequencyBranch: FrequencyBranch + model.dec_stages.9.freq_branch.sequence.0: Linear + model.dec_stages.9.freq_branch.sequence.1: BatchNorm2d + model.dec_stages.9.freq_branch.sequence.2: ReLU + model.dec_stages.9.freq_branch.sequence.3: Linear + model.dec_stages.9.freq_branch.sequence.4: BatchNorm2d + model.dec_stages.9.freq_branch.sequence.5: ReLU + model.final_transpose: Transpose + model.final_conv: Conv2d diff --git a/doc/MDX/maps/Kim_Vocal_2_onnx.map b/doc/MDX/maps/Kim_Vocal_2_onnx.map new file mode 100644 index 0000000..198a800 --- /dev/null +++ b/doc/MDX/maps/Kim_Vocal_2_onnx.map @@ -0,0 +1,642 @@ +[000] Conv_0 type: Conv W inputs: ['input', '685', '686'] outputs: ['684'] + - [0] from This is the input of the net + - [1] from weight of shape torch.Size([48, 4, 1, 1]) [name: 685] + - [2] from weight of shape torch.Size([48]) [name: 686] +[001] Relu_1 type: Relu inputs: ['684'] outputs: ['447'] + - [0] from output of Conv_0 [name: 684] +[002] Transpose_2 type: Transpose inputs: ['447'] outputs: ['448'] + - [0] from output of Relu_1 [name: 447] + +------------------------------------------- +Stage 1 +------------------------------------------- + +[003] Conv_3 type: Conv W inputs: ['448', '688', '689'] outputs: ['687'] + - [0] from output of Transpose_2 [name: 448] + - [1] from weight of shape torch.Size([48, 48, 3, 3]) [name: 688] + - [2] from weight of shape torch.Size([48]) [name: 689] +[004] Relu_4 type: Relu inputs: ['687'] outputs: ['451'] + - [0] from output of Conv_3 [name: 687] +[005] Conv_5 type: Conv W inputs: ['451', '691', '692'] outputs: ['690'] + - [0] from output of Relu_4 [name: 451] + - [1] from weight of shape torch.Size([48, 48, 3, 3]) [name: 691] + - [2] from weight of shape torch.Size([48]) [name: 692] +[006] Relu_6 type: Relu inputs: ['690'] outputs: ['454'] + - [0] from output of Conv_5 [name: 690] +[007] Conv_7 type: Conv W inputs: ['454', '694', '695'] outputs: ['693'] + - [0] from output of Relu_6 [name: 454] + - [1] from weight of shape torch.Size([48, 48, 3, 3]) [name: 694] + - [2] from weight of shape torch.Size([48]) [name: 695] +[008] Relu_8 type: Relu inputs: ['693'] outputs: ['457'] + - [0] from output of Conv_7 [name: 693] + +[009] MatMul_9 type: MatMul W inputs: ['457', '801'] outputs: ['459'] + - [0] from output of Relu_8 [name: 457] + - [1] from weight of shape torch.Size([3072, 384]) [name: 801] +[010] BatchNormalization_10 type: BatchNormalization W inputs: ['459', 'encoding_blocks.0.tdf.1.weight', 'encoding_blocks.0.tdf.1.bias', 'encoding_blocks.0.tdf.1.running_mean', 'encoding_blocks.0.tdf.1.running_var'] outputs: ['460'] + - [0] from output of MatMul_9 [name: 459] + - [1] from weight of shape torch.Size([48]) [name: encoding_blocks.0.tdf.1.weight] + - [2] from weight of shape torch.Size([48]) [name: encoding_blocks.0.tdf.1.bias] + - [3] from weight of shape torch.Size([48]) [name: encoding_blocks.0.tdf.1.running_mean] + - [4] from weight of shape torch.Size([48]) [name: encoding_blocks.0.tdf.1.running_var] +[011] Relu_11 type: Relu inputs: ['460'] outputs: ['461'] + - [0] from output of BatchNormalization_10 [name: 460] +[012] MatMul_12 type: MatMul W inputs: ['461', '802'] outputs: ['463'] + - [0] from output of Relu_11 [name: 461] + - [1] from weight of shape torch.Size([384, 3072]) [name: 802] +[013] BatchNormalization_13 type: BatchNormalization W inputs: ['463', 'encoding_blocks.0.tdf.4.weight', 'encoding_blocks.0.tdf.4.bias', 'encoding_blocks.0.tdf.4.running_mean', 'encoding_blocks.0.tdf.4.running_var'] outputs: ['464'] + - [0] from output of MatMul_12 [name: 463] + - [1] from weight of shape torch.Size([48]) [name: encoding_blocks.0.tdf.4.weight] + - [2] from weight of shape torch.Size([48]) [name: encoding_blocks.0.tdf.4.bias] + - [3] from weight of shape torch.Size([48]) [name: encoding_blocks.0.tdf.4.running_mean] + - [4] from weight of shape torch.Size([48]) [name: encoding_blocks.0.tdf.4.running_var] +[014] Relu_14 type: Relu inputs: ['464'] outputs: ['465'] + - [0] from output of BatchNormalization_13 [name: 464] + +[015] Add_15 type: Add inputs: ['457', '465'] outputs: ['466'] + - [0] from output of Relu_8 [name: 457] + - [1] from output of Relu_14 [name: 465] + +DOWN 1 + +[016] Conv_16 type: Conv W inputs: ['466', '697', '698'] outputs: ['696'] + - [0] from output of Add_15 [name: 466] + - [1] from weight of shape torch.Size([96, 48, 2, 2]) [name: 697] + - [2] from weight of shape torch.Size([96]) [name: 698] +[017] Relu_17 type: Relu inputs: ['696'] outputs: ['469'] + - [0] from output of Conv_16 [name: 696] + +------------------------------------------- +Stage 2 +------------------------------------------- + +[018] Conv_18 type: Conv W inputs: ['469', '700', '701'] outputs: ['699'] + - [0] from output of Relu_17 [name: 469] + - [1] from weight of shape torch.Size([96, 96, 3, 3]) [name: 700] + - [2] from weight of shape torch.Size([96]) [name: 701] +[019] Relu_19 type: Relu inputs: ['699'] outputs: ['472'] + - [0] from output of Conv_18 [name: 699] +[020] Conv_20 type: Conv W inputs: ['472', '703', '704'] outputs: ['702'] + - [0] from output of Relu_19 [name: 472] + - [1] from weight of shape torch.Size([96, 96, 3, 3]) [name: 703] + - [2] from weight of shape torch.Size([96]) [name: 704] +[021] Relu_21 type: Relu inputs: ['702'] outputs: ['475'] + - [0] from output of Conv_20 [name: 702] +[022] Conv_22 type: Conv W inputs: ['475', '706', '707'] outputs: ['705'] + - [0] from output of Relu_21 [name: 475] + - [1] from weight of shape torch.Size([96, 96, 3, 3]) [name: 706] + - [2] from weight of shape torch.Size([96]) [name: 707] +[023] Relu_23 type: Relu inputs: ['705'] outputs: ['478'] + - [0] from output of Conv_22 [name: 705] + +[024] MatMul_24 type: MatMul W inputs: ['478', '803'] outputs: ['480'] + - [0] from output of Relu_23 [name: 478] + - [1] from weight of shape torch.Size([1536, 192]) [name: 803] +[025] BatchNormalization_25 type: BatchNormalization W inputs: ['480', 'encoding_blocks.1.tdf.1.weight', 'encoding_blocks.1.tdf.1.bias', 'encoding_blocks.1.tdf.1.running_mean', 'encoding_blocks.1.tdf.1.running_var'] outputs: ['481'] + - [0] from output of MatMul_24 [name: 480] + - [1] from weight of shape torch.Size([96]) [name: encoding_blocks.1.tdf.1.weight] + - [2] from weight of shape torch.Size([96]) [name: encoding_blocks.1.tdf.1.bias] + - [3] from weight of shape torch.Size([96]) [name: encoding_blocks.1.tdf.1.running_mean] + - [4] from weight of shape torch.Size([96]) [name: encoding_blocks.1.tdf.1.running_var] +[026] Relu_26 type: Relu inputs: ['481'] outputs: ['482'] + - [0] from output of BatchNormalization_25 [name: 481] +[027] MatMul_27 type: MatMul W inputs: ['482', '804'] outputs: ['484'] + - [0] from output of Relu_26 [name: 482] + - [1] from weight of shape torch.Size([192, 1536]) [name: 804] +[028] BatchNormalization_28 type: BatchNormalization W inputs: ['484', 'encoding_blocks.1.tdf.4.weight', 'encoding_blocks.1.tdf.4.bias', 'encoding_blocks.1.tdf.4.running_mean', 'encoding_blocks.1.tdf.4.running_var'] outputs: ['485'] + - [0] from output of MatMul_27 [name: 484] + - [1] from weight of shape torch.Size([96]) [name: encoding_blocks.1.tdf.4.weight] + - [2] from weight of shape torch.Size([96]) [name: encoding_blocks.1.tdf.4.bias] + - [3] from weight of shape torch.Size([96]) [name: encoding_blocks.1.tdf.4.running_mean] + - [4] from weight of shape torch.Size([96]) [name: encoding_blocks.1.tdf.4.running_var] +[029] Relu_29 type: Relu inputs: ['485'] outputs: ['486'] + - [0] from output of BatchNormalization_28 [name: 485] + +[030] Add_30 type: Add inputs: ['478', '486'] outputs: ['487'] + - [0] from output of Relu_23 [name: 478] + - [1] from output of Relu_29 [name: 486] + +DOWN 2 + +[031] Conv_31 type: Conv W inputs: ['487', '709', '710'] outputs: ['708'] + - [0] from output of Add_30 [name: 487] + - [1] from weight of shape torch.Size([144, 96, 2, 2]) [name: 709] + - [2] from weight of shape torch.Size([144]) [name: 710] +[032] Relu_32 type: Relu inputs: ['708'] outputs: ['490'] + - [0] from output of Conv_31 [name: 708] + +------------------------------------------- +Stage 3 +------------------------------------------- + +[033] Conv_33 type: Conv W inputs: ['490', '712', '713'] outputs: ['711'] + - [0] from output of Relu_32 [name: 490] + - [1] from weight of shape torch.Size([144, 144, 3, 3]) [name: 712] + - [2] from weight of shape torch.Size([144]) [name: 713] +[034] Relu_34 type: Relu inputs: ['711'] outputs: ['493'] + - [0] from output of Conv_33 [name: 711] +[035] Conv_35 type: Conv W inputs: ['493', '715', '716'] outputs: ['714'] + - [0] from output of Relu_34 [name: 493] + - [1] from weight of shape torch.Size([144, 144, 3, 3]) [name: 715] + - [2] from weight of shape torch.Size([144]) [name: 716] +[036] Relu_36 type: Relu inputs: ['714'] outputs: ['496'] + - [0] from output of Conv_35 [name: 714] +[037] Conv_37 type: Conv W inputs: ['496', '718', '719'] outputs: ['717'] + - [0] from output of Relu_36 [name: 496] + - [1] from weight of shape torch.Size([144, 144, 3, 3]) [name: 718] + - [2] from weight of shape torch.Size([144]) [name: 719] +[038] Relu_38 type: Relu inputs: ['717'] outputs: ['499'] + - [0] from output of Conv_37 [name: 717] +[039] MatMul_39 type: MatMul W inputs: ['499', '805'] outputs: ['501'] + - [0] from output of Relu_38 [name: 499] + - [1] from weight of shape torch.Size([768, 96]) [name: 805] +[040] BatchNormalization_40 type: BatchNormalization W inputs: ['501', 'encoding_blocks.2.tdf.1.weight', 'encoding_blocks.2.tdf.1.bias', 'encoding_blocks.2.tdf.1.running_mean', 'encoding_blocks.2.tdf.1.running_var'] outputs: ['502'] + - [0] from output of MatMul_39 [name: 501] + - [1] from weight of shape torch.Size([144]) [name: encoding_blocks.2.tdf.1.weight] + - [2] from weight of shape torch.Size([144]) [name: encoding_blocks.2.tdf.1.bias] + - [3] from weight of shape torch.Size([144]) [name: encoding_blocks.2.tdf.1.running_mean] + - [4] from weight of shape torch.Size([144]) [name: encoding_blocks.2.tdf.1.running_var] +[041] Relu_41 type: Relu inputs: ['502'] outputs: ['503'] + - [0] from output of BatchNormalization_40 [name: 502] +[042] MatMul_42 type: MatMul W inputs: ['503', '806'] outputs: ['505'] + - [0] from output of Relu_41 [name: 503] + - [1] from weight of shape torch.Size([96, 768]) [name: 806] +[043] BatchNormalization_43 type: BatchNormalization W inputs: ['505', 'encoding_blocks.2.tdf.4.weight', 'encoding_blocks.2.tdf.4.bias', 'encoding_blocks.2.tdf.4.running_mean', 'encoding_blocks.2.tdf.4.running_var'] outputs: ['506'] + - [0] from output of MatMul_42 [name: 505] + - [1] from weight of shape torch.Size([144]) [name: encoding_blocks.2.tdf.4.weight] + - [2] from weight of shape torch.Size([144]) [name: encoding_blocks.2.tdf.4.bias] + - [3] from weight of shape torch.Size([144]) [name: encoding_blocks.2.tdf.4.running_mean] + - [4] from weight of shape torch.Size([144]) [name: encoding_blocks.2.tdf.4.running_var] +[044] Relu_44 type: Relu inputs: ['506'] outputs: ['507'] + - [0] from output of BatchNormalization_43 [name: 506] +[045] Add_45 type: Add inputs: ['499', '507'] outputs: ['508'] + - [0] from output of Relu_38 [name: 499] + - [1] from output of Relu_44 [name: 507] + +DOWN 3 + +[046] Conv_46 type: Conv W inputs: ['508', '721', '722'] outputs: ['720'] + - [0] from output of Add_45 [name: 508] + - [1] from weight of shape torch.Size([192, 144, 2, 2]) [name: 721] + - [2] from weight of shape torch.Size([192]) [name: 722] +[047] Relu_47 type: Relu inputs: ['720'] outputs: ['511'] + - [0] from output of Conv_46 [name: 720] + +------------------------------------------- +Stage 4 +------------------------------------------- + +[048] Conv_48 type: Conv W inputs: ['511', '724', '725'] outputs: ['723'] + - [0] from output of Relu_47 [name: 511] + - [1] from weight of shape torch.Size([192, 192, 3, 3]) [name: 724] + - [2] from weight of shape torch.Size([192]) [name: 725] +[049] Relu_49 type: Relu inputs: ['723'] outputs: ['514'] + - [0] from output of Conv_48 [name: 723] +[050] Conv_50 type: Conv W inputs: ['514', '727', '728'] outputs: ['726'] + - [0] from output of Relu_49 [name: 514] + - [1] from weight of shape torch.Size([192, 192, 3, 3]) [name: 727] + - [2] from weight of shape torch.Size([192]) [name: 728] +[051] Relu_51 type: Relu inputs: ['726'] outputs: ['517'] + - [0] from output of Conv_50 [name: 726] +[052] Conv_52 type: Conv W inputs: ['517', '730', '731'] outputs: ['729'] + - [0] from output of Relu_51 [name: 517] + - [1] from weight of shape torch.Size([192, 192, 3, 3]) [name: 730] + - [2] from weight of shape torch.Size([192]) [name: 731] +[053] Relu_53 type: Relu inputs: ['729'] outputs: ['520'] + - [0] from output of Conv_52 [name: 729] +[054] MatMul_54 type: MatMul W inputs: ['520', '807'] outputs: ['522'] + - [0] from output of Relu_53 [name: 520] + - [1] from weight of shape torch.Size([384, 48]) [name: 807] +[055] BatchNormalization_55 type: BatchNormalization W inputs: ['522', 'encoding_blocks.3.tdf.1.weight', 'encoding_blocks.3.tdf.1.bias', 'encoding_blocks.3.tdf.1.running_mean', 'encoding_blocks.3.tdf.1.running_var'] outputs: ['523'] + - [0] from output of MatMul_54 [name: 522] + - [1] from weight of shape torch.Size([192]) [name: encoding_blocks.3.tdf.1.weight] + - [2] from weight of shape torch.Size([192]) [name: encoding_blocks.3.tdf.1.bias] + - [3] from weight of shape torch.Size([192]) [name: encoding_blocks.3.tdf.1.running_mean] + - [4] from weight of shape torch.Size([192]) [name: encoding_blocks.3.tdf.1.running_var] +[056] Relu_56 type: Relu inputs: ['523'] outputs: ['524'] + - [0] from output of BatchNormalization_55 [name: 523] +[057] MatMul_57 type: MatMul W inputs: ['524', '808'] outputs: ['526'] + - [0] from output of Relu_56 [name: 524] + - [1] from weight of shape torch.Size([48, 384]) [name: 808] +[058] BatchNormalization_58 type: BatchNormalization W inputs: ['526', 'encoding_blocks.3.tdf.4.weight', 'encoding_blocks.3.tdf.4.bias', 'encoding_blocks.3.tdf.4.running_mean', 'encoding_blocks.3.tdf.4.running_var'] outputs: ['527'] + - [0] from output of MatMul_57 [name: 526] + - [1] from weight of shape torch.Size([192]) [name: encoding_blocks.3.tdf.4.weight] + - [2] from weight of shape torch.Size([192]) [name: encoding_blocks.3.tdf.4.bias] + - [3] from weight of shape torch.Size([192]) [name: encoding_blocks.3.tdf.4.running_mean] + - [4] from weight of shape torch.Size([192]) [name: encoding_blocks.3.tdf.4.running_var] +[059] Relu_59 type: Relu inputs: ['527'] outputs: ['528'] + - [0] from output of BatchNormalization_58 [name: 527] +[060] Add_60 type: Add inputs: ['520', '528'] outputs: ['529'] + - [0] from output of Relu_53 [name: 520] + - [1] from output of Relu_59 [name: 528] + +DOWN 4 + +[061] Conv_61 type: Conv W inputs: ['529', '733', '734'] outputs: ['732'] + - [0] from output of Add_60 [name: 529] + - [1] from weight of shape torch.Size([240, 192, 2, 2]) [name: 733] + - [2] from weight of shape torch.Size([240]) [name: 734] +[062] Relu_62 type: Relu inputs: ['732'] outputs: ['532'] + - [0] from output of Conv_61 [name: 732] + +------------------------------------------- +Stage 5 +------------------------------------------- + +[063] Conv_63 type: Conv W inputs: ['532', '736', '737'] outputs: ['735'] + - [0] from output of Relu_62 [name: 532] + - [1] from weight of shape torch.Size([240, 240, 3, 3]) [name: 736] + - [2] from weight of shape torch.Size([240]) [name: 737] +[064] Relu_64 type: Relu inputs: ['735'] outputs: ['535'] + - [0] from output of Conv_63 [name: 735] +[065] Conv_65 type: Conv W inputs: ['535', '739', '740'] outputs: ['738'] + - [0] from output of Relu_64 [name: 535] + - [1] from weight of shape torch.Size([240, 240, 3, 3]) [name: 739] + - [2] from weight of shape torch.Size([240]) [name: 740] +[066] Relu_66 type: Relu inputs: ['738'] outputs: ['538'] + - [0] from output of Conv_65 [name: 738] +[067] Conv_67 type: Conv W inputs: ['538', '742', '743'] outputs: ['741'] + - [0] from output of Relu_66 [name: 538] + - [1] from weight of shape torch.Size([240, 240, 3, 3]) [name: 742] + - [2] from weight of shape torch.Size([240]) [name: 743] +[068] Relu_68 type: Relu inputs: ['741'] outputs: ['541'] + - [0] from output of Conv_67 [name: 741] +[069] MatMul_69 type: MatMul W inputs: ['541', '809'] outputs: ['543'] + - [0] from output of Relu_68 [name: 541] + - [1] from weight of shape torch.Size([192, 24]) [name: 809] +[070] BatchNormalization_70 type: BatchNormalization W inputs: ['543', 'encoding_blocks.4.tdf.1.weight', 'encoding_blocks.4.tdf.1.bias', 'encoding_blocks.4.tdf.1.running_mean', 'encoding_blocks.4.tdf.1.running_var'] outputs: ['544'] + - [0] from output of MatMul_69 [name: 543] + - [1] from weight of shape torch.Size([240]) [name: encoding_blocks.4.tdf.1.weight] + - [2] from weight of shape torch.Size([240]) [name: encoding_blocks.4.tdf.1.bias] + - [3] from weight of shape torch.Size([240]) [name: encoding_blocks.4.tdf.1.running_mean] + - [4] from weight of shape torch.Size([240]) [name: encoding_blocks.4.tdf.1.running_var] +[071] Relu_71 type: Relu inputs: ['544'] outputs: ['545'] + - [0] from output of BatchNormalization_70 [name: 544] +[072] MatMul_72 type: MatMul W inputs: ['545', '810'] outputs: ['547'] + - [0] from output of Relu_71 [name: 545] + - [1] from weight of shape torch.Size([24, 192]) [name: 810] +[073] BatchNormalization_73 type: BatchNormalization W inputs: ['547', 'encoding_blocks.4.tdf.4.weight', 'encoding_blocks.4.tdf.4.bias', 'encoding_blocks.4.tdf.4.running_mean', 'encoding_blocks.4.tdf.4.running_var'] outputs: ['548'] + - [0] from output of MatMul_72 [name: 547] + - [1] from weight of shape torch.Size([240]) [name: encoding_blocks.4.tdf.4.weight] + - [2] from weight of shape torch.Size([240]) [name: encoding_blocks.4.tdf.4.bias] + - [3] from weight of shape torch.Size([240]) [name: encoding_blocks.4.tdf.4.running_mean] + - [4] from weight of shape torch.Size([240]) [name: encoding_blocks.4.tdf.4.running_var] +[074] Relu_74 type: Relu inputs: ['548'] outputs: ['549'] + - [0] from output of BatchNormalization_73 [name: 548] +[075] Add_75 type: Add inputs: ['541', '549'] outputs: ['550'] + - [0] from output of Relu_68 [name: 541] + - [1] from output of Relu_74 [name: 549] + +DOWN 5 + +[076] Conv_76 type: Conv W inputs: ['550', '745', '746'] outputs: ['744'] + - [0] from output of Add_75 [name: 550] + - [1] from weight of shape torch.Size([288, 240, 2, 2]) [name: 745] + - [2] from weight of shape torch.Size([288]) [name: 746] + [077] Relu_77 type: Relu inputs: ['744'] outputs: ['553'] + - [0] from output of Conv_76 [name: 744] + +------------------------------------------- +Bottleneck +------------------------------------------- + +[078] Conv_78 type: Conv W inputs: ['553', '748', '749'] outputs: ['747'] + - [0] from output of Relu_77 [name: 553] + - [1] from weight of shape torch.Size([288, 288, 3, 3]) [name: 748] + - [2] from weight of shape torch.Size([288]) [name: 749] +[079] Relu_79 type: Relu inputs: ['747'] outputs: ['556'] + - [0] from output of Conv_78 [name: 747] +[080] Conv_80 type: Conv W inputs: ['556', '751', '752'] outputs: ['750'] + - [0] from output of Relu_79 [name: 556] + - [1] from weight of shape torch.Size([288, 288, 3, 3]) [name: 751] + - [2] from weight of shape torch.Size([288]) [name: 752] +[081] Relu_81 type: Relu inputs: ['750'] outputs: ['559'] + - [0] from output of Conv_80 [name: 750] +[082] Conv_82 type: Conv W inputs: ['559', '754', '755'] outputs: ['753'] + - [0] from output of Relu_81 [name: 559] + - [1] from weight of shape torch.Size([288, 288, 3, 3]) [name: 754] + - [2] from weight of shape torch.Size([288]) [name: 755] +[083] Relu_83 type: Relu inputs: ['753'] outputs: ['562'] + - [0] from output of Conv_82 [name: 753] +[084] MatMul_84 type: MatMul W inputs: ['562', '811'] outputs: ['564'] + - [0] from output of Relu_83 [name: 562] + - [1] from weight of shape torch.Size([96, 12]) [name: 811] +[085] BatchNormalization_85 type: BatchNormalization W inputs: ['564', 'bottleneck_block.tdf.1.weight', 'bottleneck_block.tdf.1.bias', 'bottleneck_block.tdf.1.running_mean', 'bottleneck_block.tdf.1.running_var'] outputs: ['565'] + - [0] from output of MatMul_84 [name: 564] + - [1] from weight of shape torch.Size([288]) [name: bottleneck_block.tdf.1.weight] + - [2] from weight of shape torch.Size([288]) [name: bottleneck_block.tdf.1.bias] + - [3] from weight of shape torch.Size([288]) [name: bottleneck_block.tdf.1.running_mean] + - [4] from weight of shape torch.Size([288]) [name: bottleneck_block.tdf.1.running_var] +[086] Relu_86 type: Relu inputs: ['565'] outputs: ['566'] + - [0] from output of BatchNormalization_85 [name: 565] +[087] MatMul_87 type: MatMul W inputs: ['566', '812'] outputs: ['568'] + - [0] from output of Relu_86 [name: 566] + - [1] from weight of shape torch.Size([12, 96]) [name: 812] +[088] BatchNormalization_88 type: BatchNormalization W inputs: ['568', 'bottleneck_block.tdf.4.weight', 'bottleneck_block.tdf.4.bias', 'bottleneck_block.tdf.4.running_mean', 'bottleneck_block.tdf.4.running_var'] outputs: ['569'] + - [0] from output of MatMul_87 [name: 568] + - [1] from weight of shape torch.Size([288]) [name: bottleneck_block.tdf.4.weight] + - [2] from weight of shape torch.Size([288]) [name: bottleneck_block.tdf.4.bias] + - [3] from weight of shape torch.Size([288]) [name: bottleneck_block.tdf.4.running_mean] + - [4] from weight of shape torch.Size([288]) [name: bottleneck_block.tdf.4.running_var] +[089] Relu_89 type: Relu inputs: ['569'] outputs: ['570'] + - [0] from output of BatchNormalization_88 [name: 569] +[090] Add_90 type: Add inputs: ['562', '570'] outputs: ['571'] + - [0] from output of Relu_83 [name: 562] + - [1] from output of Relu_89 [name: 570] +[091] ConvTranspose_91 type: ConvTranspose W inputs: ['571', 'us.0.0.weight', 'us.0.0.bias'] outputs: ['572'] + - [0] from output of Add_90 [name: 571] + - [1] from weight of shape torch.Size([288, 240, 2, 2]) [name: us.0.0.weight] + - [2] from weight of shape torch.Size([240]) [name: us.0.0.bias] +[092] BatchNormalization_92 type: BatchNormalization W inputs: ['572', 'us.0.1.weight', 'us.0.1.bias', 'us.0.1.running_mean', 'us.0.1.running_var'] outputs: ['573'] + - [0] from output of ConvTranspose_91 [name: 572] + - [1] from weight of shape torch.Size([240]) [name: us.0.1.weight] + - [2] from weight of shape torch.Size([240]) [name: us.0.1.bias] + - [3] from weight of shape torch.Size([240]) [name: us.0.1.running_mean] + - [4] from weight of shape torch.Size([240]) [name: us.0.1.running_var] +[093] Relu_93 type: Relu inputs: ['573'] outputs: ['574'] + - [0] from output of BatchNormalization_92 [name: 573] +[094] Mul_94 type: Mul inputs: ['574', '550'] outputs: ['575'] + - [0] from output of Relu_93 [name: 574] + - [1] from output of Add_75 [name: 550] +[095] Conv_95 type: Conv W inputs: ['575', '757', '758'] outputs: ['756'] + - [0] from output of Mul_94 [name: 575] + - [1] from weight of shape torch.Size([240, 240, 3, 3]) [name: 757] + - [2] from weight of shape torch.Size([240]) [name: 758] +[096] Relu_96 type: Relu inputs: ['756'] outputs: ['578'] + - [0] from output of Conv_95 [name: 756] +[097] Conv_97 type: Conv W inputs: ['578', '760', '761'] outputs: ['759'] + - [0] from output of Relu_96 [name: 578] + - [1] from weight of shape torch.Size([240, 240, 3, 3]) [name: 760] + - [2] from weight of shape torch.Size([240]) [name: 761] +[098] Relu_98 type: Relu inputs: ['759'] outputs: ['581'] + - [0] from output of Conv_97 [name: 759] +[099] Conv_99 type: Conv W inputs: ['581', '763', '764'] outputs: ['762'] + - [0] from output of Relu_98 [name: 581] + - [1] from weight of shape torch.Size([240, 240, 3, 3]) [name: 763] + - [2] from weight of shape torch.Size([240]) [name: 764] +[100] Relu_100 type: Relu inputs: ['762'] outputs: ['584'] + - [0] from output of Conv_99 [name: 762] +[101] MatMul_101 type: MatMul W inputs: ['584', '813'] outputs: ['586'] + - [0] from output of Relu_100 [name: 584] + - [1] from weight of shape torch.Size([192, 24]) [name: 813] +[102] BatchNormalization_102 type: BatchNormalization W inputs: ['586', 'decoding_blocks.0.tdf.1.weight', 'decoding_blocks.0.tdf.1.bias', 'decoding_blocks.0.tdf.1.running_mean', 'decoding_blocks.0.tdf.1.running_var'] outputs: ['587'] + - [0] from output of MatMul_101 [name: 586] + - [1] from weight of shape torch.Size([240]) [name: decoding_blocks.0.tdf.1.weight] + - [2] from weight of shape torch.Size([240]) [name: decoding_blocks.0.tdf.1.bias] + - [3] from weight of shape torch.Size([240]) [name: decoding_blocks.0.tdf.1.running_mean] + - [4] from weight of shape torch.Size([240]) [name: decoding_blocks.0.tdf.1.running_var] +[103] Relu_103 type: Relu inputs: ['587'] outputs: ['588'] + - [0] from output of BatchNormalization_102 [name: 587] +[104] MatMul_104 type: MatMul W inputs: ['588', '814'] outputs: ['590'] + - [0] from output of Relu_103 [name: 588] + - [1] from weight of shape torch.Size([24, 192]) [name: 814] +[105] BatchNormalization_105 type: BatchNormalization W inputs: ['590', 'decoding_blocks.0.tdf.4.weight', 'decoding_blocks.0.tdf.4.bias', 'decoding_blocks.0.tdf.4.running_mean', 'decoding_blocks.0.tdf.4.running_var'] outputs: ['591'] + - [0] from output of MatMul_104 [name: 590] + - [1] from weight of shape torch.Size([240]) [name: decoding_blocks.0.tdf.4.weight] + - [2] from weight of shape torch.Size([240]) [name: decoding_blocks.0.tdf.4.bias] + - [3] from weight of shape torch.Size([240]) [name: decoding_blocks.0.tdf.4.running_mean] + - [4] from weight of shape torch.Size([240]) [name: decoding_blocks.0.tdf.4.running_var] +[106] Relu_106 type: Relu inputs: ['591'] outputs: ['592'] + - [0] from output of BatchNormalization_105 [name: 591] +[107] Add_107 type: Add inputs: ['584', '592'] outputs: ['593'] + - [0] from output of Relu_100 [name: 584] + - [1] from output of Relu_106 [name: 592] +[108] ConvTranspose_108 type: ConvTranspose W inputs: ['593', 'us.1.0.weight', 'us.1.0.bias'] outputs: ['594'] + - [0] from output of Add_107 [name: 593] + - [1] from weight of shape torch.Size([240, 192, 2, 2]) [name: us.1.0.weight] + - [2] from weight of shape torch.Size([192]) [name: us.1.0.bias] +[109] BatchNormalization_109 type: BatchNormalization W inputs: ['594', 'us.1.1.weight', 'us.1.1.bias', 'us.1.1.running_mean', 'us.1.1.running_var'] outputs: ['595'] + - [0] from output of ConvTranspose_108 [name: 594] + - [1] from weight of shape torch.Size([192]) [name: us.1.1.weight] + - [2] from weight of shape torch.Size([192]) [name: us.1.1.bias] + - [3] from weight of shape torch.Size([192]) [name: us.1.1.running_mean] + - [4] from weight of shape torch.Size([192]) [name: us.1.1.running_var] +[110] Relu_110 type: Relu inputs: ['595'] outputs: ['596'] + - [0] from output of BatchNormalization_109 [name: 595] +[111] Mul_111 type: Mul inputs: ['596', '529'] outputs: ['597'] + - [0] from output of Relu_110 [name: 596] + - [1] from output of Add_60 [name: 529] +[112] Conv_112 type: Conv W inputs: ['597', '766', '767'] outputs: ['765'] + - [0] from output of Mul_111 [name: 597] + - [1] from weight of shape torch.Size([192, 192, 3, 3]) [name: 766] + - [2] from weight of shape torch.Size([192]) [name: 767] +[113] Relu_113 type: Relu inputs: ['765'] outputs: ['600'] + - [0] from output of Conv_112 [name: 765] +[114] Conv_114 type: Conv W inputs: ['600', '769', '770'] outputs: ['768'] + - [0] from output of Relu_113 [name: 600] + - [1] from weight of shape torch.Size([192, 192, 3, 3]) [name: 769] + - [2] from weight of shape torch.Size([192]) [name: 770] +[115] Relu_115 type: Relu inputs: ['768'] outputs: ['603'] + - [0] from output of Conv_114 [name: 768] +[116] Conv_116 type: Conv W inputs: ['603', '772', '773'] outputs: ['771'] + - [0] from output of Relu_115 [name: 603] + - [1] from weight of shape torch.Size([192, 192, 3, 3]) [name: 772] + - [2] from weight of shape torch.Size([192]) [name: 773] +[117] Relu_117 type: Relu inputs: ['771'] outputs: ['606'] + - [0] from output of Conv_116 [name: 771] +[118] MatMul_118 type: MatMul W inputs: ['606', '815'] outputs: ['608'] + - [0] from output of Relu_117 [name: 606] + - [1] from weight of shape torch.Size([384, 48]) [name: 815] +[119] BatchNormalization_119 type: BatchNormalization W inputs: ['608', 'decoding_blocks.1.tdf.1.weight', 'decoding_blocks.1.tdf.1.bias', 'decoding_blocks.1.tdf.1.running_mean', 'decoding_blocks.1.tdf.1.running_var'] outputs: ['609'] + - [0] from output of MatMul_118 [name: 608] + - [1] from weight of shape torch.Size([192]) [name: decoding_blocks.1.tdf.1.weight] + - [2] from weight of shape torch.Size([192]) [name: decoding_blocks.1.tdf.1.bias] + - [3] from weight of shape torch.Size([192]) [name: decoding_blocks.1.tdf.1.running_mean] + - [4] from weight of shape torch.Size([192]) [name: decoding_blocks.1.tdf.1.running_var] +[120] Relu_120 type: Relu inputs: ['609'] outputs: ['610'] + - [0] from output of BatchNormalization_119 [name: 609] +[121] MatMul_121 type: MatMul W inputs: ['610', '816'] outputs: ['612'] + - [0] from output of Relu_120 [name: 610] + - [1] from weight of shape torch.Size([48, 384]) [name: 816] +[122] BatchNormalization_122 type: BatchNormalization W inputs: ['612', 'decoding_blocks.1.tdf.4.weight', 'decoding_blocks.1.tdf.4.bias', 'decoding_blocks.1.tdf.4.running_mean', 'decoding_blocks.1.tdf.4.running_var'] outputs: ['613'] + - [0] from output of MatMul_121 [name: 612] + - [1] from weight of shape torch.Size([192]) [name: decoding_blocks.1.tdf.4.weight] + - [2] from weight of shape torch.Size([192]) [name: decoding_blocks.1.tdf.4.bias] + - [3] from weight of shape torch.Size([192]) [name: decoding_blocks.1.tdf.4.running_mean] + - [4] from weight of shape torch.Size([192]) [name: decoding_blocks.1.tdf.4.running_var] +[123] Relu_123 type: Relu inputs: ['613'] outputs: ['614'] + - [0] from output of BatchNormalization_122 [name: 613] +[124] Add_124 type: Add inputs: ['606', '614'] outputs: ['615'] + - [0] from output of Relu_117 [name: 606] + - [1] from output of Relu_123 [name: 614] +[125] ConvTranspose_125 type: ConvTranspose W inputs: ['615', 'us.2.0.weight', 'us.2.0.bias'] outputs: ['616'] + - [0] from output of Add_124 [name: 615] + - [1] from weight of shape torch.Size([192, 144, 2, 2]) [name: us.2.0.weight] + - [2] from weight of shape torch.Size([144]) [name: us.2.0.bias] +[126] BatchNormalization_126 type: BatchNormalization W inputs: ['616', 'us.2.1.weight', 'us.2.1.bias', 'us.2.1.running_mean', 'us.2.1.running_var'] outputs: ['617'] + - [0] from output of ConvTranspose_125 [name: 616] + - [1] from weight of shape torch.Size([144]) [name: us.2.1.weight] + - [2] from weight of shape torch.Size([144]) [name: us.2.1.bias] + - [3] from weight of shape torch.Size([144]) [name: us.2.1.running_mean] + - [4] from weight of shape torch.Size([144]) [name: us.2.1.running_var] +[127] Relu_127 type: Relu inputs: ['617'] outputs: ['618'] + - [0] from output of BatchNormalization_126 [name: 617] +[128] Mul_128 type: Mul inputs: ['618', '508'] outputs: ['619'] + - [0] from output of Relu_127 [name: 618] + - [1] from output of Add_45 [name: 508] +[129] Conv_129 type: Conv W inputs: ['619', '775', '776'] outputs: ['774'] + - [0] from output of Mul_128 [name: 619] + - [1] from weight of shape torch.Size([144, 144, 3, 3]) [name: 775] + - [2] from weight of shape torch.Size([144]) [name: 776] +[130] Relu_130 type: Relu inputs: ['774'] outputs: ['622'] + - [0] from output of Conv_129 [name: 774] +[131] Conv_131 type: Conv W inputs: ['622', '778', '779'] outputs: ['777'] + - [0] from output of Relu_130 [name: 622] + - [1] from weight of shape torch.Size([144, 144, 3, 3]) [name: 778] + - [2] from weight of shape torch.Size([144]) [name: 779] +[132] Relu_132 type: Relu inputs: ['777'] outputs: ['625'] + - [0] from output of Conv_131 [name: 777] +[133] Conv_133 type: Conv W inputs: ['625', '781', '782'] outputs: ['780'] + - [0] from output of Relu_132 [name: 625] + - [1] from weight of shape torch.Size([144, 144, 3, 3]) [name: 781] + - [2] from weight of shape torch.Size([144]) [name: 782] +[134] Relu_134 type: Relu inputs: ['780'] outputs: ['628'] + - [0] from output of Conv_133 [name: 780] +[135] MatMul_135 type: MatMul W inputs: ['628', '817'] outputs: ['630'] + - [0] from output of Relu_134 [name: 628] + - [1] from weight of shape torch.Size([768, 96]) [name: 817] +[136] BatchNormalization_136 type: BatchNormalization W inputs: ['630', 'decoding_blocks.2.tdf.1.weight', 'decoding_blocks.2.tdf.1.bias', 'decoding_blocks.2.tdf.1.running_mean', 'decoding_blocks.2.tdf.1.running_var'] outputs: ['631'] + - [0] from output of MatMul_135 [name: 630] + - [1] from weight of shape torch.Size([144]) [name: decoding_blocks.2.tdf.1.weight] + - [2] from weight of shape torch.Size([144]) [name: decoding_blocks.2.tdf.1.bias] + - [3] from weight of shape torch.Size([144]) [name: decoding_blocks.2.tdf.1.running_mean] + - [4] from weight of shape torch.Size([144]) [name: decoding_blocks.2.tdf.1.running_var] +[137] Relu_137 type: Relu inputs: ['631'] outputs: ['632'] + - [0] from output of BatchNormalization_136 [name: 631] +[138] MatMul_138 type: MatMul W inputs: ['632', '818'] outputs: ['634'] + - [0] from output of Relu_137 [name: 632] + - [1] from weight of shape torch.Size([96, 768]) [name: 818] +[139] BatchNormalization_139 type: BatchNormalization W inputs: ['634', 'decoding_blocks.2.tdf.4.weight', 'decoding_blocks.2.tdf.4.bias', 'decoding_blocks.2.tdf.4.running_mean', 'decoding_blocks.2.tdf.4.running_var'] outputs: ['635'] + - [0] from output of MatMul_138 [name: 634] + - [1] from weight of shape torch.Size([144]) [name: decoding_blocks.2.tdf.4.weight] + - [2] from weight of shape torch.Size([144]) [name: decoding_blocks.2.tdf.4.bias] + - [3] from weight of shape torch.Size([144]) [name: decoding_blocks.2.tdf.4.running_mean] + - [4] from weight of shape torch.Size([144]) [name: decoding_blocks.2.tdf.4.running_var] +[140] Relu_140 type: Relu inputs: ['635'] outputs: ['636'] + - [0] from output of BatchNormalization_139 [name: 635] +[141] Add_141 type: Add inputs: ['628', '636'] outputs: ['637'] + - [0] from output of Relu_134 [name: 628] + - [1] from output of Relu_140 [name: 636] +[142] ConvTranspose_142 type: ConvTranspose W inputs: ['637', 'us.3.0.weight', 'us.3.0.bias'] outputs: ['638'] + - [0] from output of Add_141 [name: 637] + - [1] from weight of shape torch.Size([144, 96, 2, 2]) [name: us.3.0.weight] + - [2] from weight of shape torch.Size([96]) [name: us.3.0.bias] +[143] BatchNormalization_143 type: BatchNormalization W inputs: ['638', 'us.3.1.weight', 'us.3.1.bias', 'us.3.1.running_mean', 'us.3.1.running_var'] outputs: ['639'] + - [0] from output of ConvTranspose_142 [name: 638] + - [1] from weight of shape torch.Size([96]) [name: us.3.1.weight] + - [2] from weight of shape torch.Size([96]) [name: us.3.1.bias] + - [3] from weight of shape torch.Size([96]) [name: us.3.1.running_mean] + - [4] from weight of shape torch.Size([96]) [name: us.3.1.running_var] +[144] Relu_144 type: Relu inputs: ['639'] outputs: ['640'] + - [0] from output of BatchNormalization_143 [name: 639] +[145] Mul_145 type: Mul inputs: ['640', '487'] outputs: ['641'] + - [0] from output of Relu_144 [name: 640] + - [1] from output of Add_30 [name: 487] +[146] Conv_146 type: Conv W inputs: ['641', '784', '785'] outputs: ['783'] + - [0] from output of Mul_145 [name: 641] + - [1] from weight of shape torch.Size([96, 96, 3, 3]) [name: 784] + - [2] from weight of shape torch.Size([96]) [name: 785] +[147] Relu_147 type: Relu inputs: ['783'] outputs: ['644'] + - [0] from output of Conv_146 [name: 783] +[148] Conv_148 type: Conv W inputs: ['644', '787', '788'] outputs: ['786'] + - [0] from output of Relu_147 [name: 644] + - [1] from weight of shape torch.Size([96, 96, 3, 3]) [name: 787] + - [2] from weight of shape torch.Size([96]) [name: 788] +[149] Relu_149 type: Relu inputs: ['786'] outputs: ['647'] + - [0] from output of Conv_148 [name: 786] +[150] Conv_150 type: Conv W inputs: ['647', '790', '791'] outputs: ['789'] + - [0] from output of Relu_149 [name: 647] + - [1] from weight of shape torch.Size([96, 96, 3, 3]) [name: 790] + - [2] from weight of shape torch.Size([96]) [name: 791] +[151] Relu_151 type: Relu inputs: ['789'] outputs: ['650'] + - [0] from output of Conv_150 [name: 789] +[152] MatMul_152 type: MatMul W inputs: ['650', '819'] outputs: ['652'] + - [0] from output of Relu_151 [name: 650] + - [1] from weight of shape torch.Size([1536, 192]) [name: 819] +[153] BatchNormalization_153 type: BatchNormalization W inputs: ['652', 'decoding_blocks.3.tdf.1.weight', 'decoding_blocks.3.tdf.1.bias', 'decoding_blocks.3.tdf.1.running_mean', 'decoding_blocks.3.tdf.1.running_var'] outputs: ['653'] + - [0] from output of MatMul_152 [name: 652] + - [1] from weight of shape torch.Size([96]) [name: decoding_blocks.3.tdf.1.weight] + - [2] from weight of shape torch.Size([96]) [name: decoding_blocks.3.tdf.1.bias] + - [3] from weight of shape torch.Size([96]) [name: decoding_blocks.3.tdf.1.running_mean] + - [4] from weight of shape torch.Size([96]) [name: decoding_blocks.3.tdf.1.running_var] +[154] Relu_154 type: Relu inputs: ['653'] outputs: ['654'] + - [0] from output of BatchNormalization_153 [name: 653] +[155] MatMul_155 type: MatMul W inputs: ['654', '820'] outputs: ['656'] + - [0] from output of Relu_154 [name: 654] + - [1] from weight of shape torch.Size([192, 1536]) [name: 820] +[156] BatchNormalization_156 type: BatchNormalization W inputs: ['656', 'decoding_blocks.3.tdf.4.weight', 'decoding_blocks.3.tdf.4.bias', 'decoding_blocks.3.tdf.4.running_mean', 'decoding_blocks.3.tdf.4.running_var'] outputs: ['657'] + - [0] from output of MatMul_155 [name: 656] + - [1] from weight of shape torch.Size([96]) [name: decoding_blocks.3.tdf.4.weight] + - [2] from weight of shape torch.Size([96]) [name: decoding_blocks.3.tdf.4.bias] + - [3] from weight of shape torch.Size([96]) [name: decoding_blocks.3.tdf.4.running_mean] + - [4] from weight of shape torch.Size([96]) [name: decoding_blocks.3.tdf.4.running_var] +[157] Relu_157 type: Relu inputs: ['657'] outputs: ['658'] + - [0] from output of BatchNormalization_156 [name: 657] +[158] Add_158 type: Add inputs: ['650', '658'] outputs: ['659'] + - [0] from output of Relu_151 [name: 650] + - [1] from output of Relu_157 [name: 658] +[159] ConvTranspose_159 type: ConvTranspose W inputs: ['659', 'us.4.0.weight', 'us.4.0.bias'] outputs: ['660'] + - [0] from output of Add_158 [name: 659] + - [1] from weight of shape torch.Size([96, 48, 2, 2]) [name: us.4.0.weight] + - [2] from weight of shape torch.Size([48]) [name: us.4.0.bias] +[160] BatchNormalization_160 type: BatchNormalization W inputs: ['660', 'us.4.1.weight', 'us.4.1.bias', 'us.4.1.running_mean', 'us.4.1.running_var'] outputs: ['661'] + - [0] from output of ConvTranspose_159 [name: 660] + - [1] from weight of shape torch.Size([48]) [name: us.4.1.weight] + - [2] from weight of shape torch.Size([48]) [name: us.4.1.bias] + - [3] from weight of shape torch.Size([48]) [name: us.4.1.running_mean] + - [4] from weight of shape torch.Size([48]) [name: us.4.1.running_var] +[161] Relu_161 type: Relu inputs: ['661'] outputs: ['662'] + - [0] from output of BatchNormalization_160 [name: 661] +[162] Mul_162 type: Mul inputs: ['662', '466'] outputs: ['663'] + - [0] from output of Relu_161 [name: 662] + - [1] from output of Add_15 [name: 466] +[163] Conv_163 type: Conv W inputs: ['663', '793', '794'] outputs: ['792'] + - [0] from output of Mul_162 [name: 663] + - [1] from weight of shape torch.Size([48, 48, 3, 3]) [name: 793] + - [2] from weight of shape torch.Size([48]) [name: 794] +[164] Relu_164 type: Relu inputs: ['792'] outputs: ['666'] + - [0] from output of Conv_163 [name: 792] +[165] Conv_165 type: Conv W inputs: ['666', '796', '797'] outputs: ['795'] + - [0] from output of Relu_164 [name: 666] + - [1] from weight of shape torch.Size([48, 48, 3, 3]) [name: 796] + - [2] from weight of shape torch.Size([48]) [name: 797] +[166] Relu_166 type: Relu inputs: ['795'] outputs: ['669'] + - [0] from output of Conv_165 [name: 795] +[167] Conv_167 type: Conv W inputs: ['669', '799', '800'] outputs: ['798'] + - [0] from output of Relu_166 [name: 669] + - [1] from weight of shape torch.Size([48, 48, 3, 3]) [name: 799] + - [2] from weight of shape torch.Size([48]) [name: 800] +[168] Relu_168 type: Relu inputs: ['798'] outputs: ['672'] + - [0] from output of Conv_167 [name: 798] +[169] MatMul_169 type: MatMul W inputs: ['672', '821'] outputs: ['674'] + - [0] from output of Relu_168 [name: 672] + - [1] from weight of shape torch.Size([3072, 384]) [name: 821] +[170] BatchNormalization_170 type: BatchNormalization W inputs: ['674', 'decoding_blocks.4.tdf.1.weight', 'decoding_blocks.4.tdf.1.bias', 'decoding_blocks.4.tdf.1.running_mean', 'decoding_blocks.4.tdf.1.running_var'] outputs: ['675'] + - [0] from output of MatMul_169 [name: 674] + - [1] from weight of shape torch.Size([48]) [name: decoding_blocks.4.tdf.1.weight] + - [2] from weight of shape torch.Size([48]) [name: decoding_blocks.4.tdf.1.bias] + - [3] from weight of shape torch.Size([48]) [name: decoding_blocks.4.tdf.1.running_mean] + - [4] from weight of shape torch.Size([48]) [name: decoding_blocks.4.tdf.1.running_var] +[171] Relu_171 type: Relu inputs: ['675'] outputs: ['676'] + - [0] from output of BatchNormalization_170 [name: 675] +[172] MatMul_172 type: MatMul W inputs: ['676', '822'] outputs: ['678'] + - [0] from output of Relu_171 [name: 676] + - [1] from weight of shape torch.Size([384, 3072]) [name: 822] +[173] BatchNormalization_173 type: BatchNormalization W inputs: ['678', 'decoding_blocks.4.tdf.4.weight', 'decoding_blocks.4.tdf.4.bias', 'decoding_blocks.4.tdf.4.running_mean', 'decoding_blocks.4.tdf.4.running_var'] outputs: ['679'] + - [0] from output of MatMul_172 [name: 678] + - [1] from weight of shape torch.Size([48]) [name: decoding_blocks.4.tdf.4.weight] + - [2] from weight of shape torch.Size([48]) [name: decoding_blocks.4.tdf.4.bias] + - [3] from weight of shape torch.Size([48]) [name: decoding_blocks.4.tdf.4.running_mean] + - [4] from weight of shape torch.Size([48]) [name: decoding_blocks.4.tdf.4.running_var] +[174] Relu_174 type: Relu inputs: ['679'] outputs: ['680'] + - [0] from output of BatchNormalization_173 [name: 679] +[175] Add_175 type: Add inputs: ['672', '680'] outputs: ['681'] + - [0] from output of Relu_168 [name: 672] + - [1] from output of Relu_174 [name: 680] +[176] Transpose_176 type: Transpose inputs: ['681'] outputs: ['682'] + - [0] from output of Add_175 [name: 681] +[177] Conv_177 type: Conv W inputs: ['682', 'final_conv.0.weight', 'final_conv.0.bias'] outputs: ['output'] + - [0] from output of Transpose_176 [name: 682] + - [1] from weight of shape torch.Size([4, 48, 1, 1]) [name: final_conv.0.weight] + - [2] from weight of shape torch.Size([4]) [name: final_conv.0.bias] +The output is from Conv_177 diff --git a/doc/MDX/maps/mapping.txt b/doc/MDX/maps/mapping.txt new file mode 100644 index 0000000..8ff2eca --- /dev/null +++ b/doc/MDX/maps/mapping.txt @@ -0,0 +1,232 @@ + ('initial_conv.weight', '685', False), + ('initial_conv.bias', '686', False), + # Encoder Stage 1 + ('enc_stage_1.time_branch.sequence.0.weight', '688', False), + ('enc_stage_1.time_branch.sequence.0.bias', '689', False), + ('enc_stage_1.time_branch.sequence.2.weight', '691', False), + ('enc_stage_1.time_branch.sequence.2.bias', '692', False), + ('enc_stage_1.time_branch.sequence.4.weight', '694', False), + ('enc_stage_1.time_branch.sequence.4.bias', '695', False), + ('enc_stage_1.freq_branch.sequence.0.weight', '801', True), + ('enc_stage_1.freq_branch.sequence.1.weight', 'encoding_blocks.0.tdf.1.weight', False), + ('enc_stage_1.freq_branch.sequence.1.bias', 'encoding_blocks.0.tdf.1.bias', False), + ('enc_stage_1.freq_branch.sequence.1.running_mean', 'encoding_blocks.0.tdf.1.running_mean', False), + ('enc_stage_1.freq_branch.sequence.1.running_var', 'encoding_blocks.0.tdf.1.running_var', False), + ('enc_stage_1.freq_branch.sequence.3.weight', '802', True), + ('enc_stage_1.freq_branch.sequence.4.weight', 'encoding_blocks.0.tdf.4.weight', False), + ('enc_stage_1.freq_branch.sequence.4.bias', 'encoding_blocks.0.tdf.4.bias', False), + ('enc_stage_1.freq_branch.sequence.4.running_mean', 'encoding_blocks.0.tdf.4.running_mean', False), + ('enc_stage_1.freq_branch.sequence.4.running_var', 'encoding_blocks.0.tdf.4.running_var', False), + ('enc_down_1.0.weight', '697', False), + ('enc_down_1.0.bias', '698', False), + # Encoder Stage 2 + ('enc_stage_2.time_branch.sequence.0.weight', '700', False), + ('enc_stage_2.time_branch.sequence.0.bias', '701', False), + ('enc_stage_2.time_branch.sequence.2.weight', '703', False), + ('enc_stage_2.time_branch.sequence.2.bias', '704', False), + ('enc_stage_2.time_branch.sequence.4.weight', '706', False), + ('enc_stage_2.time_branch.sequence.4.bias', '707', False), + ('enc_stage_2.freq_branch.sequence.0.weight', '803', True), + ('enc_stage_2.freq_branch.sequence.1.weight', 'encoding_blocks.1.tdf.1.weight', False), + ('enc_stage_2.freq_branch.sequence.1.bias', 'encoding_blocks.1.tdf.1.bias', False), + ('enc_stage_2.freq_branch.sequence.1.running_mean', 'encoding_blocks.1.tdf.1.running_mean', False), + ('enc_stage_2.freq_branch.sequence.1.running_var', 'encoding_blocks.1.tdf.1.running_var', False), + ('enc_stage_2.freq_branch.sequence.3.weight', '804', True), + ('enc_stage_2.freq_branch.sequence.4.weight', 'encoding_blocks.1.tdf.4.weight', False), + ('enc_stage_2.freq_branch.sequence.4.bias', 'encoding_blocks.1.tdf.4.bias', False), + ('enc_stage_2.freq_branch.sequence.4.running_mean', 'encoding_blocks.1.tdf.4.running_mean', False), + ('enc_stage_2.freq_branch.sequence.4.running_var', 'encoding_blocks.1.tdf.4.running_var', False), + ('enc_down_2.0.weight', '709', False), + ('enc_down_2.0.bias', '710', False), + # Encoder Stage 3 + ('enc_stage_3.time_branch.sequence.0.weight', '712', False), + ('enc_stage_3.time_branch.sequence.0.bias', '713', False), + ('enc_stage_3.time_branch.sequence.2.weight', '715', False), + ('enc_stage_3.time_branch.sequence.2.bias', '716', False), + ('enc_stage_3.time_branch.sequence.4.weight', '718', False), + ('enc_stage_3.time_branch.sequence.4.bias', '719', False), + ('enc_stage_3.freq_branch.sequence.0.weight', '805', True), + ('enc_stage_3.freq_branch.sequence.1.weight', 'encoding_blocks.2.tdf.1.weight', False), + ('enc_stage_3.freq_branch.sequence.1.bias', 'encoding_blocks.2.tdf.1.bias', False), + ('enc_stage_3.freq_branch.sequence.1.running_mean', 'encoding_blocks.2.tdf.1.running_mean', False), + ('enc_stage_3.freq_branch.sequence.1.running_var', 'encoding_blocks.2.tdf.1.running_var', False), + ('enc_stage_3.freq_branch.sequence.3.weight', '806', True), + ('enc_stage_3.freq_branch.sequence.4.weight', 'encoding_blocks.2.tdf.4.weight', False), + ('enc_stage_3.freq_branch.sequence.4.bias', 'encoding_blocks.2.tdf.4.bias', False), + ('enc_stage_3.freq_branch.sequence.4.running_mean', 'encoding_blocks.2.tdf.4.running_mean', False), + ('enc_stage_3.freq_branch.sequence.4.running_var', 'encoding_blocks.2.tdf.4.running_var', False), + ('enc_down_3.0.weight', '721', False), + ('enc_down_3.0.bias', '722', False), + # Encoder Stage 4 + ('enc_stage_4.time_branch.sequence.0.weight', '724', False), + ('enc_stage_4.time_branch.sequence.0.bias', '725', False), + ('enc_stage_4.time_branch.sequence.2.weight', '727', False), + ('enc_stage_4.time_branch.sequence.2.bias', '728', False), + ('enc_stage_4.time_branch.sequence.4.weight', '730', False), + ('enc_stage_4.time_branch.sequence.4.bias', '731', False), + ('enc_stage_4.freq_branch.sequence.0.weight', '807', True), + ('enc_stage_4.freq_branch.sequence.1.weight', 'encoding_blocks.3.tdf.1.weight', False), + ('enc_stage_4.freq_branch.sequence.1.bias', 'encoding_blocks.3.tdf.1.bias', False), + ('enc_stage_4.freq_branch.sequence.1.running_mean', 'encoding_blocks.3.tdf.1.running_mean', False), + ('enc_stage_4.freq_branch.sequence.1.running_var', 'encoding_blocks.3.tdf.1.running_var', False), + ('enc_stage_4.freq_branch.sequence.3.weight', '808', True), + ('enc_stage_4.freq_branch.sequence.4.weight', 'encoding_blocks.3.tdf.4.weight', False), + ('enc_stage_4.freq_branch.sequence.4.bias', 'encoding_blocks.3.tdf.4.bias', False), + ('enc_stage_4.freq_branch.sequence.4.running_mean', 'encoding_blocks.3.tdf.4.running_mean', False), + ('enc_stage_4.freq_branch.sequence.4.running_var', 'encoding_blocks.3.tdf.4.running_var', False), + ('enc_down_4.0.weight', '733', False), + ('enc_down_4.0.bias', '734', False), + # Encoder Stage 5 + ('enc_stage_5.time_branch.sequence.0.weight', '736', False), + ('enc_stage_5.time_branch.sequence.0.bias', '737', False), + ('enc_stage_5.time_branch.sequence.2.weight', '739', False), + ('enc_stage_5.time_branch.sequence.2.bias', '740', False), + ('enc_stage_5.time_branch.sequence.4.weight', '742', False), + ('enc_stage_5.time_branch.sequence.4.bias', '743', False), + ('enc_stage_5.freq_branch.sequence.0.weight', '809', True), + ('enc_stage_5.freq_branch.sequence.1.weight', 'encoding_blocks.4.tdf.1.weight', False), + ('enc_stage_5.freq_branch.sequence.1.bias', 'encoding_blocks.4.tdf.1.bias', False), + ('enc_stage_5.freq_branch.sequence.1.running_mean', 'encoding_blocks.4.tdf.1.running_mean', False), + ('enc_stage_5.freq_branch.sequence.1.running_var', 'encoding_blocks.4.tdf.1.running_var', False), + ('enc_stage_5.freq_branch.sequence.3.weight', '810', True), + ('enc_stage_5.freq_branch.sequence.4.weight', 'encoding_blocks.4.tdf.4.weight', False), + ('enc_stage_5.freq_branch.sequence.4.bias', 'encoding_blocks.4.tdf.4.bias', False), + ('enc_stage_5.freq_branch.sequence.4.running_mean', 'encoding_blocks.4.tdf.4.running_mean', False), + ('enc_stage_5.freq_branch.sequence.4.running_var', 'encoding_blocks.4.tdf.4.running_var', False), + ('enc_down_5.0.weight', '745', False), + ('enc_down_5.0.bias', '746', False), + # Bottleneck + ('bottleneck.time_branch.sequence.0.weight', '748', False), + ('bottleneck.time_branch.sequence.0.bias', '749', False), + ('bottleneck.time_branch.sequence.2.weight', '751', False), + ('bottleneck.time_branch.sequence.2.bias', '752', False), + ('bottleneck.time_branch.sequence.4.weight', '754', False), + ('bottleneck.time_branch.sequence.4.bias', '755', False), + ('bottleneck.freq_branch.sequence.0.weight', '811', True), + ('bottleneck.freq_branch.sequence.1.weight', 'bottleneck_block.tdf.1.weight', False), + ('bottleneck.freq_branch.sequence.1.bias', 'bottleneck_block.tdf.1.bias', False), + ('bottleneck.freq_branch.sequence.1.running_mean', 'bottleneck_block.tdf.1.running_mean', False), + ('bottleneck.freq_branch.sequence.1.running_var', 'bottleneck_block.tdf.1.running_var', False), + ('bottleneck.freq_branch.sequence.3.weight', '812', True), + ('bottleneck.freq_branch.sequence.4.weight', 'bottleneck_block.tdf.4.weight', False), + ('bottleneck.freq_branch.sequence.4.bias', 'bottleneck_block.tdf.4.bias', False), + ('bottleneck.freq_branch.sequence.4.running_mean', 'bottleneck_block.tdf.4.running_mean', False), + ('bottleneck.freq_branch.sequence.4.running_var', 'bottleneck_block.tdf.4.running_var', False), + # Decoder Stage 1 + ('dec_up_1.0.weight', 'us.0.0.weight', False), + ('dec_up_1.0.bias', 'us.0.0.bias', False), + ('dec_up_1.1.weight', 'us.0.1.weight', False), + ('dec_up_1.1.bias', 'us.0.1.bias', False), + ('dec_up_1.1.running_mean', 'us.0.1.running_mean', False), + ('dec_up_1.1.running_var', 'us.0.1.running_var', False), + ('dec_stage_1.time_branch.sequence.0.weight', '757', False), + ('dec_stage_1.time_branch.sequence.0.bias', '758', False), + ('dec_stage_1.time_branch.sequence.2.weight', '760', False), + ('dec_stage_1.time_branch.sequence.2.bias', '761', False), + ('dec_stage_1.time_branch.sequence.4.weight', '763', False), + ('dec_stage_1.time_branch.sequence.4.bias', '764', False), + ('dec_stage_1.freq_branch.sequence.0.weight', '813', True), + ('dec_stage_1.freq_branch.sequence.1.weight', 'decoding_blocks.0.tdf.1.weight', False), + ('dec_stage_1.freq_branch.sequence.1.bias', 'decoding_blocks.0.tdf.1.bias', False), + ('dec_stage_1.freq_branch.sequence.1.running_mean', 'decoding_blocks.0.tdf.1.running_mean', False), + ('dec_stage_1.freq_branch.sequence.1.running_var', 'decoding_blocks.0.tdf.1.running_var', False), + ('dec_stage_1.freq_branch.sequence.3.weight', '814', True), + ('dec_stage_1.freq_branch.sequence.4.weight', 'decoding_blocks.0.tdf.4.weight', False), + ('dec_stage_1.freq_branch.sequence.4.bias', 'decoding_blocks.0.tdf.4.bias', False), + ('dec_stage_1.freq_branch.sequence.4.running_mean', 'decoding_blocks.0.tdf.4.running_mean', False), + ('dec_stage_1.freq_branch.sequence.4.running_var', 'decoding_blocks.0.tdf.4.running_var', False), + # Decoder Stage 2 + ('dec_up_2.0.weight', 'us.1.0.weight', False), + ('dec_up_2.0.bias', 'us.1.0.bias', False), + ('dec_up_2.1.weight', 'us.1.1.weight', False), + ('dec_up_2.1.bias', 'us.1.1.bias', False), + ('dec_up_2.1.running_mean', 'us.1.1.running_mean', False), + ('dec_up_2.1.running_var', 'us.1.1.running_var', False), + ('dec_stage_2.time_branch.sequence.0.weight', '766', False), + ('dec_stage_2.time_branch.sequence.0.bias', '767', False), + ('dec_stage_2.time_branch.sequence.2.weight', '769', False), + ('dec_stage_2.time_branch.sequence.2.bias', '770', False), + ('dec_stage_2.time_branch.sequence.4.weight', '772', False), + ('dec_stage_2.time_branch.sequence.4.bias', '773', False), + ('dec_stage_2.freq_branch.sequence.0.weight', '815', True), + ('dec_stage_2.freq_branch.sequence.1.weight', 'decoding_blocks.1.tdf.1.weight', False), + ('dec_stage_2.freq_branch.sequence.1.bias', 'decoding_blocks.1.tdf.1.bias', False), + ('dec_stage_2.freq_branch.sequence.1.running_mean', 'decoding_blocks.1.tdf.1.running_mean', False), + ('dec_stage_2.freq_branch.sequence.1.running_var', 'decoding_blocks.1.tdf.1.running_var', False), + ('dec_stage_2.freq_branch.sequence.3.weight', '816', True), + ('dec_stage_2.freq_branch.sequence.4.weight', 'decoding_blocks.1.tdf.4.weight', False), + ('dec_stage_2.freq_branch.sequence.4.bias', 'decoding_blocks.1.tdf.4.bias', False), + ('dec_stage_2.freq_branch.sequence.4.running_mean', 'decoding_blocks.1.tdf.4.running_mean', False), + ('dec_stage_2.freq_branch.sequence.4.running_var', 'decoding_blocks.1.tdf.4.running_var', False), + # Decoder Stage 3 + ('dec_up_3.0.weight', 'us.2.0.weight', False), + ('dec_up_3.0.bias', 'us.2.0.bias', False), + ('dec_up_3.1.weight', 'us.2.1.weight', False), + ('dec_up_3.1.bias', 'us.2.1.bias', False), + ('dec_up_3.1.running_mean', 'us.2.1.running_mean', False), + ('dec_up_3.1.running_var', 'us.2.1.running_var', False), + ('dec_stage_3.time_branch.sequence.0.weight', '775', False), + ('dec_stage_3.time_branch.sequence.0.bias', '776', False), + ('dec_stage_3.time_branch.sequence.2.weight', '778', False), + ('dec_stage_3.time_branch.sequence.2.bias', '779', False), + ('dec_stage_3.time_branch.sequence.4.weight', '781', False), + ('dec_stage_3.time_branch.sequence.4.bias', '782', False), + ('dec_stage_3.freq_branch.sequence.0.weight', '817', True), + ('dec_stage_3.freq_branch.sequence.1.weight', 'decoding_blocks.2.tdf.1.weight', False), + ('dec_stage_3.freq_branch.sequence.1.bias', 'decoding_blocks.2.tdf.1.bias', False), + ('dec_stage_3.freq_branch.sequence.1.running_mean', 'decoding_blocks.2.tdf.1.running_mean', False), + ('dec_stage_3.freq_branch.sequence.1.running_var', 'decoding_blocks.2.tdf.1.running_var', False), + ('dec_stage_3.freq_branch.sequence.3.weight', '818', True), + ('dec_stage_3.freq_branch.sequence.4.weight', 'decoding_blocks.2.tdf.4.weight', False), + ('dec_stage_3.freq_branch.sequence.4.bias', 'decoding_blocks.2.tdf.4.bias', False), + ('dec_stage_3.freq_branch.sequence.4.running_mean', 'decoding_blocks.2.tdf.4.running_mean', False), + ('dec_stage_3.freq_branch.sequence.4.running_var', 'decoding_blocks.2.tdf.4.running_var', False), + # Decoder Stage 4 + ('dec_up_4.0.weight', 'us.3.0.weight', False), + ('dec_up_4.0.bias', 'us.3.0.bias', False), + ('dec_up_4.1.weight', 'us.3.1.weight', False), + ('dec_up_4.1.bias', 'us.3.1.bias', False), + ('dec_up_4.1.running_mean', 'us.3.1.running_mean', False), + ('dec_up_4.1.running_var', 'us.3.1.running_var', False), + ('dec_stage_4.time_branch.sequence.0.weight', '784', False), + ('dec_stage_4.time_branch.sequence.0.bias', '785', False), + ('dec_stage_4.time_branch.sequence.2.weight', '787', False), + ('dec_stage_4.time_branch.sequence.2.bias', '788', False), + ('dec_stage_4.time_branch.sequence.4.weight', '790', False), + ('dec_stage_4.time_branch.sequence.4.bias', '791', False), + ('dec_stage_4.freq_branch.sequence.0.weight', '819', True), + ('dec_stage_4.freq_branch.sequence.1.weight', 'decoding_blocks.3.tdf.1.weight', False), + ('dec_stage_4.freq_branch.sequence.1.bias', 'decoding_blocks.3.tdf.1.bias', False), + ('dec_stage_4.freq_branch.sequence.1.running_mean', 'decoding_blocks.3.tdf.1.running_mean', False), + ('dec_stage_4.freq_branch.sequence.1.running_var', 'decoding_blocks.3.tdf.1.running_var', False), + ('dec_stage_4.freq_branch.sequence.3.weight', '820', True), + ('dec_stage_4.freq_branch.sequence.4.weight', 'decoding_blocks.3.tdf.4.weight', False), + ('dec_stage_4.freq_branch.sequence.4.bias', 'decoding_blocks.3.tdf.4.bias', False), + ('dec_stage_4.freq_branch.sequence.4.running_mean', 'decoding_blocks.3.tdf.4.running_mean', False), + ('dec_stage_4.freq_branch.sequence.4.running_var', 'decoding_blocks.3.tdf.4.running_var', False), + # Decoder Stage 5 + ('dec_up_5.0.weight', 'us.4.0.weight', False), + ('dec_up_5.0.bias', 'us.4.0.bias', False), + ('dec_up_5.1.weight', 'us.4.1.weight', False), + ('dec_up_5.1.bias', 'us.4.1.bias', False), + ('dec_up_5.1.running_mean', 'us.4.1.running_mean', False), + ('dec_up_5.1.running_var', 'us.4.1.running_var', False), + ('dec_stage_5.time_branch.sequence.0.weight', '793', False), + ('dec_stage_5.time_branch.sequence.0.bias', '794', False), + ('dec_stage_5.time_branch.sequence.2.weight', '796', False), + ('dec_stage_5.time_branch.sequence.2.bias', '797', False), + ('dec_stage_5.time_branch.sequence.4.weight', '799', False), + ('dec_stage_5.time_branch.sequence.4.bias', '800', False), + ('dec_stage_5.freq_branch.sequence.0.weight', '821', True), + ('dec_stage_5.freq_branch.sequence.1.weight', 'decoding_blocks.4.tdf.1.weight', False), + ('dec_stage_5.freq_branch.sequence.1.bias', 'decoding_blocks.4.tdf.1.bias', False), + ('dec_stage_5.freq_branch.sequence.1.running_mean', 'decoding_blocks.4.tdf.1.running_mean', False), + ('dec_stage_5.freq_branch.sequence.1.running_var', 'decoding_blocks.4.tdf.1.running_var', False), + ('dec_stage_5.freq_branch.sequence.3.weight', '822', True), + ('dec_stage_5.freq_branch.sequence.4.weight', 'decoding_blocks.4.tdf.4.weight', False), + ('dec_stage_5.freq_branch.sequence.4.bias', 'decoding_blocks.4.tdf.4.bias', False), + ('dec_stage_5.freq_branch.sequence.4.running_mean', 'decoding_blocks.4.tdf.4.running_mean', False), + ('dec_stage_5.freq_branch.sequence.4.running_var', 'decoding_blocks.4.tdf.4.running_var', False), + # Final Layer + ('final_conv.weight', 'final_conv.0.weight', False), + ('final_conv.bias', 'final_conv.0.bias', False), diff --git a/doc/Resources.md b/doc/Resources.md new file mode 100644 index 0000000..b286a97 --- /dev/null +++ b/doc/Resources.md @@ -0,0 +1,16 @@ +# MDX-Net + +- [RVC Studio](https://github.com/SayanoAI/RVC-Studio): has a [HuggingFace repo](https://huggingface.co/datasets/SayanoAI/RVC-Studio) + - [Kim Vocals 2](https://huggingface.co/datasets/SayanoAI/RVC-Studio/resolve/main/MDXNET/Kim_Vocal_2.onnx?download=true) + - [UVR-MDX-NET-vocal_FT](https://huggingface.co/datasets/SayanoAI/RVC-Studio/resolve/main/MDXNET/UVR-MDX-NET-vocal_FT.onnx?download=true) + - Contains other UVR files + - MIT License +- https://github.com/Anjok07/ultimatevocalremovergui +- https://huggingface.co/seanghay/uvr_models/resolve/main/ +- https://huggingface.co/spaces/r3gm/SoniTranslate_translate_audio_of_a_video_content/tree/main +- https://huggingface.co/Blane187/all_public_uvr_models/tree/main +- https://huggingface.co/Politrees/UVR_resources +- https://github.com/Politrees/UVR_resources/tree/main +- https://huggingface.co/spaces/ake178178/Ultimate-Vocal-Remover-WebUI-kb +- https://arxiv.org/pdf/2111.12203 +- https://huggingface.co/set-soft/audio_separation