Source-linked AI summary
MMDenseLSTM: An efficient combination of convolutional and recurrent neural networks for audio source separation
Naoya Takahashi, Nabarun Goswami, Yuki Mitsufuji
TL;DR
Audio source separation needs models that capture long-term context without the cost of simply blending separate CNN and recurrent systems. MMDenseLSTM integrates LSTM and DenseNet across multiple scales and bands, outperforming MMDenseNet, LSTM, and their blend, including an ideal binary mask in a singing voice task.
Problem
Audio source separation requires modeling long temporal contexts, while standard CNNs need considerable depth and simple system blending increases model size and computational cost.
Method
MMDenseLSTM combines DenseNet and LSTM at multiple scales and frequency bands, selectively inserting LSTM blocks at low scales.
Results
MMDenseLSTM achieves state-of-the-art results on DSD100 and MUSDB18, outperforms a naive blend with fewer parameters, and surpasses an ideal binary mask for singing voice separation.
Takeaways & Limitations
Combining DenseNet and LSTM within one multi-scale architecture improves separation performance and efficiency compared with separate-system blending.
Abstract
from arXiv · showhide
Deep neural networks have become an indispensable technique for audio source separation (ASS). It was recently reported that a variant of CNN architecture called MMDenseNet was successfully employed to solve the ASS problem of estimating source amplitudes, and state-of-the-art results were obtained for DSD100 dataset. To further enhance MMDenseNet, here we propose a novel architecture that integrates long short-term memory (LSTM) in multiple scales with skip connections to efficiently model long-term structures within an audio context. The experimental results show that the proposed method outperforms MMDenseNet, LSTM and a blend of the two networks. The number of parameters and processing time of the proposed model are significantly less than those for simple blending. Furthermore, the proposed method yields better results than those obtained using ideal binary masks for a singing voice separation task.
1. INTRODUCTION
MMDenseLSTM combines DenseNet and LSTM across multiple scales and frequency bands to improve audio source separation while retaining a small model size. It outperforms prior methods on DSD100 and MUSDB18 and can surpass an ideal binary mask in a singing voice task.
- MMDenseNet extends DenseNet with multi-scale processing and frequency-band-specific networks for audio spectrogram modeling.Multi-scale processing addresses long contexts, while band-specific networks accommodate different local structures across frequency bands.
- MMDenseLSTM combines DenseNet and LSTM at multiple scales and bands, increasing separation performance and efficiency.The architecture combines the two components at multiple low scales rather than operating at a single scale.
- The proposed method outperforms current state-of-the-art methods on the DSD100 and MUSDB18 datasets.
- MMDenseLSTM can outperform an ideal binary mask for singing voice separation when trained with a larger dataset.The reported training condition uses 900 songs.
2. MULTI-SCALE MULTI-BAND DENSELSTM
The proposed architecture integrates dense blocks and LSTM blocks within a multi-scale, multi-band DenseNet design. LSTM blocks are selectively placed at low-resolution scales to model global structure while limiting model size.
- 2.1. MMDenseNet: MDenseNet applies dense blocks across progressively downsampled and upsampled scales, with skip connections between blocks at the same scale.
- 2.1. MMDenseNet: MMDenseNet splits spectrogram inputs into multiple frequency bands and applies a band-dedicated MDenseNet to each band.
- 2.2. Combining LSTM with MMDenseNet: Low-scale LSTM insertion captures global structure with fewer parameters, while dense blocks at scale 1 model fine local structures.
- 2.2. Combining LSTM with MMDenseNet: Naively inserting LSTM blocks at every scale greatly increases model size, motivating selective placement in the upsampling path at low scales.
- 2.2. Combining LSTM with MMDenseNet: MMDenseLSTM concatenates outputs from band-specific and full-band MDenseLSTM modules, then integrates them with a final dense block.
- 2.3. Architectural details: LSTM blocks are inserted only at bottlenecks and selected scale-2 upsampling blocks, reducing model size while retaining a 356-frame effective context.
3. EXPERIMENTS
Experiments validate MMDenseLSTM’s architecture choices and show gains over established separation systems on DSD100 and MUSDB18. Low-scale LSTM insertion improves efficiency, while larger-scale training yields strong benchmark performance with fewer parameters than naive blending.
- Architecture validation: The Sa configuration performed best because LSTM layers modeled global modulations using local features extracted by dense layers at that scale.Subsequent experiments used the Sa configuration.
- Architecture validation: Low-scale LSTM insertion in the up-scaling path produced the best performance among seven tested insertion scales.The comparison used a single 64-unit LSTM layer in band 1 and measured parameter increases alongside MSE.
- Architecture validation: LSTM feature-map norms were comparable to the highest dense feature-map norms, while lower-norm dense maps learned sparse local features.This analysis compared feature maps within the LSTM block d4 of band 1.
- Comparison with state-of-the-art methods: 0.43dB over MMDenseNet and 0.41dB over BLEND2, MMDenseLSTM achieved state-of-the-art results on MUSDB18 while using 24 times fewer parameters than naive BLSTM–MMDenseNet combination.The larger training set contained 900 songs, and MMDenseLSTM also outperformed the ideal binary mask for accompaniment.
4. CONCLUSION
MMDenseLSTM efficiently combines DenseNet and LSTM to improve audio source separation. It achieves state-of-the-art results on DSD100 and MUSDB18, surpasses naive blending with fewer parameters, and exceeds the ideal binary mask for singing voice separation, though gains are smaller for bass.
- 4. CONCLUSION: MMDenseLSTM achieves state-of-the-art results on DSD100 and MUSDB18 while outperforming naive BLSTM–MMDenseNet blending with far fewer parameters.For singing voice separation trained on 900 songs, it also outperformed an ideal binary mask; improvement over MMDenseNet was smaller for bass.
- 4. CONCLUSION: MMDenseLSTM outperformed an ideal binary mask for singing voice separation when trained with 900 songs.
- 4. CONCLUSION: Improvement over MMDenseNet was smaller for bass, which the authors identify for future investigation.