Source-linked AI summary
Multi-scale Multi-band DenseNets for Audio Source Separation
Naoya Takahashi, Yuki Mitsufuji
TL;DR
Audio source separation needs models that handle long contexts and detailed spectrogram structure efficiently. This paper extends DenseNet with multi-scale processing, skip connections, and frequency-band-specific dense blocks. On SiSEC 2016, the approach outperforms the state of the art while using fewer parameters and less training time.
Problem
Audio source separation requires estimating instrumental spectrograms from mixtures while modeling long contexts and complex spectrogram structure.
Method
The paper extends DenseNet with down-sampling and up-sampling across multi-scale dense blocks, inter-block skip connections, and parallel full-band and sub-band networks.
Results
The proposed approach outperforms the state of the art on SiSEC 2016 while substantially reducing model size and training time.
Takeaways & Limitations
Multi-scale and multi-band DenseNet architectures provide an efficient approach to modeling both global context and fine-grained spectrogram structure for source separation.
Abstract
from arXiv · showhide
This paper deals with the problem of audio source separation. To handle the complex and ill-posed nature of the problems of audio source separation, the current state-of-the-art approaches employ deep neural networks to obtain instrumental spectra from a mixture. In this study, we propose a novel network architecture that extends the recently developed densely connected convolutional network (DenseNet), which has shown excellent results on image classification tasks. To deal with the specific problem of audio source separation, an up-sampling layer, block skip connection and band-dedicated dense blocks are incorporated on top of DenseNet. The proposed approach takes advantage of long contextual information and outperforms state-of-the-art results on SiSEC 2016 competition by a large margin in terms of signal-to-distortion ratio. Moreover, the proposed architecture requires significantly fewer parameters and considerably less training time compared with other methods.
1. INTRODUCTION
Audio source separation requires models that capture long temporal contexts while preserving detailed spectrogram structure. The paper extends DenseNet with multi-scale and frequency-band-specific processing to improve separation performance and efficiency.
- Motivation: Audio source separation has progressed from classical models to deep networks, with FNNs using multiple frames and LSTMs modeling longer contexts.The cited introduction describes local Gaussian, non-negative factorization, kernel additive, FNN, and LSTM approaches.
- Motivation: Deep CNNs can capture long contexts through larger receptive fields, but considerable depth complicates training and can degrade performance.
- DenseNet foundation: DenseNet concatenates preceding feature maps, enabling cross-layer interactions and feature reuse that efficiently support spectrogram estimation.This feature reuse is presented as particularly suitable for estimating instrumental spectrograms buried in interference.
- Proposed architecture: The proposed multi-scale DenseNet uses down-sampling and up-sampling across dense blocks so lower resolutions capture global context while higher resolutions recover spectrogram details.The architecture is designed to model long contexts and fine-grained structures within a practical model size.
- Contributions: The paper adds frequency-band-specific dense blocks and reports that the proposed method largely outperforms the best SiSEC 2016 state-of-the-art score while reducing parameters and training time.Band-specific modeling lets kernels focus on distributions that differ across frequency bands.
2. MULTI-SCALE MULTI-BAND DENSENET
The proposed architecture extends DenseNet with multi-scale processing, up-sampling, and inter-block skip connections, then specializes modeling across frequency bands. MDenseNets for individual bands and the full band are combined to produce the final output.
- DenseNet: DenseNet concatenates outputs from all preceding layers, enabling feature reuse and efficient parameter usage.The growth rate k controls the linear increase in input feature maps with depth.
- Multi-Scale DenseNet: MDenseNet alternates dense blocks with down-sampling and up-sampling layers to capture long contexts while recovering higher-resolution feature maps.Inter-block skip connections link dense blocks at the same scale, allowing signal flow without passing through lower-resolution blocks.
- Multi-band MDenseNet: MMDenseNet applies multi-scale DenseNets to separate frequency bands and the full band, then concatenates their outputs before a final dense block.Band-limited networks capture fine structure, while the full-band network models rough global structure.
- Multi-band MDenseNet: The multi-band design permits separate architectures and computational-resource allocation for each frequency band.In this work, the frequency range is split into two bands, with a relatively larger model assigned to the lower-frequency band.
3. EXPERIMENTS
The experiments evaluate the proposed architectures on DSD100 using SDR and compare them with state-of-the-art baselines. MMDenseNet improves separation performance, while multi-scale paths contribute and DenseNet-based models reduce parameters and training time.
- 3.1. Setup: The method was evaluated on DSD100, using mixture spectrograms to estimate bass, drums, other, and vocal source spectrograms.The dataset contains separate development and test sets, with stereo songs sampled at 44.1 kHz.
- 3.2. State of the art comparison: MMDenseNet significantly improved SDR and largely outperformed all baselines, while MMDenseNet+ achieved the best overall result.MMDenseNet+ improved every instrument except drums after adding MedleyDB training data.
- 3.2. State of the art comparison: 0.97 dB improvement was obtained on average over the best SiSEC 2016 results.The comparison used signal-to-distortion ratio computed with the BSS Eval toolbox.
- 3.3. Architecture validation: The skip-connection and up-sampling paths had roughly equal average filter-weight l2-norms, indicating contributions from dense blocks at different scales.The analysis compares filter weights in the up-sampling path with those in the skip-connection path.
- 3.4. Model efficiency: MDenseNet matched BLEND with 1.5% of its parameters, while MMDenseNet outperformed BLEND with 3.6% of its parameters.The proposed models also required less training time than BLSTM and BLEND.
4. CONCLUSION
The paper extends DenseNet for audio source separation with multi-scale and multi-band architectures. The resulting models capture global and local structure, improve performance on DSD100, and reduce model size and training time.
- 4. CONCLUSION: The proposed architectures connect dense blocks across multiple scales through down-sampling and up-sampling layers.This design models fine-grained local structure and global structure.
- 4. CONCLUSION: The multi-band DenseNet improves convolutional-kernel modeling efficiency and produces considerable performance improvement.Frequency bands can receive architectures suited to their differing importance.
- 4. CONCLUSION: On the SiSEC 2016 DSD100 dataset, the approach outperforms the state of the art while significantly reducing model size and training time.This is the paper’s concluding experimental claim.