Source-linked AI summary
Universal Sound Separation
Ilya Kavalerov, Scott Wisdom, Hakan Erdogan, Brian Patton, Kevin Wilson, Jonathan Le Roux, John R. Hershey
TL;DR
Speech separation methods had not been tested systematically on mixtures of arbitrary sound types, leaving their generality unknown. The paper introduces a broad sound-mixture dataset and compares mask-based architectures, transforms, and window sizes. STFTs outperform learned bases for universal separation, while the best methods achieve 13.4 dB SI-SDRi for speech/non-speech separation and 9.8 dB for two-source universal separation.
Problem
It is unknown whether speech separation methods learn general cues that apply to mixtures of arbitrary sounds.
Method
The paper constructs a dataset of mixtures containing many sound types and systematically compares masking architectures, analysis-synthesis bases, and window sizes.
Results
STFTs outperform learned bases for universal sound separation, while the best models achieve 13.4 dB SI-SDRi for speech/non-speech separation and 9.8 dB for two-source universal separation.
Takeaways & Limitations
Optimal basis and window size depend on the task: learned bases suit speech/non-speech separation, whereas STFTs suit universal sound separation.
Abstract
from arXiv · showhide
Recent deep learning approaches have achieved impressive performance on speech enhancement and separation tasks. However, these approaches have not been investigated for separating mixtures of arbitrary sounds of different types, a task we refer to as universal sound separation, and it is unknown how performance on speech tasks carries over to non-speech tasks. To study this question, we develop a dataset of mixtures containing arbitrary sounds, and use it to investigate the space of mask-based separation architectures, varying both the overall network architecture and the framewise analysis-synthesis basis for signal transformations. These network architectures include convolutional long short-term memory networks and time-dilated convolution stacks inspired by the recent success of time-domain enhancement networks like ConvTasNet. For the latter architecture, we also propose novel modifications that further improve separation performance. In terms of the framewise analysis-synthesis basis, we explore both a short-time Fourier transform (STFT) and a learnable basis, as used in ConvTasNet. For both of these bases, we also examine the effect of window size. In particular, for STFTs, we find that longer windows (25-50 ms) work best for speech/non-speech separation, while shorter windows (2.5 ms) work best for arbitrary sounds. For learnable bases, shorter windows (2.5 ms) work best on all tasks. Surprisingly, for universal sound separation, STFTs outperform learnable bases. Our best methods produce an improvement in scale-invariant signal-to-distortion ratio of over 13 dB for speech/non-speech separation and close to 10 dB for universal sound separation.
1. INTRODUCTION
The paper introduces universal sound separation to test whether separation methods developed for speech can generalize to mixtures of arbitrary sounds. It constructs a broad sound-mixture dataset, evaluates multiple architectures and transforms, and reports strong improvements.
- Universal sound separation asks whether models learn general separation cues or mainly exploit speech-specific sound structure.
- The paper constructs a dataset containing mixtures of hundreds of sound types to study separation beyond speech.
- Almost 10 dB average SI-SDR improvement is achieved on universal sound separation.
- The experiments systematically compare masking networks, analysis-synthesis transforms, and window sizes across speech/non-speech and universal separation tasks.
- The study evaluates ConvTasNet on both speech/non-speech and universal sound separation for the first time.
- The proposed architectural variations include alternative normalization, improved initialization, longer-range skip-residual connections, and iterative processing.
2. PRIOR WORK
Prior work largely targets speech or limited non-speech mixtures, while this paper investigates arbitrary real-world sounds drawn from many sound classes. Its architecture comparisons build on established mask-based separation approaches and leave embedding-space methods for future work.
- Mask-based multi-source separation has used BLSTMs, CNNs, DenseNet-LSTM systems, encoder-decoder networks, and TDCNs.
- Embedding-space approaches such as deep clustering are acknowledged as effective but are left for future work.
- Earlier source-separation studies focused mainly on speech enhancement, speech separation, or restricted combinations such as distress sounds, speech, music, and singing voice.
- The paper claims the first investigation of arbitrary real-world sound separation sourced from a large number of sound classes.
3. MODELS
The models use neural mask predictors with either CLDNN or TDCN-family architectures and compare STFT with learned analysis-synthesis bases. The study also tests architectural improvements and iterative refinement.
- Masking networks: Masking networks predict sigmoid-valued coefficients between 0 and 1 that modulate analysis-basis coefficients.
- Masking network architectures: CLDNN combines 14 dilated 2D convolutional layers, a bidirectional LSTM, and two dense layers.
- Masking network architectures: TDCN++ modifies ConvTasNet’s TDCN with feature-wise normalization, longer-range skip-residual connections, and learnable dense-layer scaling.
- Masking network architectures: iTDCN++ feeds initial separated estimates and the original mixture into a second TDCN++ separation network.
- Analysis-synthesis bases: The learnable analysis transform operates framewise like an STFT, while synthesis uses a transposed convolution to reconstruct an overlap-added time-domain signal.
- Analysis-synthesis bases: Window size is optimized for each architecture, transform, and data type, with mixture-consistency projections applied to all outputs.
4. EXPERIMENTS
Experiments evaluate mask-based separation systems across speech/non-speech and universal sound tasks, varying network architecture, analysis-synthesis basis, and window size. Results show task-dependent optimal windows and basis preferences, with strong SI-SDR improvement on both speech/non-speech and universal separation.
- Dataset and evaluation: The dataset contains mixtures of diverse sound types, enabling evaluation of two-source and three-source universal sound separation.Source material includes crawling insects, animal calls, doors, construction noises, instruments, speech, music, and artificial sounds.
- Systems and setup: Experiments compare CLDNN and TDCN masking networks with STFT or learnable analysis-synthesis bases across multiple window sizes.All systems predict sigmoid masks over basis coefficients, and models are compared using architecture- and basis-specific optimal windows selected by cross-validation.
- Window-size effects: 25-50 ms windows are best for speech/non-speech separation, whereas universal sound separation favors shorter windows across masking networks and bases.For universal separation, TDCNs prefer 2.5 or 5 ms frames, while the CLDNN optimum is 5 ms or less.
- Basis comparisons: STFTs outperform learnable bases for universal sound separation, while learnable bases achieve the best speech/non-speech results.The reported task-dependent preference is attributed to the broader variety and more percussive structure of arbitrary sounds versus speech.
- Results: The iTDCN++ with a 2.5 ms STFT achieves 9.8 dB and 8.7 dB average SI-SDR improvement for two-source and three-source universal separation, respectively.For speech/non-speech separation, the iTDCN++ with a 2.5 ms learnable basis achieves 13.4 dB SI-SDRi.
- Results: Oracle binary masking achieves 16.3 dB mean SI-SDRi on the two-source universal test set, indicating remaining potential on the dataset.Figure 3 compares the best model with oracle masking using a 10 ms STFT window and 5 ms hop.
5. CONCLUSION
The paper establishes universal sound separation as a new task, evaluates architectures and signal representations, and finds strong performance across speech/non-speech and arbitrary-sound separation.
- Almost 10 dB average SI-SDR improvement is achieved on universal sound separation, while speech/non-speech separation improves by over 13 dB.The best models use iterative TDCN++ architecture.
- STFTs work better for arbitrary sound separation, whereas learned bases work better for speech/non-speech separation.The optimal basis and window size differ between speech and arbitrary-sound tasks.
- The study compares network architectures, analysis-synthesis transforms, and window sizes for both speech/non-speech and universal sound separation.It also introduces longer-range skip-residual connections and iterative processing as architectural variations.