Source-linked AI summary
Attention is All You Need in Speech Separation
Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, Jianyuan Zhong
TL;DR
Speech separation systems need long-range modeling, but RNNs constrain parallel computation and Transformer-based monaural separation remains relatively underexplored. SepFormer addresses this with an RNN-free, dual-path multi-scale Transformer masking network that learns short- and long-term dependencies. It reaches state-of-the-art performance on WSJ0-2mix and WSJ0-3mix while retaining competitive performance with 8× downsampling and reducing speed and memory demands.
Problem
RNN-based speech separation models handle long input sequences but limit parallel computation, while Transformer-based monaural separation has received little research.
Method
SepFormer is an RNN-free model using a dual-path, multi-scale Transformer pipeline to learn short- and long-term dependencies for speech separation.
Results
SepFormer achieves state-of-the-art performance with 22.3 dB SI-SNRi on WSJ0-2mix and 19.5 dB SI-SNRi on WSJ0-3mix.
Takeaways & Limitations
Parallel computation and competitive performance with 8× encoded-representation downsampling make SepFormer faster and less memory-demanding than recent RNN-based separation systems.
Abstract
from arXiv · showhide
Recurrent Neural Networks (RNNs) have long been the dominant architecture in sequence-to-sequence learning. RNNs, however, are inherently sequential models that do not allow parallelization of their computations. Transformers are emerging as a natural alternative to standard RNNs, replacing recurrent computations with a multi-head attention mechanism. In this paper, we propose the SepFormer, a novel RNN-free Transformer-based neural network for speech separation. The SepFormer learns short and long-term dependencies with a multi-scale approach that employs transformers. The proposed model achieves state-of-the-art (SOTA) performance on the standard WSJ0-2/3mix datasets. It reaches an SI-SNRi of 22.3 dB on WSJ0-2mix and an SI-SNRi of 19.5 dB on WSJ0-3mix. The SepFormer inherits the parallelization advantages of Transformers and achieves a competitive performance even when downsampling the encoded representation by a factor of 8. It is thus significantly faster and it is less memory-demanding than the latest speech separation systems with comparable performance.
1. INTRODUCTION
RNNs model long-term speech contexts but their sequential computation limits parallelization, motivating Transformer-based separation. SepFormer replaces RNNs with a multi-scale Transformer pipeline and reports state-of-the-art separation with faster, less memory-demanding processing.
- RNNs learn long-term dependencies and manage speech contexts, but their sequential computation impairs effective parallelization.
- Transformer-based models remain relatively underexplored for monaural audio source separation, despite strong progress from deep learning and end-to-end systems.
- SepFormer replaces DPRNN's RNN components with multi-scale Transformers that learn short- and long-term dependencies within a dual-path framework.
- 22.3 dB SI-SNRi on WSJ0-2mix and 19.5 dB SI-SNRi on WSJ0-3mix establish SepFormer's reported state-of-the-art performance.
- SepFormer processes time steps in parallel and remains competitive with 8× downsampling, making it faster and less memory-demanding than recent RNN-based systems.
2. THE MODEL
SepFormer combines learned-domain masking with a dual-path Transformer masking network that models short- and long-term dependencies, then reconstructs separated sources from masked encoder representations.
- 2.2. Masking Network: SepFormer uses an encoder, Transformer-based masking network, and decoder within a learned-domain masking architecture.The encoder produces representations, the masking network estimates speaker masks, and the decoder reconstructs separated time-domain signals.
- 2.1. Encoder: The encoder converts the time-domain mixture x into an STFT-like representation h using a single convolutional layer.The convolutional stride affects the model’s performance, speed, and memory use.
- 2.2. Masking Network: The masking network normalizes h, projects it linearly, and divides it into overlapping chunks represented as h′.Chunks have size C and a 50% overlap between adjacent chunks.
- 2.3. SepFormer Block: The SepFormer block applies IntraTransformer within each chunk and InterTransformer across chunks to model short- and long-term dependencies.The transformation is h′′ = f_inter(P(f_intra(h′))), and the block is repeated N times.
- 2.4. Decoder: The masking output is converted into speaker-specific masks, multiplied element-wise with encoder features, and decoded by transposed convolution.The decoder uses the encoder’s stride and kernel size, producing the separated source b_s^k.
- 2.3.1. Intra and Inter Transformers: Transformer layers add sinusoidal positional encoding, layer normalization, multi-head attention, feed-forward processing, and residual connections.Positional encoding supplies sequence-order information, while residual connections span Transformer layers and the overall Transformer architecture.
3. EXPERIMENTAL SETUP
The experiments evaluate SepFormer on WSJ0-2mix and WSJ0-3mix using controlled mixtures and specified training configurations, including chunked Transformer settings and augmentation.
- 3.1. Dataset: WSJ0-2mix and WSJ0-3mix contain mixtures of two and three speakers created from WSJ0 utterances.Source levels are sampled uniformly from 0 dB to 5 dB, and the waveform sampling rate is 8 kHz.
- 3.1. Dataset: The datasets use 30 hours for training, 10 hours for validation, and 5 hours for testing, with different speakers across training and test sets.This setup separates speaker identities between training and evaluation.
- 3.2. Model Configuration: The encoder uses 256 convolutional filters with kernel size 16 samples and stride 8 samples, while the decoder mirrors these settings.The decoder uses the same kernel size and stride factor as the encoder.
- 3.2. Model Configuration: The best masking networks use chunks of C = 250 with 50% overlap, 8 IntraT and 8 InterT layers, and N = 2 dual-path repetitions.Each Transformer layer uses 8 attention heads and 1024-dimensional positional feed-forward networks; the model has 26 million parameters.
- 3.3. Data Augmentation: Dynamic mixing creates new mixtures from single-speaker sources and is extended with speed perturbation between 95% slow-down and 105% speed-up.The augmentation is applied on the fly before mixing the sources.
- 3.4. Training: Training uses Adam with learning rate 15e−5, gradient clipping at L2 norm 5, batch size 1, and utterance-level permutation-invariant SI-SNR.The learning rate is annealed after validation performance fails to improve for the specified number of epochs.
4. RESULTS
SepFormer achieves state-of-the-art separation performance on WSJ0-2mix and WSJ0-3mix while enabling parallel computation and favorable speed–memory trade-offs. Ablations show that transformer depth, positional encoding, and dynamic mixing affect performance.
- 22.3 dB SI-SNRi and 22.4 dB SDRi are achieved on the WSJ0-2mix test set with dynamic mixing.
- Ablation Study: Eight IntraT and InterT layers replicated twice produce the best ablation results, while a single InterT layer still reaches 19.2 dB.
- Ablation Study: Positional encoding and dynamic mixing improve performance, while eight and sixteen attention heads produce only a slight difference.
- 19.5 dB SI-SNRi and 19.7 dB SDRi are achieved on WSJ0-3mix, outperforming previously proposed systems.
- Speed and Memory Comparison: Parallelizing computations across time steps enables faster training and inference than RNN-based systems.
- Speed and Memory Comparison: SepFormer trains faster than DPRNN and DPTNeT under matched hardware and training settings.
- Speed and Memory Comparison: SepFormer is faster and less memory-demanding than DPTNet, DPRNN, and Wavesplit, partly because stride-factor-8 encoding processes less data.
5. CONCLUSIONS
SepFormer is an RNN-free Transformer architecture for speech separation that learns short- and long-term dependencies through a multi-scale masking network. It achieves state-of-the-art separation on WSJ0-2mix and WSJ0-3mix while enabling parallel computation and reduced memory usage.
- SepFormer uses a transformer-only masking network with a multi-scale approach to learn short- and long-term dependencies.
- Parallel computation across time steps enables significant speed-up during training and inference.
- Downsampling the encoded representation by a factor of 8 maintains competitive performance while reducing memory usage.
- The reported efficiency gains are especially notable relative to recent models such as DPRNN, DPTNet, and Wavesplit.