PaperlayerView exact revision on arXiv

Source-linked AI summary

Attention is All You Need in Speech Separation

Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, Jianyuan Zhong

arXiv:2010.13154v2eess.AScs.LGcs.SDeess.SP

TL;DR

Speech separation systems need long-range modeling, but RNNs constrain parallel computation and Transformer-based monaural separation remains relatively underexplored. SepFormer addresses this with an RNN-free, dual-path multi-scale Transformer masking network that learns short- and long-term dependencies. It reaches state-of-the-art performance on WSJ0-2mix and WSJ0-3mix while retaining competitive performance with 8× downsampling and reducing speed and memory demands.

  • Problem

    RNN-based speech separation models handle long input sequences but limit parallel computation, while Transformer-based monaural separation has received little research.

    Source: §1 · p. 1

  • Method

    SepFormer is an RNN-free model using a dual-path, multi-scale Transformer pipeline to learn short- and long-term dependencies for speech separation.

    Source: §2.2 · p. 2

  • Results

    SepFormer achieves state-of-the-art performance with 22.3 dB SI-SNRi on WSJ0-2mix and 19.5 dB SI-SNRi on WSJ0-3mix.

    Source: §1 · p. 1, §4.3 · p. 4

  • Takeaways & Limitations

    Parallel computation and competitive performance with 8× encoded-representation downsampling make SepFormer faster and less memory-demanding than recent RNN-based separation systems.

    Source: §4.3 · p. 4, §4.4 · p. 4, §5 · p. 4

Abstract

from arXiv · show

Recurrent Neural Networks (RNNs) have long been the dominant architecture in sequence-to-sequence learning. RNNs, however, are inherently sequential models that do not allow parallelization of their computations. Transformers are emerging as a natural alternative to standard RNNs, replacing recurrent computations with a multi-head attention mechanism. In this paper, we propose the SepFormer, a novel RNN-free Transformer-based neural network for speech separation. The SepFormer learns short and long-term dependencies with a multi-scale approach that employs transformers. The proposed model achieves state-of-the-art (SOTA) performance on the standard WSJ0-2/3mix datasets. It reaches an SI-SNRi of 22.3 dB on WSJ0-2mix and an SI-SNRi of 19.5 dB on WSJ0-3mix. The SepFormer inherits the parallelization advantages of Transformers and achieves a competitive performance even when downsampling the encoded representation by a factor of 8. It is thus significantly faster and it is less memory-demanding than the latest speech separation systems with comparable performance.

1. INTRODUCTION

RNNs model long-term speech contexts but their sequential computation limits parallelization, motivating Transformer-based separation. SepFormer replaces RNNs with a multi-scale Transformer pipeline and reports state-of-the-art separation with faster, less memory-demanding processing.

  • RNNs learn long-term dependencies and manage speech contexts, but their sequential computation impairs effective parallelization.Source: §1 · p. 1
  • Transformer-based models remain relatively underexplored for monaural audio source separation, despite strong progress from deep learning and end-to-end systems.Source: §1 · p. 1
  • SepFormer replaces DPRNN's RNN components with multi-scale Transformers that learn short- and long-term dependencies within a dual-path framework.Source: §1 · p. 1
  • 22.3 dB SI-SNRi on WSJ0-2mix and 19.5 dB SI-SNRi on WSJ0-3mix establish SepFormer's reported state-of-the-art performance.Source: §1 · p. 1
  • SepFormer processes time steps in parallel and remains competitive with 8× downsampling, making it faster and less memory-demanding than recent RNN-based systems.Source: §1 · pp. 1–2

2. THE MODEL

SepFormer combines learned-domain masking with a dual-path Transformer masking network that models short- and long-term dependencies, then reconstructs separated sources from masked encoder representations.

2.2. Masking Network

  • SepFormer uses an encoder, Transformer-based masking network, and decoder within a learned-domain masking architecture.The encoder produces representations, the masking network estimates speaker masks, and the decoder reconstructs separated time-domain signals.Source: §2 · p. 2, §2.2 · p. 2, §2.4 · p. 2

2.1. Encoder

  • The encoder converts the time-domain mixture x into an STFT-like representation h using a single convolutional layer.The convolutional stride affects the model’s performance, speed, and memory use.Source: §2.1 · p. 2

2.2. Masking Network

  • The masking network normalizes h, projects it linearly, and divides it into overlapping chunks represented as h′.Chunks have size C and a 50% overlap between adjacent chunks.Source: §2.2 · p. 2

2.3. SepFormer Block

  • The SepFormer block applies IntraTransformer within each chunk and InterTransformer across chunks to model short- and long-term dependencies.The transformation is h′′ = f_inter(P(f_intra(h′))), and the block is repeated N times.Source: §2.2 · p. 2, §2.3 · p. 2

2.4. Decoder

  • The masking output is converted into speaker-specific masks, multiplied element-wise with encoder features, and decoded by transposed convolution.The decoder uses the encoder’s stride and kernel size, producing the separated source b_s^k.Source: §2.2 · p. 2, §2.4 · p. 2

2.3.1. Intra and Inter Transformers

  • Transformer layers add sinusoidal positional encoding, layer normalization, multi-head attention, feed-forward processing, and residual connections.Positional encoding supplies sequence-order information, while residual connections span Transformer layers and the overall Transformer architecture.Source: §2.3.1 · p. 2

3. EXPERIMENTAL SETUP

The experiments evaluate SepFormer on WSJ0-2mix and WSJ0-3mix using controlled mixtures and specified training configurations, including chunked Transformer settings and augmentation.

3.1. Dataset

  • WSJ0-2mix and WSJ0-3mix contain mixtures of two and three speakers created from WSJ0 utterances.Source levels are sampled uniformly from 0 dB to 5 dB, and the waveform sampling rate is 8 kHz.Source: §3.1 · p. 2
  • The datasets use 30 hours for training, 10 hours for validation, and 5 hours for testing, with different speakers across training and test sets.This setup separates speaker identities between training and evaluation.Source: §3.1 · p. 2

3.2. Model Configuration

  • The encoder uses 256 convolutional filters with kernel size 16 samples and stride 8 samples, while the decoder mirrors these settings.The decoder uses the same kernel size and stride factor as the encoder.Source: §3.2 · p. 3
  • The best masking networks use chunks of C = 250 with 50% overlap, 8 IntraT and 8 InterT layers, and N = 2 dual-path repetitions.Each Transformer layer uses 8 attention heads and 1024-dimensional positional feed-forward networks; the model has 26 million parameters.Source: §3.2 · p. 3

3.3. Data Augmentation

  • Dynamic mixing creates new mixtures from single-speaker sources and is extended with speed perturbation between 95% slow-down and 105% speed-up.The augmentation is applied on the fly before mixing the sources.Source: §3.2 · p. 3

3.4. Training

  • Training uses Adam with learning rate 15e−5, gradient clipping at L2 norm 5, batch size 1, and utterance-level permutation-invariant SI-SNR.The learning rate is annealed after validation performance fails to improve for the specified number of epochs.Source: §3.2 · p. 3

4. RESULTS

SepFormer achieves state-of-the-art separation performance on WSJ0-2mix and WSJ0-3mix while enabling parallel computation and favorable speed–memory trade-offs. Ablations show that transformer depth, positional encoding, and dynamic mixing affect performance.

  • 22.3 dB SI-SNRi and 22.4 dB SDRi are achieved on the WSJ0-2mix test set with dynamic mixing.Source: §4.1 · p. 3

Ablation Study

  • Eight IntraT and InterT layers replicated twice produce the best ablation results, while a single InterT layer still reaches 19.2 dB.Source: §4.2 · p. 3
  • Positional encoding and dynamic mixing improve performance, while eight and sixteen attention heads produce only a slight difference.Source: §4.2 · p. 4
  • 19.5 dB SI-SNRi and 19.7 dB SDRi are achieved on WSJ0-3mix, outperforming previously proposed systems.Source: §4.3 · p. 4

Speed and Memory Comparison

  • Parallelizing computations across time steps enables faster training and inference than RNN-based systems.Source: §4.3 · p. 4
  • SepFormer trains faster than DPRNN and DPTNeT under matched hardware and training settings.Source: §4.4 · p. 4
  • SepFormer is faster and less memory-demanding than DPTNet, DPRNN, and Wavesplit, partly because stride-factor-8 encoding processes less data.Source: §4.4 · p. 4

5. CONCLUSIONS

SepFormer is an RNN-free Transformer architecture for speech separation that learns short- and long-term dependencies through a multi-scale masking network. It achieves state-of-the-art separation on WSJ0-2mix and WSJ0-3mix while enabling parallel computation and reduced memory usage.

  • SepFormer uses a transformer-only masking network with a multi-scale approach to learn short- and long-term dependencies.Source: §5 · p. 4
  • Parallel computation across time steps enables significant speed-up during training and inference.Source: §5 · p. 4
  • Downsampling the encoded representation by a factor of 8 maintains competitive performance while reducing memory usage.Source: §5 · p. 4
  • The reported efficiency gains are especially notable relative to recent models such as DPRNN, DPTNet, and Wavesplit.Source: §5 · p. 4

Related papers

Loading 2010.13154v2…