PaperlayerView exact revision on arXiv

Source-linked AI summary

MossFormer: Pushing the Performance Limit of Monaural Speech Separation using Gated Single-Head Transformer with Convolution-Augmented Joint Self-Attentions

Shengkui Zhao, Bin Ma

arXiv:2302.11824v1cs.SDcs.LGeess.AS

TL;DR

Dual-path Transformer models still handle long-range cross-chunk interactions indirectly and underuse convolutions for local patterns. MossFormer combines gated joint local and global attention with convolutions, achieving state-of-the-art results on WSJ0-2/3mix and WHAM!/WHAMR! and the SI-SDRi upper bound on WSJ0-3mix.

  • Problem

    Dual-path Transformer models have limited attention context, model cross-chunk dependencies indirectly, and do not fully exploit convolutions for local patterns.

    Source: §1 · p. 1

  • Method

    MossFormer combines gated single-head attention with joint local and global self-attention and convolution modules.

    Source: §1 · p. 2, §2.3 · p. 2

  • Results

    MossFormer reports state-of-the-art results on WSJ0-2/3mix and WHAM!/WHAMR! and reaches the SI-SDRi upper bound on WSJ0-3mix.

    Source: §1 · p. 2, §4 · p. 4

  • Takeaways & Limitations

    The component studies demonstrate the importance of the model’s components, alongside improved results on the named benchmarks.

    Source: §4 · p. 4

Abstract

from arXiv · show

Transformer based models have provided significant performance improvements in monaural speech separation. However, there is still a performance gap compared to a recent proposed upper bound. The major limitation of the current dual-path Transformer models is the inefficient modelling of long-range elemental interactions and local feature patterns. In this work, we achieve the upper bound by proposing a gated single-head transformer architecture with convolution-augmented joint self-attentions, named \textit{MossFormer} (\textit{Mo}naural \textit{s}peech \textit{s}eparation Trans\textit{Former}). To effectively solve the indirect elemental interactions across chunks in the dual-path architecture, MossFormer employs a joint local and global self-attention architecture that simultaneously performs a full-computation self-attention on local chunks and a linearised low-cost self-attention over the full sequence. The joint attention enables MossFormer model full-sequence elemental interaction directly. In addition, we employ a powerful attentive gating mechanism with simplified single-head self-attentions. Besides the attentive long-range modelling, we also augment MossFormer with convolutions for the position-wise local pattern modelling. As a consequence, MossFormer significantly outperforms the previous models and achieves the state-of-the-art results on WSJ0-2/3mix and WHAM!/WHAMR! benchmarks. Our model achieves the SI-SDRi upper bound of 21.2 dB on WSJ0-3mix and only 0.3 dB below the upper bound of 23.1 dB on WSJ0-2mix.

1. INTRODUCTION

MossFormer combines gated joint local and global attention with convolutions to address indirect long-range interactions and under-modeled local patterns in dual-path Transformers. It reports state-of-the-art results across WSJ0-2/3mix and WHAM!/WHAMR!, reaching the SI-SDRi upper bound on WSJ0-3mix.

  • Earlier dual-path models split long encoded sequences into chunks for intra- and inter-chunk processing, with DPRNN outperforming Conv-TasNet.Source: §1 · p. 1
  • The architecture pairs a convolutional encoder-decoder with a masking net built from MossFormer blocks and convolutions.Source: Figure 1 · p. 1
  • Dual-path Transformer attention is limited to short contexts by quadratic computation, leaving cross-chunk dependencies implicit and convolutions underused for local patterns.Source: §1 · p. 1
  • MossFormer outperforms previous models and reports state-of-the-art results on WSJ0-2/3mix and WHAM!/WHAMR!, reaching the SI-SDRi upper bound on WSJ0-3mix.Source: §1 · p. 2
  • Its gated single-head Transformer combines joint local and global self-attention with convolutions for direct long-range interaction and local-pattern modeling.The attentive gating mechanism allows simplified single-head self-attention.Source: §1 · p. 2

2. THE MOSSFORMER MODEL

MossFormer uses a time-domain encoder-decoder and a masking net built around repeated blocks that combine convolution modules, attentive gating, and joint local-global attention. Its design targets both long-range interactions and local feature patterns.

2.1. Encoder and Decoder

  • The encoder extracts features, speaker masks multiply the encoded sequence, and a transposed-convolution decoder reconstructs each source waveform.The encoder uses a 1D convolution and ReLU; the decoder uses the encoder’s kernel size and stride.Source: §2.1 · p. 2

2.2. Masking Net

  • The masking net normalizes the encoded sequence, adds positional encodings, and processes it through repeated MossFormer blocks before producing speaker-specific masks.The output sequence is expanded, passed through parallel pointwise convolutions and a GLU, then reshaped into masks for separate decoding.Source: §2.2 · p. 2

2.3. MossFormer Block

  • Each MossFormer block combines four convolution modules, scale and offset operations, joint local-global single-head attention, and three gating operations.Blocks pass residual outputs forward through the masking net.Source: §2.3 · p. 2, Figure 2 · p. 2, §2.2 · p. 2

2.3.1. Convolution Module

  • The convolution module projects normalized features through a linear layer and SiLU, then applies a 1D depthwise convolution to model fine-grained local patterns.Skip connections and dropout are used to help training and regularization.Source: §2.3.1 · p. 2, Figure 3 · p. 3

2.3.2. Attentive Gating Mechanism

  • The attentive gating mechanism forms U and V with convolution modules and combines them through gating, with the linear layer reducing feature width from 2N to N.The gates allow use of a simpler single-head self-attention.Source: §2.3 · p. 2, §2.3.2 · p. 3

2.3.3. Joint Local and Global Single-Head Self-Attention

  • Joint attention combines low-cost linearized global interactions over the full sequence with quadratic attention computed independently within non-overlapping chunks.The method uses squared ReLU instead of softmax for the local attention.Source: §2.3.3 · p. 3

3. EXPERIMENTS

Experiments evaluate MossFormer on clean, noisy, and reverberated speech-separation benchmarks, comparing model sizes and design choices. The reported results include new state-of-the-art performance and ablations of key components.

3.1. Dataset

  • The evaluation uses 8 kHz WSJ0-2mix/3mix and WHAM!/WHAMR! data, with training and validation utterances randomly segmented into 4-second samples.Dynamic mixing with speed perturbation is also considered as data augmentation.Source: §3.1 · p. 3

3.2. Training Setup

  • The three model sizes contain 11M, 25M, and 42M parameters; training uses SI-SDR loss and Adam for up to 200 epochs on one 16-GB V100 GPU.The batch size is 1, and the learning rate is reduced after 85 epochs when validation plateaus.Source: §3.2 · p. 3

3.3. Results

  • MossFormer(L) records 22.8 dB and 21.2 dB on WSJ0-2mix/3mix, matching the 3mix upper bound and setting new state-of-the-art results on both.The reported upper bounds are 23.1 dB and 21.2 dB, respectively.Source: §3.3 · p. 4, Table 2 · p. 4
  • On noisy and reverberated benchmarks, MossFormer(M) and (L) outperform previous models, with (L) setting new state-of-the-art results on WHAM! and WHAMR!.These tasks add noise and reverberation to WSJ0-2mix, requiring denoising and dereverberation in addition to separation.Source: §3.3 · p. 4, Table 2 · p. 4

3.4. Ablation Studies

  • Ablations show that convolution, triple gating, and joint attention contribute to performance; removing O′ drops the result from 20.9 dB to 20.4 dB.Neither local nor global attention alone performs well, while global-only attention exceeds local-only attention.Source: §3.4 · p. 4, Table 3 · p. 4
  • Among tested settings, Sigmoid gating works best, kernel sizes improve performance through 31 but worsen at 65, D=128 performs best, and chunk-size gains stop from 256 to 384.Larger chunk sizes tend to perform better until that increase yields no further improvement.Source: §3.4 · p. 4, Table 4 · p. 4, Table 5 · p. 4

4. CONCLUSIONS

MossFormer unifies modelling of local feature patterns and global long-range dependencies in an attentive gating model. It combines local quadratic and low-cost global attention directly, alongside convolution modules, and reports new state-of-the-art results across the named benchmarks.

4. CONCLUSIONS

  • MossFormer unifies local feature-pattern modelling and global long-range dependency modelling in an attentive gating model.It combines quadratic local attention with low-cost global attention to model long-range interactions directly, and uses convolution modules for local patterns.Source: §4 · p. 4
  • The studies demonstrate the importance of each component, and MossFormer achieves new state-of-the-art results on WSJ0-2/3mix and WHAM!/WHAMR!.Source: §4 · p. 4
  • Unlike dual-path models that model long-range interactions implicitly, MossFormer uses joint local and global attention to model them directly.Source: §4 · p. 4
Loading 2302.11824v1…