PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual-branch Attention-In-Attention Transformer for single-channel speech enhancement

Guochen Yu, Andong Li, Chengshi Zheng, Yinuo Guo, Yutian Wang, Hui Wang

arXiv:2110.06467v5cs.SDcs.AIeess.AS

TL;DR

Monaural speech enhancement seeks to recover clean speech from noisy single-channel recordings, while conventional magnitude-focused estimation may retain noisy phase. DB-AIAT pairs coarse magnitude masking with complex spectral refinement and attention-in-attention modeling; on Voice Bank + DEMAND it is reported to outperform state-of-the-art baselines with a relatively light model size.

  • Problem

    Monaural speech enhancement aims to recover clean speech from noise-corrupted mixtures using one recording channel, while common magnitude or mask estimation leaves noisy phase involved in reconstruction.

    Source: §1 · p. 1

  • Method

    DB-AIAT uses parallel magnitude masking and complex refining branches with attention-in-attention modules to model temporal-frequency dependencies and aggregate hierarchical context.

    Source: §1 · p. 1, §2.1 · p. 2, §2.3 · p. 2

  • Results

    On Voice Bank + DEMAND, DB-AIAT consistently outperforms state-of-the-art baselines with a relatively light model size.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The two branches collaboratively recover the clean complex spectrum by estimating overall magnitude and refining missing spectral details and phase information.

    Source: §2.1 · p. 2, §5 · p. 4

Abstract

from arXiv · show

Curriculum learning begins to thrive in the speech enhancement area, which decouples the original spectrum estimation task into multiple easier sub-tasks to achieve better performance. Motivated by that, we propose a dual-branch attention-in-attention transformer dubbed DB-AIAT to handle both coarse- and fine-grained regions of the spectrum in parallel. From a complementary perspective, a magnitude masking branch is proposed to coarsely estimate the overall magnitude spectrum, and simultaneously a complex refining branch is elaborately designed to compensate for the missing spectral details and implicitly derive phase information. Within each branch, we propose a novel attention-in-attention transformer-based module to replace the conventional RNNs and temporal convolutional networks for temporal sequence modeling. Specifically, the proposed attention-in-attention transformer consists of adaptive temporal-frequency attention transformer blocks and an adaptive hierarchical attention module, aiming to capture long-term temporal-frequency dependencies and further aggregate global hierarchical contextual information. Experimental results on Voice Bank + DEMAND demonstrate that DB-AIAT yields state-of-the-art performance (e.g., 3.31 PESQ, 95.6% STOI and 10.79dB SSNR) over previous advanced systems with a relatively small model size (2.81M).

1. INTRODUCTION

The introduction frames monaural speech enhancement as recovering clean speech from a noisy single-channel recording, while noting the perceptual importance of phase. DB-AIAT combines magnitude masking, complex refinement, and attention-in-attention temporal-frequency modeling, and is reported to outperform state-of-the-art baselines with a relatively light model size.

  • Monaural enhancement seeks clean speech from a noise-corrupted single channel; common mask or magnitude prediction retains noisy phase, despite its perceptual importance.Source: §1 · p. 1
  • Prior decoupling-style methods estimate magnitude first and then refine the spectrum, separating magnitude and phase optimization [13] [14].Source: §1 · p. 1
  • DB-AIAT consistently outperforms state-of-the-art baselines on Voice Bank + DEMAND with a relatively light model size.Source: §1 · p. 1
  • Its magnitude masking branch suppresses noise, while its complex refining branch compensates for lost spectral details and phase mismatch in parallel.The branches collaborate on overall spectrum recovery.Source: §1 · p. 1
  • AIAT models long-range dependencies across time and frequency and aggregates global hierarchical context, addressing limits attributed to RNNs and TCNs.Source: §1 · p. 1

2. METHODLOGY

DB-AIAT combines magnitude masking and complex-spectrum refinement to recover clean speech spectra in parallel. Its shared attention-in-attention module models temporal-frequency dependencies and aggregates hierarchical context.

2.1. Dual-branch magnitude masking and complex refining

  • The magnitude masking branch estimates a coarse target magnitude, while the complex refining branch models missing spectral details and residual noise before their outputs are combined.The refining branch predicts residual details rather than the whole complex spectrum, and the branches exchange information through alternate interconnections.Source: §2.1 · p. 2

2.2. Densely convolutional encoder

  • Each branch’s dense encoder uses convolutional layers and a four-layer dilated DenseNet with dilation rates {1, 2, 4, 8}.The second 2-D convolution halves the frequency dimension.Source: §2.2 · p. 2

2.3. Attention-in-attention transformer

  • Four adaptive temporal-frequency attention transformer blocks model long-range dependencies, while adaptive hierarchical attention weights and aggregates their intermediate features into global context.The hierarchical context is combined with the final block output using a learnable scalar initialized to 0.Source: §2.3 · pp. 2–3

2.4. Mask/Complex decoder

  • The magnitude decoder upsamples compressed features and estimates a gain mask, while parallel real and imaginary decoders reconstruct both complex-spectrum components.The magnitude mask is applied element-wise to the input magnitude.Source: §2.4 · p. 3

2.5. Loss function

  • The two-branch loss combines magnitude and real-imaginary component losses, with the weighting coefficient empirically set to 0.5.The RI loss compares estimated and clean-speech RI components.Source: §2.5 · p. 3

3. EXPERIMENTS

The experiments use Voice Bank + DEMAND data with unseen speakers and noise conditions for testing. Audio is resampled, framed, and transformed for model training.

3.1. Datasets

  • Training uses 11,572 noisy-clean pairs from 28 speakers, while testing uses 824 pairs from two unseen speakers and five unseen DEMAND noise types.Training mixtures use 10 noise types at 0, 5, 10, and 15 dB SNR; test mixtures use 2.5, 7.5, 12.5, and 17.5 dB SNR.Source: §3.1 · p. 3

3.2. Implementation setup

  • Audio is resampled to 16 kHz, chunked into 3-second segments, and represented with a 320-point STFT using 20 ms Hanning windows and 50% overlap.The resulting spectral features are 161-dimensional; magnitude is power-compressed with coefficient 0.5 while phase remains unaltered.Source: §3.2 · p. 3
  • Models are trained for 80 epochs with Adam at a 5e-4 learning rate and utterance-level batches of four.Source: §3.2 · p. 3

4. RESULTS AND ANALYSIS

DB-AIAT outperforms advanced time- and time-frequency-domain baselines on most reported metrics while using 2.81 M parameters. Ablations report gains from the parallel branches and the attention-in-attention design.

4. RESULTS AND ANALYSIS

  • The evaluation reports PESQ, STOI, SSNR, CSIG, CBAK, and COVL, with higher values indicating better performance.Source: §4 · p. 3

4.1. Comparison with previous advanced baselines

  • DB-AIAT improves over DEMUCS by average 0.24 PESQ, 0.6% STOI, 0.30 CSIG, 0.35 CBAK, and 0.33 COVL, with 2.81 M parameters.Table 1 compares time- and time-frequency-domain methods; the reported comparison also finds DB-AIAT exceeds existing state-of-the-art methods on most metrics.Source: §4.1 · pp. 3–4, Table 1 · p. 4

4.2. The effects of Dual-branch strategy and AIAT structure

  • DB-AIAT yields significant improvements on all metrics over the single-branch methods, supporting the complementary parallel-branch strategy.Source: Table 2 · p. 4, §4.2 · p. 4
  • DB-ATFAT exceeds DB-ATAT by average 0.36 PESQ, 0.8% STOI, 0.33 CSIG, 0.39 CBAK, and 0.39 COVL when combining temporal and frequency dependencies.DB-ATAT and DB-AFAT capture long-term dependencies along only the time and frequency dimensions, respectively.Source: Table 2 · p. 4, §4.2 · p. 4
  • Adding the adaptive hierarchical attention module to DB-ATFAT yields relatively better performance, indicating further speech-quality improvement with the full AIAT structure.Source: Table 2 · p. 4, §4.2 · p. 4

5. CONCLUSIONS

The conclusion presents DB-AIAT as a dual-branch method for collaborative clean complex-spectrum recovery. Its branches combine magnitude estimation and complex-detail refinement, with AIAT modeling temporal-frequency dependencies and hierarchical context.

  • The magnitude masking branch coarsely estimates the clean-speech magnitude spectrum, while the complex refining branch derives residual spectral details in parallel.Source: §5 · p. 4
  • Interconnection between the branches lets the complex refining branch compensate for missing complex spectral details and restore phase information.Source: §5 · p. 4
  • Each branch uses an attention-in-attention transformer between densely connected encoder-decoder architecture to model temporal sequences, strengthen long-range temporal-frequency dependencies, and aggregate global hierarchical context.Experiments on a public dataset report state-of-the-art performance over previous competitive systems.Source: §5 · p. 4
Loading 2110.06467v5…