Source-linked AI summary
T-GSA: Transformer with Gaussian-weighted self-attention for speech enhancement
Jaeyoung Kim, Mostafa El-Khamy, Jungwon Lee
TL;DR
Speech enhancement differs from NLP because acoustic context is more local, limiting the benefit of standard Transformer attention. The paper proposes T-GSA, which Gaussian-weights attention by target-context distance while preserving correlation signs, and reports significantly better SDR and PESQ than prior recurrent and Transformer networks. The complex extension improves SDR but degrades PESQ, leaving consistent gains across both metrics as future work.
Problem
Transformer networks performed well in NLP but did not show improvements in acoustic signal processing, where closer acoustic components are more correlated.
Method
T-GSA attenuates self-attention weights according to target-context distance using a trainable Gaussian variance while preserving correlation signs.
Results
T-GSA significantly improved both SDR and PESQ over previous state-of-the-art recurrent and Transformer networks.
Takeaways & Limitations
Gaussian weighting outperformed attention biasing and prior networks, indicating that negative correlations remain important for denoising performance.
Takeaways & Limitations
The complex Transformer improved SDR by 0.1 to 0.2 dB but degraded PESQ, possibly because of overfitting or difficulty predicting the phase spectrum.
Abstract
from arXiv · showhide
Transformer neural networks (TNN) demonstrated state-of-art performance on many natural language processing (NLP) tasks, replacing recurrent neural networks (RNNs), such as LSTMs or GRUs. However, TNNs did not perform well in speech enhancement, whose contextual nature is different than NLP tasks, like machine translation. Self-attention is a core building block of the Transformer, which not only enables parallelization of sequence computation, but also provides the constant path length between symbols that is essential to learning long-range dependencies. In this paper, we propose a Transformer with Gaussian-weighted self-attention (T-GSA), whose attention weights are attenuated according to the distance between target and context symbols. The experimental results show that the proposed T-GSA has significantly improved speech-enhancement performance, compared to the Transformer and RNNs.
1. INTRODUCTION
Transformers offer parallel sequence processing and fixed path lengths, but their unweighted attention is poorly matched to acoustics, where nearby components are more correlated. T-GSA addresses this mismatch by attenuating attention with target-context distance, with learned Gaussian variance.
- Transformer motivation: Transformers process input sequences in parallel, reducing training and inference times compared with recurrent processing.This preserves the Transformer’s sequence-wide self-attention while enabling parallel computation.
- Transformer motivation: Fixed path lengths help Transformers learn long-range dependencies, unlike RNN paths that grow with target-context distance.Self-attention provides the fixed path between symbols before attention weights or correlations are computed.
- Acoustic mismatch: Transformer networks did not improve acoustic signal processing because acoustic signals are more correlated with closer components than distant ones.The paper identifies this contextual mismatch as a reason NLP-oriented Transformer behavior does not transfer directly to speech enhancement.
- Proposed solution: T-GSA attenuates attention weights according to target-context distance using a Gaussian variance learned during training.The proposed positional encoding penalizes attention to more distant symbols while adapting the attenuation through the learned variance.
- Reported outcome: The evaluation reports that T-GSA significantly improves over existing Transformer architectures and the prior best recurrent LSTM-based model.The comparison is stated at the paper level for speech-enhancement performance.
2. PROPOSED ARCHITECTURES
The proposed architectures adapt Transformer encoders to speech enhancement by localizing attention with Gaussian weighting and extending processing to complex spectra.
- Transformer speech-enhancement architecture: The denoising network uses only a Transformer encoder because input and output sequences have equal length.Alignment between input and output sequences is therefore unnecessary.
- Transformer speech-enhancement architecture: The network takes noisy-speech STFT magnitude spectra as input and predicts a time-frequency mask for clean-speech estimation.The estimated clean magnitude is combined with the noisy input phase and reconstructed using ISTFT.
- GSA: Gaussian-weighted self-attention: T-GSA scales the key-query score matrix element-wise with a Gaussian weighting matrix before computing self-attention.The Gaussian matrix attenuates attention to distant context frames more strongly than attention to nearby frames.
- GSA: Gaussian-weighted self-attention: The trainable Gaussian variance parameter lets acoustic training data learn context localization for speech enhancement.The diagonal scaling factors for target frames are fixed at 1.
- GSA: Gaussian-weighted self-attention: Applying absolute values before softmax makes attention depend on score magnitude, using positive and negative correlations equally.This differs from additive attention biasing, which can alter attention signs; GSA preserves correlation signs while changing their scale with distance.
- Extension to complex Transformer architecture: The complex Transformer separately processes real and imaginary inputs initially, then mixes the two paths in later attention blocks and estimates a complex mask.The mixed attention is intended to exploit cross-correlation between real and imaginary spectrum components.
3. EXPERIMENTAL RESULTS
Experiments evaluated Transformer variants on QUT-NOISE-TIMIT and VoiceBank-DEMAND against recurrent and generative baselines. Gaussian-weighted attention produced the strongest reported performance, while the complex variant improved SDR but reduced PESQ relative to the real Transformer.
- Experimental Settings: Two datasets supported evaluation: QUT-NOISE-TIMIT and VoiceBank-DEMAND, with distinct noise mixtures and test conditions.QUT-NOISE-TIMIT used six SNR ranges for evaluation, while VoiceBank-DEMAND used test noise types that did not overlap with training noise types.
- Main Result: T-GSA significantly outperformed CNN-LSTM and prior Transformer networks across QUT-NOISE-TIMIT SNR ranges on SDR and PESQ.The compared Transformer models included original attention, attention biasing, real Gaussian-weighted attention, and complex Gaussian-weighted attention.
- Main Result: 0.1 to 0.2 dB SDR improvement was obtained by the complex Transformer over the real Transformer across all SNR ranges.PESQ nevertheless degraded for the complex Transformer, possibly because of overfitting or difficulty predicting the phase spectrum.
- Comparison with Generative Models: T-GSA outperformed all listed generative models on every perceptual speech metric in the VoiceBank-DEMAND comparison.The reported improvement was attributed partly to joint SDR and PESQ optimization, which benefited both CNN-LSTM and T-GSA.
4. CONCLUSION
The paper proposes Gaussian-weighted self-attention for speech enhancement, attenuating attention according to target-context distance. Evaluation showed significant SDR and PESQ improvements over recurrent and Transformer networks.
- 4. CONCLUSION: T-GSA attenuates attention weights according to target-frame and attended-symbol distance while preserving correlation signs.The proposed architecture applies Gaussian weighting to self-attention for speech enhancement.
- 4. CONCLUSION: T-GSA significantly improved both SDR and PESQ over previous recurrent and Transformer networks.This conclusion summarizes the reported performance evaluation of the proposed self-attention scheme.