Source-linked AI summary
MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectra
Ye-Xin Lu, Yang Ai, Zhen-Hua Ling
TL;DR
Speech enhancement must recover clean speech from noisy recordings, while phase remains difficult to enhance despite its importance for perceptual quality. MP-SENet uses a codec architecture with parallel magnitude-mask and phase decoders to directly denoise both spectra. On VoiceBank+DEMAND, it achieves state-of-the-art performance, including a PESQ score of 3.50.
Problem
Phase is often omitted from time-frequency speech enhancement because its wrapping and nonstructural properties make direct enhancement difficult, although phase affects perceptual quality, especially at low SNR.
Method
MP-SENet encodes noisy magnitude and wrapped phase spectra, then uses parallel magnitude-mask and phase decoders to recover clean spectra before ISTFT reconstruction.
Results
MP-SENet achieves a PESQ score of 3.50 on VoiceBank+DEMAND and outperforms existing advanced speech enhancement methods.
Takeaways & Limitations
Direct parallel enhancement of magnitude and phase spectra achieves state-of-the-art performance on the VoiceBank+DEMAND dataset.
Abstract
from arXiv · showhide
This paper proposes MP-SENet, a novel Speech Enhancement Network which directly denoises Magnitude and Phase spectra in parallel. The proposed MP-SENet adopts a codec architecture in which the encoder and decoder are bridged by convolution-augmented transformers. The encoder aims to encode time-frequency representations from the input noisy magnitude and phase spectra. The decoder is composed of parallel magnitude mask decoder and phase decoder, directly recovering clean magnitude spectra and clean-wrapped phase spectra by incorporating learnable sigmoid activation and parallel phase estimation architecture, respectively. Multi-level losses defined on magnitude spectra, phase spectra, short-time complex spectra, and time-domain waveforms are used to train the MP-SENet model jointly. Experimental results show that our proposed MP-SENet achieves a PESQ of 3.50 on the public VoiceBank+DEMAND dataset and outperforms existing advanced speech enhancement methods.
1. Introduction
Speech enhancement methods operate in either the time or time-frequency domain, but phase is difficult to enhance directly despite its importance for perceptual quality, especially at low SNR. MP-SENet addresses this challenge by denoising magnitude and phase spectra in parallel.
- Existing speech enhancement methods: Speech enhancement methods are broadly divided into time-domain and time-frequency-domain approaches for recovering clean speech from noisy recordings.Time-frequency methods predict clean frame-level representations and reconstruct enhanced waveforms.
- Phase enhancement challenge: Phase is often omitted from time-frequency representations because its wrapping and nonstructural properties make direct enhancement difficult.Earlier methods commonly enhanced only magnitude spectra and reused noisy phase during reconstruction.
- Phase enhancement challenge: Phase information is essential to speech perceptual quality, particularly under low signal-to-noise conditions.This motivates methods that recover phase rather than relying solely on noisy phase spectra.
- Limits of prior approaches: Existing complex-spectrum and magnitude-refinement methods still exhibit compensation effects between magnitude and phase spectra.These effects can produce discontinuities and damage harmonic structures in enhanced spectrograms.
- Proposed direction: MP-SENet proposes parallel magnitude and phase spectra denoising within a time-frequency-domain monaural speech enhancement model.Its codec architecture uses two-stage convolution-augmented transformers between the encoder and decoder, with separate magnitude-mask and phase decoders.
2. Methodology
MP-SENet is a codec-style time-frequency speech-enhancement model that denoises magnitude and phase spectra in parallel. Its architecture combines convolutional and transformer-based processing with specialized magnitude and phase decoders and jointly optimized multi-level losses.
- Encoder: MP-SENet encodes noisy magnitude and wrapped phase spectra into compressed time-frequency representations using convolutional blocks, a dilated DenseNet, and four TS-Conformers.The encoder uses dilations of 1, 2, 4, and 8 to extend the receptive field along time while dense connections help avoid vanishing gradients.
- Magnitude mask decoder: The magnitude decoder predicts a compressed mask with learnable sigmoid activation and applies it to the noisy magnitude spectrum to recover enhanced magnitude.Power-law compression uses c = 0.3, while the learnable activation adapts its shape across frequency bands through trainable parameters.
- Phase decoder: The phase decoder directly predicts clean wrapped phase using parallel pseudo-real and pseudo-imaginary components followed by a two-argument arctangent.This parallel phase estimation architecture is designed for the wrapping and nonstructural characteristics of phase spectra.
- Training criteria: MP-SENet trains with time, magnitude, complex-spectrum, metric, and anti-wrapping phase losses combined in a weighted generator objective.The generator loss combines five terms, while discriminator training uses scaled PESQ targets and is optimized jointly with the generator.
- Training criteria: The training objective minimizes the weighted generator loss and discriminator loss simultaneously, with generator weights of 0.2, 0.9, 0.1, 0.05, and 0.3.These weights correspond respectively to time, magnitude, complex, metric, and phase losses.
3. Experiments
Experiments on VoiceBank+DEMAND compare MP-SENet with representative time-domain and time-frequency enhancement methods, then test its components through ablation studies. MP-SENet achieves strong objective performance, while phase decoding and explicit phase optimization materially support its results.
- Experimental setup: The experiments use VoiceBank+DEMAND audio with multiple noise types and SNR conditions, comparing MP-SENet against representative time-domain and time-frequency methods.The evaluation uses six objective speech-quality metrics.
- Comparison with advanced SE methods: 3.50 PESQ was achieved by MP-SENet, which outperformed other methods on most metrics.The model was evaluated against time-domain methods, TF-domain methods, and four TF-domain state-of-the-art approaches.
- Comparison with advanced SE methods: 0.51, 0.52, 0.40, 0.60, and 0.46 improvements over PHASEN were obtained on PESQ, CSIG, CBAK, COVL, and SSNR, respectively.MP-SENet performed best on PESQ and three MOS-based metrics among the four TF-domain state-of-the-art approaches, but trailed DB-AIAT and CMGAN on SSNR.
- Spectrogram comparison: MP-SENet alleviated low-frequency harmonic-structure damage observed in CMGAN spectrograms and achieved state-of-the-art objective performance with 2.05 M parameters.The authors connect this visual result to directly predicting phase rather than using complex-spectrum refinement.
- Ablation study: Removing magnitude compression or replacing LSigmoid degraded performance, while removing the phase decoder caused all metrics to greatly degrade.These ablations support the roles of precise magnitude prediction and phase prediction in MP-SENet.
- Ablation study: Explicit phase optimization contributed more strongly than implicit complex-spectrum optimization, while removing the metric discriminator still produced performance comparable to CMGAN.Both phase and complex losses contributed to overall performance, but explicit phase optimization was described as pivotal.
4. Conclusions
The paper concludes that MP-SENet parallelly denoises magnitude and phase spectra through a Conformer-embedded codec architecture. On VoiceBank+DEMAND, it achieves state-of-the-art performance, while extending parallel enhancement to other speech tasks remains future work.
- Contributions: MP-SENet parallelly denoises magnitude and phase spectra using an encoder with parallel magnitude-mask and phase decoders.The overall structure is a Conformer-embedded codec architecture.
- Contributions: The direct enhancement of phase spectra is identified as MP-SENet’s major breakthrough.The conclusion attributes the model’s design to direct magnitude and phase reconstruction.
- Conclusions: MP-SENet achieves state-of-the-art performance on the VoiceBank+DEMAND dataset compared with other advanced speech-enhancement methods.Ablation studies also verify the effectiveness of its components and optimization methods.
- Future work: Applying parallel magnitude and phase enhancement to speech dereverberation, speech separation, and speech superresolution is left for future work.These applications are explicitly identified as the paper’s future focus.