Source-linked AI summary
Conditional Generative Adversarial Networks for Speech Enhancement and Noise-Robust Speaker Verification
Daniel Michelsanti, Zheng-Hua Tan
TL;DR
Degraded speech limits quality, intelligibility, and noise robustness in speech applications, motivating new enhancement methods. The paper adapts conditional Pix2Pix GANs to map noisy spectrograms to enhanced ones and evaluates PESQ, STOI, and ASV EER. Overall, Pix2Pix outperforms STSA-MMSE and is competitive with DNN-SE, with performance depending on the evaluation condition.
Problem
Speech enhancement is needed to improve degraded-speech quality and intelligibility and the noise robustness of automatic systems, while GAN use for speech enhancement had not been explored.
Method
The paper adapts conditional Pix2Pix GANs, using a U-Net generator and PatchGAN discriminator, to learn noisy-to-clean speech-spectrogram mappings.
Results
Pix2Pix is overall superior to STSA-MMSE and competitive with DNN-SE, with better PESQ and clean-model EER but worse STOI and multi-condition EER than DNN-SE.
Takeaways & Limitations
Pix2Pix provides a competitive GAN-based speech-enhancement front-end across perceptual quality, intelligibility, and speaker-verification evaluations.
Takeaways & Limitations
Future work calls for evaluation in more critical SNR situations and task-specific modifications, including smaller output windows and a perceptual loss.
Abstract
from arXiv · showhide
Improving speech system performance in noisy environments remains a challenging task, and speech enhancement (SE) is one of the effective techniques to solve the problem. Motivated by the promising results of generative adversarial networks (GANs) in a variety of image processing tasks, we explore the potential of conditional GANs (cGANs) for SE, and in particular, we make use of the image processing framework proposed by Isola et al. [1] to learn a mapping from the spectrogram of noisy speech to an enhanced counterpart. The SE cGAN consists of two networks, trained in an adversarial manner: a generator that tries to enhance the input noisy spectrogram, and a discriminator that tries to distinguish between enhanced spectrograms provided by the generator and clean ones from the database using the noisy spectrogram as a condition. We evaluate the performance of the cGAN method in terms of perceptual evaluation of speech quality (PESQ), short-time objective intelligibility (STOI), and equal error rate (EER) of speaker verification (an example application). Experimental results show that the cGAN method overall outperforms the classical short-time spectral amplitude minimum mean square error (STSA-MMSE) SE algorithm, and is comparable to a deep neural network-based SE approach (DNN-SE).
1. Introduction
Speech enhancement addresses degraded speech in applications including speaker verification, recognition, communications, and hearing assistance. This paper explores conditional GANs for mapping noisy to clean speech spectrograms, extending GAN use beyond prior speech applications.
- Motivation: Speech enhancement targets improved quality and intelligibility for humans and greater noise robustness for automatic speech systems.For automatic speaker verification, the goal includes reducing EER under adverse conditions.
- Motivation: Degraded speech remains challenging and important across speaker verification, speech recognition, mobile communications, and hearing assistive devices.
- Prior approaches: Earlier approaches included statistical methods such as Wiener filtering and STSA-MMSE, followed by DNN, DAE, and CNN methods.
- Prior approaches: GANs had succeeded in computer vision and image processing, but their adoption for speech-related tasks was rare and lacked objective speech-quality evaluation in the cited voice-conversion work.
- Contribution: The paper adapts Pix2Pix to learn noisy-to-clean speech-spectrogram mappings and a training loss for speech enhancement.
2. Pix2Pix framework for speech enhancement
The speech-enhancement system adapts conditional Pix2Pix GANs to map noisy spectrograms toward clean ones. Its adversarial framework combines conditioned discrimination with L1 reconstruction, using U-Net and PatchGAN components for spectrogram processing.
- Conditional GAN formulation: In a cGAN, generator and discriminator training is conditioned on extra information y while optimizing an adversarial min-max objective.
- Conditional GAN formulation: Pix2Pix omits random noise z because accurate noisy-to-clean mapping is prioritized over modeling the full output-distribution entropy.
- Loss and architecture: Pix2Pix combines adversarial loss with L1 distance to ground truth, preferring L1 because it encourages less blurring and tends to generalize better than perceptual losses.
- Loss and architecture: The generator is a U-Net, while the discriminator is a PatchGAN that classifies local patches and averages their scores into one output.
- Implementation: The implementation uses 256 × 256 × 1 generator inputs, 256 × 256 2-channel discriminator inputs, and 5×5 convolutional filters.
- Implementation: Speech is represented with a 512-point STFT using a 32 ms Hamming window and 16 ms hop, retaining 257 positive-frequency magnitude bins before resizing and normalization.
- Implementation: Training uses Adam-based SGD for 10 epochs with batch size 1, updates G twice per iteration, and scales the L1 loss by 100.
- Inference: At inference, G processes the noisy spectrogram magnitude and the signal is reconstructed by inverse STFT using the noisy input phase.
3. Experiments
The experiments evaluate speech quality, intelligibility, and speaker verification using multiple speech corpora, simulated noises, SNRs, enhancement front-ends, and ASV training conditions.
- Evaluation: PESQ, STOI, and ASV EER measure speech quality, intelligibility, and speaker-verification performance, respectively.
- Evaluation: ASV uses a GMM-UBM framework with 57-dimensional MFCC features and 512 GMM mixtures, suited to the study’s short utterances.
- Systems and tests: The study compares no enhancement, STSA-MMSE, noise-specific and noise-general DNN-SE, and noise-specific and noise-general Pix2Pix front-ends.
- Data: TIMIT supplies UBM training data, while RSR2015 supplies enhancement training, speaker-model training, and evaluation utterances across separated speaker and session sets.
- Data: Five noises are added at different SNRs, with distinct noise data used for training and testing; Pix2Pix and DNN front-ends train at 10 and 20 dB SNR.
- Systems and tests: The three tests assess PESQ/STOI, ASV with enhanced clean training, and multi-condition ASV training using one or all noise types.
4. Results and Discussion
Pix2Pix speech enhancement performs strongly across perceptual quality and speaker-verification evaluations, generally surpassing STSA-MMSE and remaining competitive with DNN-SE. Its advantages vary by metric, SNR, noise type, and speaker-model training condition.
- PESQ: NS-Pix2Pix and NG-Pix2Pix achieve the best average PESQ scores in most conditions, especially between 5 and 15 dB SNR.At 20 dB, enhancement can introduce distortion that outweighs noise-reduction benefits for some noises.
- STOI: Pix2Pix front-ends perform similarly to STSA-MMSE on STOI, while DNN-SE is superior in almost all tested conditions.Pix2Pix approaches DNN-SE at low SNRs for Cantine and Market noises.
- ASV: Pix2Pix generally outperforms baseline methods for ASV with clean speaker models, although DNN-SE is marginally better for Babble noise.The Babble exception is NG-DNN at 8.73% EER versus 8.76% for NS-Pix2Pix.
- ASV: Under multi-condition training, DNN-SE generally performs better than in the clean-speaker-model setting, while Pix2Pix remains generally better than STSA-MMSE.The NS-Pix2Pix front-end performs worse for white noise.
- Overall comparison: Overall, Pix2Pix is competitive with DNN-SE and superior to STSA-MMSE, with the comparison depending on PESQ, STOI, EER, and training condition.Pix2Pix has better PESQ and clean-model EER, but worse STOI and multi-condition EER than DNN-SE.
5. Conclusion
The study evaluates cGANs for speech enhancement using speech-quality, intelligibility, and speaker-verification measures. Results indicate global superiority to STSA-MMSE and comparability with DNN-SE, while broader evaluation in critical SNR conditions remains future work.
- cGAN speech enhancement was evaluated using estimated speech perceptual quality, speech intelligibility, and ASV equal error rate.
- ASV performance was reported in terms of EER for a multi-condition speaker model.
- cGANs were globally superior to the classical STSA-MMSE algorithm and comparable to the DNN-SE algorithm.
- Future work includes more extensive evaluation in more critical SNR situations and modifications tailored specifically to speech enhancement.