Source-linked AI summary
Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition
Chris Donahue, Bo Li, Rohit Prabhavalkar
TL;DR
The paper asks whether GAN-based speech enhancement improves ASR robustness under additive and reverberant noise. It compares time-domain and spectral GAN enhancement, finding that FSEGAN helps clean-trained ASR but remains below MTR alone, while hybrid retraining yields a 7% relative improvement over stereo MTR.
Problem
The effectiveness of GAN-based speech enhancement for ASR, particularly with reverberant as well as additive noise, had not been established.
Method
The paper evaluates time-domain SEGAN and proposes FSEGAN, which maps noisy log-Mel filterbank spectra to clean spectra for ASR enhancement.
Results
Hybrid retraining with noisy and FSEGAN-enhanced features improves performance by 7% relative to stereo MTR training, while FSEGAN alone remains below MTR.
Takeaways & Limitations
For ASR, simpler regression enhancement may be preferable to GAN-based enhancement, although retrained hybrid inputs can improve existing MTR systems.
Takeaways & Limitations
The SEGAN generator learned to ignore latent noise vectors, so the evaluated generator was made deterministic.
Abstract
from arXiv · showhide
We investigate the effectiveness of generative adversarial networks (GANs) for speech enhancement, in the context of improving noise robustness of automatic speech recognition (ASR) systems. Prior work demonstrates that GANs can effectively suppress additive noise in raw waveform speech signals, improving perceptual quality metrics; however this technique was not justified in the context of ASR. In this work, we conduct a detailed study to measure the effectiveness of GANs in enhancing speech contaminated by both additive and reverberant noise. Motivated by recent advances in image processing, we propose operating GANs on log-Mel filterbank spectra instead of waveforms, which requires less computation and is more robust to reverberant noise. While GAN enhancement improves the performance of a clean-trained ASR system on noisy speech, it falls short of the performance achieved by conventional multi-style training (MTR). By appending the GAN-enhanced features to the noisy inputs and retraining, we achieve a 7% WER improvement relative to the MTR system.
1. INTRODUCTION
The paper evaluates GAN-based speech enhancement for ASR under additive and reverberant noise, finding that spectral enhancement is more promising than time-domain SEGAN but remains below conventional MTR unless retrained with both feature types.
- Motivation: GAN-based enhancement is studied for improving ASR robustness to additive and reverberant noise.The study uses an existing clean-trained ASR model to isolate enhancement effects.
- Time-domain baseline: SEGAN does not improve ASR performance under the evaluated noise conditions.
- Spectral enhancement: FSEGAN applies GAN enhancement to log-Mel filterbank spectra and improves ASR performance dramatically.The approach passes enhanced spectral features directly to the ASR model.
- Spectral enhancement: GAN-enhanced spectra appear more realistic than L1-enhanced spectra, but ASR performance is comparable between the two objectives.The visual comparison is described as anecdotal and suggests that fine-grained generated details may not benefit ASR.
- Comparison with MTR: MTR-trained ASR is robust to noise, but GAN enhancement degrades performance even with retraining when used alone.Retraining with both noisy and enhanced features improves the MTR model.
2. GENERATIVE ADVERSARIAL NETWORKS
GANs learn realistic samples through adversarial training, while conditional GANs use an input condition to generate related outputs. Image-to-image translation adapts this framework to matched input-output pairs and can combine adversarial and reconstruction losses.
- GANs: GANs contain a generator that produces samples and a discriminator that distinguishes real samples from generated ones.The two models are trained against each other in an adversarial framework.
- Conditional GANs: Conditional GANs provide an input condition x to the generator and model an implicit posterior pG(ŷ | x).The generator minimizes the objective while the discriminator maximizes it.
- Image translation: Pix2pix applies conditional GANs to image-to-image translation using matched pairs from two domains.Its generator outputs an image with the same resolution as the input, while the discriminator evaluates paired images.
- Hybrid objectives: A hybrid objective can combine the GAN objective with an L1 reconstruction penalty.
3. METHOD
The method adapts pix2pix-style GAN enhancement from time-domain waveforms to time-frequency spectra. It uses deterministic generators to map noisy speech representations toward clean speech while adversarially evaluating paired inputs and outputs.
- 3.1. SEGAN: SEGAN enhances speech in the time domain by mapping two-second noisy waveform windows to clean speech.The generator is applied repeatedly to one-second windows during inference.
- 3.1. SEGAN: The SEGAN generator uses convolutional encoding and decoding with skip connections, while omitting batch normalization and latent codes.Its implementation uses one-dimensional filters and a least squares GAN objective.
- 3.2. FSEGAN: In the FSEGAN strategy, G maps stereo noisy spectra x to enhanced G(x), while D classifies real and enhanced pairs.
- 3.2. FSEGAN: FSEGAN maps time-windowed noisy speech spectra to clean speech spectra in the frequency domain.The approach reconstructs magnitude information after phase is discarded in common ASR preprocessing.
- 3.2. FSEGAN: The FSEGAN generator uses seven encoder and seven decoder layers with skip connections and produces a single-channel output.The generator is deterministic because latent codes and batch normalization are excluded.
4. EXPERIMENTS
The experiments use WSJ speech, simulated additive and reverberant noise, and monaural LAS models to evaluate GAN enhancement for ASR. The preprocessing converts speech into normalized log-Mel spectral features.
- Data and noise: Experiments use the 16 kHz WSJ SI-284 training set, with dev93 for validation and eval92 for evaluation.The training set contains 81 hours, 284 speakers, and 37k utterances.
- Data and noise: Additive-noise mixtures span 0 dB to 30 dB SNR during training, averaging 11 dB, with slightly offset test SNRs.The additive sources are musical and ambient signals collected from YouTube and daily-life recordings.
- Data and noise: Reverberation is generated by a room simulator that randomizes source, microphone, room geometry, and T60 settings.Training and testing use distinct room-configuration sets, randomized during training and fixed during testing.
- ASR models: The study compares clean-trained ASR-Clean and multi-style-trained ASR-MTR monaural LAS models.Both models use the same LAS architecture, while ASR-MTR is trained with one noisy-speech channel.
- Feature processing: Time-domain speech is transformed with an STFT, magnitude filtering through 128 Mel filters, logarithmic scaling, and frequency-bin normalization.The STFT uses a 32 ms window and 10 ms hop; phase is discarded.
- ASR models: The LAS encoder combines convolutional layers with bidirectional recurrent layers, while the decoder uses attention and character-level softmax outputs.The encoder includes a bidirectional convolutional LSTM followed by three bidirectional LSTM layers.
- Evaluation: GAN experiments use multi-style matched pairs of noisy and clean speech, and evaluation uses no overlap with early stopping based on ASR-Clean validation WER.The experiment results are reported in Table 1.
5. RESULTS
Frequency-domain FSEGAN improves clean-trained ASR more than the time-domain SEGAN approach, but does not surpass MTR without retraining. Retraining with both noisy and enhanced features produces the strongest reported result.
- GAN enhancement: 12% relative degradation occurs when SEGAN enhances the MTR test set for ASR-Clean.With additive noise alone, SEGAN instead improves ASR-Clean performance by 21% relative, indicating difficulty suppressing reverberation.
- GAN enhancement: 54% relative improvement is achieved by FSEGAN for ASR-Clean, compared with 33% versus 20% WER for MTR training.FSEGAN still does not exceed the performance achieved with MTR training.
- GAN enhancement: FSEGAN enhancement appears to reduce additive noise and reverberant smearing, although it degrades ASR-MTR performance.The authors hypothesize that enhancement introduces previously unseen distortions.
- Retraining: Retraining ASR-MTR with FSEGAN-enhanced features improves performance by 17% relative to naively feeding enhanced features, but remains below MTR training.The retraining experiment is summarized in Table 2.
- Retraining: Hybrid retraining with original noisy and enhanced features exceeds stereo MTR training alone by 7% relative.The hybrid representation stacks noisy and enhanced features along the channel axis.
- Retraining: Training the same enhancer with an L1 objective achieves better ASR performance than the adversarial approach.This result suggests limited usefulness of GANs for ASR enhancement in this setting.
6. CONCLUSIONS
The paper introduces FSEGAN for frequency-domain speech enhancement and reports ASR gains over a prior time-domain approach. Its experiments also indicate that simpler regression methods may be preferable for ASR.
- Conclusions: FSEGAN performs GAN-based speech enhancement in the frequency domain and improves ASR performance over a prior time-domain approach.The method uses spectral feature mapping.
- Conclusions: With retraining, FSEGAN can improve the performance of existing MTR-trained ASR systems.The conclusion confines this finding to systems that are retrained with the enhancement setup.
- Conclusions: The experiments indicate that simpler regression approaches may be preferable to GAN-based enhancement for ASR.FSEGAN produces plausible spectra and may be more useful for telephonic applications with an invertible feature representation.