PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Environmentally Robust Speech Recognition: An Overview of Recent Developments

Zixing Zhang, Jürgen Geiger, Jouni Pohjalainen, Amr El-Desoky Mousa, Wenyu Jin, Björn Schuller

arXiv:1705.10874v3cs.SDcs.CLcs.LG

TL;DR

Non-stationary environmental noise and reverberation remain important challenges for automatic speech recognition. The paper surveys supervised deep-learning approaches across channels and ASR processing stages, concluding that robust systems are shifting toward raw-speech inputs, joint training, and multi-channel methods.

  • Problem

    Non-stationary environmental noise and reverberation degrade speech recognition, while existing surveys do not focus on deep-learning approaches.

  • Method

    The paper systematically surveys representative deep-learning methods for robust ASR across single- and multi-channel front-end, back-end, and joint architectures.

  • Results

    The survey identifies shifts toward extracting features from noisy raw speech, joint or end-to-end training, and multi-channel approaches.

  • Takeaways & Limitations

    The overview provides a taxonomy and comparison of promising deep-learning techniques for improving environmental robustness in ASR.

Abstract

from arXiv · show

Eliminating the negative effect of non-stationary environmental noise is a long-standing research topic for automatic speech recognition that stills remains an important challenge. Data-driven supervised approaches, including ones based on deep neural networks, have recently emerged as potential alternatives to traditional unsupervised approaches and with sufficient training, can alleviate the shortcomings of the unsupervised methods in various real-life acoustic environments. In this light, we review recently developed, representative deep learning approaches for tackling non-stationary additive and convolutional degradation of speech with the aim of providing guidelines for those involved in the development of environmentally robust speech recognition systems. We separately discuss single- and multi-channel techniques developed for the front-end and back-end of speech recognition systems, as well as joint front-end and back-end training frameworks.

I. INTRODUCTION

Environmental noise and reverberation continue to degrade everyday speech recognition, especially when acoustic conditions vary over time. This survey organizes deep-learning approaches for robust ASR by channel count and processing stage.

  • Ambient noise and reverberation remain central causes of speech-recognition degradation in everyday situations.
  • Non-stationary noise from transient events, competing speakers, and music is difficult to handle with conventional approaches.Short-term stationary additive noise can be addressed with standard unsupervised noise-reduction techniques when target-signal absence is reliably detected.
  • Deep learning uses supervised data-driven learning to estimate cleaner signals or features, or to recognize noisy speech directly.Deep neural networks are central to these recent developments.
  • The survey classifies approaches by single- versus multi-channel processing and by front-end, back-end, or joint ASR processing.It highlights advantages, disadvantages, interrelations, and differences among prominent techniques.
  • A. Problem Description: Clean and noisy speech have highly non-linear relationships across temporal, spectral, power-spectral, Mel-spectral, log-Mel, and cepstral domains.Time-varying room impulse responses and additive noises such as abrupt sounds, side talk, music, and driving noise further complicate cancellation.

B. Deep Learning vs Traditional Approaches

Traditional robust-ASR methods rely on signal statistics or generative acoustic models and struggle with complex, changing environments. Deep learning instead learns nonlinear relationships and supports enhancement, adaptation, and integrated recognition pipelines.

  • Robust ASR separates front-end enhancement from back-end model updating, with enhancement producing cleaner signals or features for recognition.Back-end updating adapts acoustic models to new data.
  • Unsupervised methods often fail to track non-stationary noise, while NMF struggles to represent speech and noise in complex acoustic environments.
  • Deep neural networks learn complicated input-output relationships through stacked nonlinear layers, making them suited to the complex nonlinear noise problem.
  • Convolutional and recurrent layers provide representations of local time-frequency patterns and access to previously processed inputs, respectively.Stacked neural layers support front-end noise and reverberation reduction.
  • Traditional back-end adaptation methods were designed for generative models, assume matched adapted and observed data, and fail to model large-scale complex environments.The shift toward discriminative DNN acoustic models motivates new noise-adaptation techniques.

C. Standard Corpora and Evaluation Metrics

Evaluation corpora span increasingly realistic combinations of vocabulary size, recording conditions, noise types, and channel counts. ASR performance is primarily measured with word error or word accuracy, while front-end methods use additional enhancement metrics.

  • Aurora databases range from artificially simulated digit-recognition data to Aurora-4, which uses WSJ0 for large-vocabulary continuous speech recognition.
  • CHiME databases incorporate additive and convolutional noise, with earlier versions covering digit recognition and LVCSR and later versions addressing broader settings.
  • AMI and Voice Search provide hundreds of spontaneous-speech recordings in real-life scenarios.
  • Standard corpora progress from small to large vocabularies, artificial to realistic recordings, additive to convolutional noise, and single to multiple channels.
  • Word Error Rate and Word Accuracy Rate are the de facto ASR metrics, while front-end evaluation also uses segSNR, SDR, distance measures, and perceptual metrics.

III. FRONT-END TECHNIQUES

Deep-learning front-end methods enhance signals or features from noisy speech, using representations ranging from raw audio to MFCCs. They are commonly trained under mapping- or masking-based supervision.

  • Front-end techniques include speech enhancement, source separation, and feature enhancement, with feature enhancement targeting task-specific representations such as MFCCs.
  • Speech representations can be formed through operations including STFT and squared magnitude, and enhancement can operate from temporal signals through cepstral MFCCs.
  • Deep-learning front-end models are normally supervised, taking a noisy-speech representation y as input and a clean-speech representation x as target.
  • Mapping-based methods predict representations extracted directly from clean speech, whereas masking-based methods predict masks calculated between clean and noisy speech.

A. Mapping-based Deep Enhancement Methods

Mapping-based enhancement methods learn nonlinear transformations from corrupted speech or features to clean speech representations, commonly using reconstruction objectives. Studies report that stacked autoencoder variants improve enhancement and ASR performance over traditional baselines under additive and convolutional noise.

  • Mapping-based methods learn a nonlinear function from noisy speech to desired clean speech.
  • Training reconstructs clean target features from corresponding corrupted input features by minimizing mean squared error.
  • After enhancement, estimated clean features are converted back to time-domain signals using phase information from the original noisy speech.
  • Based on Stacked AutoEncoder or Deep Bolzmann Machine: Stacked autoencoders map noisy speech to clean speech in the Mel-spectral domain using layered encoding and decoding stages.The encoder and decoder can use tied weight matrices, with successive autoencoders trained on transformed representations.
  • Based on Stacked AutoEncoder or Deep Bolzmann Machine: SAE-based enhancement notably outperforms MMSE for speech distorted by factory and car noises.
  • Based on Stacked AutoEncoder or Deep Bolzmann Machine: Weighted reconstruction losses assign frequency-band weights because identical distortions can affect speech quality differently across bands.
  • Based on Stacked AutoEncoder or Deep Bolzmann Machine: SDAE-based mapping remarkably outperforms spectral subtraction for ASR with Mel filterbank features corrupted by additive or convolutional noise.The networks are pre-trained with multi-condition data and fine-tuned by mapping noisy speech to clean speech.

2) Based on LSTM-RNN:

LSTM-RNN-based methods address the limited sequence-context modeling of earlier denoising networks and target highly non-stationary additive noise and reverberation. Reported evaluations show improved noisy-speech recognition compared with systems without LSTM-RNNs, while related CNN, temporal, and adversarial approaches expand enhancement capabilities.

  • Based on LSTM-RNN: Earlier SAE, DBM, and SDAE denoising networks have limited sequence-context modeling, whereas RNNs can capture context over long sequences.Expanding sequential frames into a long vector is described as a naive context-dependent alternative.
  • Based on LSTM-RNN: LSTM-RNNs were applied to highly non-stationary additive noise and later extended to reverberation.
  • Based on LSTM-RNN: Noisy-speech recognition systems with LSTM-RNNs perform much better than systems without LSTM-RNNs during decoding.
  • Based on CNN: CNN encoder-decoder networks treat spectrograms as images and use convolutional layers to discover representations for speech enhancement.
  • Based on CNN: Spectral or cepstral representations discard potentially valuable phase information, so enhancement may reconstruct speech using a distorted noisy phase spectrum.
  • Based on CNN: WaveNet uses causal dilated convolutions with exponentially increasing dilation to expand receptive fields and reduce computational complexity for temporal-domain enhancement.
  • Based on CNN: Adversarial training uses a generative network and discriminative network in a minimax game to map noisy speech toward clean speech.
  • Based on CNN: Adversarial training was reported to outperform traditional approaches such as Wiener filtering.

4) Brief Discussion:

Mask-based enhancement has progressed from binary and magnitude-only masks toward phase-aware and adversarial strategies. The reviewed evidence highlights stronger performance from ratio, phase-sensitive, and reduced-domain approaches, while noting that spectrogram structure remains underused.

  • Training trends: Adversarial training replaces a single-network strategy with generator–discriminator enhancement, reflecting a shift toward image-generation formulations.The discriminator is used to improve the generative quality of the enhancement network.
  • Open directions: Spectrogram-oriented methods often overlook harmonic frequency correlations, despite their distinction from the local correlations typical of visual images.The review identifies this mismatch as requiring further research.
  • Masking approaches: IRM-based masking generally outperforms IBM-based masking on objective intelligibility, quality, and ASR metrics.A neural network was also proposed to combine IBM- and IRM-enhanced speech to exploit their complementary strengths.
  • Representation trends: Mel- and log-Mel-domain masking extends enhancement beyond the T-F domain, with Mel-domain methods achieving better SDR than T-F-domain methods.The approaches were applied to reduced feature spaces frequently used in ASR.
  • Phase-aware masks: Phase-sensitive masking addresses distorted phase information, and experiments on CHiME-2 found it outperformed phase-nonsensitive approaches.PSM incorporates the phase difference between clean and noisy speech.
  • Phase-aware masks: Complex IRM extends IRM into the complex domain, whereas PSM corresponds to its real component; both phase-based masks suppressed reverberated noise more effectively than normal IRMs.PSM does not completely restore phase, motivating the development of cIRM.

2) Objective Functions and Training Strategies:

Mask-based systems use objective functions that supervise either mask estimation or reconstructed clean spectra, with phase-sensitive and multi-task variants extending the training targets. Reported experiments favor signal approximation and combined objectives, while joint source-and-mask training can outperform isolated training.

  • Objective functions: Mask Approximation minimizes mean squared error between an estimated T-F mask and its target mask.At test time, the estimated mask is applied sequentially to the noisy spectrum, followed by inverse STFT reconstruction.
  • Objective functions: Signal Approximation minimizes mean squared error between the estimated clean spectrum and the target clean spectrum.The estimated clean spectrum is formed by elementwise multiplication of the noisy spectrum and estimated mask.
  • Objective functions: Signal Approximation empirically outperformed Mask Approximation for source separation, while combining both objectives further improved enhancement in magnitude and Mel-spectral domains.The combined objective was reported to improve speech enhancement performance across both domains.
  • Phase-sensitive training: Phase-sensitive Signal Approximation keeps mask prediction but uses complex-domain signals so estimates shrink when noise is high.The network does not directly predict phase.
  • Training strategies: Multi-task learning jointly estimates speech, noise, and the mask, and experiments found it superior to isolated training.The framework assumes that learning noise and speech distortion relationships can help estimate clean speech.
  • Noise types: Masking-based approaches initially designed for additive noise have also been reported to eliminate convolutional noise.This finding spans recent masking-based studies.

IV. BACK-END TECHNIQUES

Back-end techniques leave noisy observations unchanged and use neural networks to model relationships between noisy speech and phonetic targets. The section covers NN/HMM hybrids, multi-condition training, adaptation, noise-aware training, and multi-task learning.

  • Back-end methods preserve noisy observations while neural networks learn mappings from them to phonetic targets.
  • NN/HMM systems combine neural-network discriminative classification with HMM context modeling, including tandem, double-stream, and hybrid structures.LSTM layers increasingly replace fully connected DNN layers in these structures.
  • Multi-condition training reduces training–test acoustic mismatch by exposing models to varied noisy conditions, but requires large datasets covering those conditions.
  • Model adaptation modifies a pretrained acoustic model to compensate for acoustic mismatch, although adapting all weights with limited data can overfit and create noise-dependent parameters.
  • Noise-aware training supplies an estimated noise representation alongside the original observation so the DNN receives an additional noise cue.
  • Dynamic noise-aware training estimates time-varying noise with masking-based methods, improving efficiency for non-stationary noise but requiring an extra DNN.
  • Multi-task acoustic models jointly optimize senone classification and clean-feature reconstruction, assuming clean-speech-oriented representations are easier to classify.

V. JOINT FRONT- AND BACK-END TRAINING TECHNIQUES

Joint front-end and back-end frameworks address the mismatch between enhancement objectives and recognition performance by training processing stages together. The section describes cascaded, parallel, end-to-end, and multi-channel front-end methods, including beamforming and neural post-filtering.

  • Separately optimized enhancement and recognition systems may target metrics such as segSNR, SDR, and PESQ that do not directly correlate with recognition performance.
  • A simple joint strategy retrains the pretrained acoustic model on speech enhanced by the front end.
  • Cascaded joint DNNs reconstruct clean features from noisy features and noise estimates, map them to phonetic targets, and then fine-tune both networks together.
  • Directly concatenated separation and acoustic-model networks can jointly adjust all weights, allowing enhancement to provide acoustically useful speech and recognition to guide enhancement.
  • Parallel joint training mutually concatenates hidden-layer activations, enabling enhancement and recognition networks to cooperate beyond one-way cascaded communication.
  • End-to-end architectures jointly optimize front-end representation-learning networks and back-end prediction networks for noisy speech recognition.
  • Multi-channel front ends use beamforming or post-filtering, with adaptive beamformers addressing time-varying noise through signal and noise statistics.
  • NN-supported beamformers estimate either speech/noise masks for covariance-based filters or filter coefficients directly.

B. Joint Front- and Back-End Multi-Channel Techniques

Joint multi-channel ASR systems process microphone signals within recognition networks rather than relying only on separate enhancement stages. Architectures progress from feature concatenation to convolutional and raw-waveform models that exploit spatial information.

  • B. Joint Front- and Back-End Multi-Channel Techniques: Joint multi-channel ASR aims to reduce WER directly by integrating microphone-channel processing with the acoustic model.
  • B. Joint Front- and Back-End Multi-Channel Techniques: A simple architecture concatenates features from individual microphone channels into one vector before feeding them to a DNN acoustic model.
  • B. Joint Front- and Back-End Multi-Channel Techniques: A more sophisticated network applies separate convolutional layers to each channel, then uses cross-channel max pooling before shared fully connected layers.
  • B. Joint Front- and Back-End Multi-Channel Techniques: Raw-speech extensions remove cross-layer max pooling and can exploit spatial information in fine temporal structure, including multichannel FFT phase.
  • B. Joint Front- and Back-End Multi-Channel Techniques: Table IV benchmarks deep-learning-related systems on Aurora-4, CHiME-2, CHiME-4, and AMI, with results distinguished by evaluation subsets and channel counts where reported.
  • B. Joint Front- and Back-End Multi-Channel Techniques: By separating spatial and spectral filters into distinct network layers, the investigated system notably improved WER.

VII. CONCLUSIONS

The survey organizes environmentally robust deep-learning ASR methods by channel count and ASR processing stage, and benchmarks representative systems across four standard corpora. It identifies a shift toward raw-speech, joint, end-to-end, and multi-channel approaches, while noting persistent performance and resource challenges.

  • The survey categorizes deep-learning methods by single- versus multi-channel processing and by front-end, back-end, or joint ASR stages.
  • The evaluation uses Aurora-4, CHiME-2, CHiME-4, and AMI to compare representative approaches.
  • Robust ASR is shifting from hand-crafted features toward representative features extracted directly from noisy raw speech, preserving information such as phase.
  • Separate front-end and back-end systems are increasingly replaced by joint or end-to-end training that optimizes processing stages simultaneously.
  • Multi-channel approaches are increasingly important because of the promising performance they offer.
  • Training on diverse and severe acoustic environments requires larger datasets, more complex deep networks, and greater computing resources.
  • A performance gap remains between state-of-the-art noise-robust systems and systems evaluated in clean, degradation-free environments.
Loading 1705.10874v3…