Source-linked AI summary
Convolutional-Recurrent Neural Networks for Speech Enhancement
Han Zhao, Shuayb Zarar, Ivan Tashev, Chin-Hui Lee
TL;DR
Speech enhancement must handle deployment noise that differs from clean training data, while existing approaches can depend on unrealistic noise assumptions or limited MLP structure. The paper proposes EHNET, an end-to-end convolutional-recurrent model that exploits local spectrogram structure and temporal correlations without assuming the noise type. On synthetic data, EHNET outperforms the compared methods, improving PESQ by 0.6 on seen noise and 0.64 on unseen noise.
Problem
Speech enhancement seeks to reduce noise caused by mismatch between clean training data and noisy deployment conditions, while statistical methods may rely on unrealistic noise assumptions.
Method
EHNET is a purely data-driven end-to-end model combining convolutional networks for local spectrogram patterns with bidirectional recurrent networks for temporal correlations.
Results
EHNET outperforms the compared methods, improving PESQ by 0.6 on seen noise and 0.64 on unseen noise.
Takeaways & Limitations
EHNET achieves better generalization on both seen and unseen noise while exploiting speech-related local frequency and temporal structure.
Abstract
from arXiv · showhide
We propose an end-to-end model based on convolutional and recurrent neural networks for speech enhancement. Our model is purely data-driven and does not make any assumptions about the type or the stationarity of the noise. In contrast to existing methods that use multilayer perceptrons (MLPs), we employ both convolutional and recurrent neural network architectures. Thus, our approach allows us to exploit local structures in both the frequency and temporal domains. By incorporating prior knowledge of speech signals into the design of model structures, we build a model that is more data-efficient and achieves better generalization on both seen and unseen noise. Based on experiments with synthetic data, we demonstrate that our model outperforms existing methods, improving PESQ by up to 0.6 on seen noise and 0.64 on unseen noise.
1. INTRODUCTION
Speech enhancement is important for robust ASR and communication because deployment noise differs from training conditions. EHNET addresses limitations of noise-assumption-based and MLP approaches with a data-driven convolutional-recurrent architecture.
- Speech enhancement reduces noise before speech reaches ASR and communication systems, helping address clean-training versus noisy-deployment mismatch.
- Statistical methods often rely on unrealistic assumptions about noise and may introduce additional artifacts into recovered speech.
- Prior deep-learning approaches commonly use MLPs to approximate mappings from noisy utterances to clean speech, but their temporal modeling is limited in the cited discussion.
- EHNET combines convolutional and recurrent networks in a purely data-driven model without assumptions about the underlying noise.
- EHNET uses convolution to exploit local spectrogram structure and bidirectional recurrence to model temporal dynamics, supporting data efficiency and generalization to seen and unseen noise.
2. MODELS AND LEARNING
EHNET formulates speech enhancement as multivariate regression from noisy to clean spectrograms. Its convolutional, bidirectional recurrent, and output components incorporate local frequency-temporal structure and frame dependencies into an end-to-end model.
- Problem formulation: EHNET learns a parameterized mapping from noisy spectrograms to clean spectrograms by minimizing squared prediction error over paired training data.
- Convolutional component: Convolutional kernels extract local spectrogram patterns, addressing continuity across time and similarity across adjacent frequency bins more efficiently than fully connected layers.
- Convolutional component: Zero-padding and an odd temporal kernel width preserve the input spectrogram's number of time steps after convolution.
- Convolutional component: A frequency stride of b/2 reduces parameters and computation in the recurrent component while retaining prediction accuracy, according to the design description.
- Bidirectional recurrent component: A bidirectional recurrent component models dynamic correlations between adjacent frames, using recurrent connections in both temporal directions.
- Bidirectional recurrent component: The convolutional feature maps are vertically concatenated into a stacked 2D map before entering the bidirectional LSTM.
- Output and learning: A linear regression output with truncation predicts each clean spectrogram frame while enforcing nonnegative values, followed by joint end-to-end optimization.
3. EXPERIMENTS
Experiments use a synthetic dataset and five metrics to compare EHNET with statistical, feedforward, and recurrent baselines under seen and unseen noise. EHNET improves perceptual quality and recognition accuracy while producing spectrograms closer to clean speech.
- Dataset and Setup: The synthetic dataset contains 7,500 training, 1,500 validation, and 1,500 test recordings generated with 48 room impulse responses and randomly selected noises.Spectrograms use 256 frequency bins and approximately 500 frames per utterance.
- Dataset and Setup: Evaluations use SNR, LSD, time-domain MSE, WER, and PESQ, with a fixed DNN-based speech recognizer for WER measurement.The recognizer is not fine-tuned during the experiment.
- Dataset and Setup: Compared methods include Microsoft’s production enhancement system, symmetric and causal DNN baselines, and the recurrent RNN-NG model.The DNN baselines use three hidden layers with 2,048 units, while RNN-NG uses three 500-unit hidden layers.
- Dataset and Setup: EHNET uses 256 convolution kernels of size 32 × 11, followed by two bidirectional LSTM layers with 1,024 hidden units each.Training runs for 200 epochs with scheduled learning rates and validation-based early stopping.
- Results and Analysis: 0.6 PESQ improvement is achieved on seen noise without hurting recognition accuracy, while unseen noise yields a larger 0.64 improvement and increased recognition accuracy.Other methods often improve SNR while decreasing recognition accuracy.
- Results and Analysis: EHNET produces the best visual tradeoff between background-noise removal and speech preservation, with denoised spectrograms closest to clean speech.The comparison finds EHNET better than DNN and RNN at preserving frequency-bin signals and better than MS at removing background noise.
4. CONCLUSION
EHNET combines convolutional and recurrent neural networks for speech enhancement, using local spectrogram patterns and temporal correlations. It consistently outperforms competitors across five metrics and generalizes to unseen noises.
- Conclusion: EHNET combines convolutional and recurrent neural networks for speech enhancement.Its convolutional component detects local spectrogram patterns, while bidirectional recurrent connections model correlations between adjacent frames.
- Conclusion: EHNET requires less computation than MLPs and RNNs because convolutions use sparse connections.The architecture’s inductive bias is presented as well suited to speech enhancement.
- Conclusion: EHNET consistently outperforms all competitors on all five metrics and generalizes to unseen noises.The reported results support its effectiveness for speech enhancement in the evaluated setting.