Source-linked AI summary
Weighted Speech Distortion Losses for Neural-network-based Real-time Speech Enhancement
Yangyang Xia, Sebastian Braun, Chandan K. A. Reddy, Harishchandra Dubey, Ross Cutler, Ivan Tashev
TL;DR
Real-time speech enhancement must balance speech quality and intelligibility against noise reduction, while many DNN approaches do not enforce online causality. This paper trains a compact, single-frame RNN with MSE-based losses that separately weight speech distortion and noise reduction. The authors report that the approach outperforms other competitive online methods in objective and subjective tests.
Problem
Speech enhancement must preserve perceived speech quality while suppressing noise, and many DNN methods’ performance as look-ahead decreases is not well investigated.
Source: §1 · p. 1Method
The paper uses a compact online RNN to enhance single-frame STFT magnitudes and proposes MSE-based losses with separate control of speech-distortion weighting.
Source: §1 · p. 1, §2 · pp. 1–2, §4.3 · p. 3Results
Both objective and subjective tests show that the method outperforms other competitive online methods.
Source: §6 · p. 4Takeaways & Limitations
The speech-distortion weighting can be adjusted to balance speech distortion and noise reduction for different applications.
Source: §5.2 · p. 4, §6 · p. 4The real-time formulation restricts gain estimation to past and present input information and assumes microphone signals are represented in the STFT domain.
Source: §2 · pp. 1–2
Abstract
from arXiv · showhide
This paper investigates several aspects of training a RNN (recurrent neural network) that impact the objective and subjective quality of enhanced speech for real-time single-channel speech enhancement. Specifically, we focus on a RNN that enhances short-time speech spectra on a single-frame-in, single-frame-out basis, a framework adopted by most classical signal processing methods. We propose two novel mean-squared-error-based learning objectives that enable separate control over the importance of speech distortion versus noise reduction. The proposed loss functions are evaluated by widely accepted objective quality and intelligibility measures and compared to other competitive online methods. In addition, we study the impact of feature normalization and varying batch sequence lengths on the objective quality of enhanced speech. Finally, we show subjective ratings for the proposed approach and a state-of-the-art real-time RNN-based method.
1. INTRODUCTION
Real-time enhancement must balance perceived speech quality against noise suppression, while many DNN methods do not enforce the causal constraints required for online use. The paper studies an online RNN system and training choices for controlling this trade-off.
1. INTRODUCTION
- Many DNN-based methods do not enforce causality, and the performance of high-quality systems as look-ahead decreases is not well investigated.Online processing is required for real-time communication applications.Source: §1 · p. 1
- Preserving perceived speech quality while suppressing noise is a central challenge in speech enhancement.Deep learning makes it relatively easy to incorporate learning objectives intended to improve speech quality and intelligibility.Source: §1 · p. 1
- The paper proposes an online speech-enhancement system for real-time applications and a compact RNN that produces a gain function from one noisy frame.It also studies features, normalization, error weighting, and training strategies.Source: §1 · p. 1
- Two MSE-based losses provide separate control over speech distortion and noise reduction.The paper evaluates their effects using subjective and objective speech-quality and intelligibility measures.Source: §1 · p. 1
2. PROBLEM FORMULATION
The problem is represented in the STFT domain, where the observed signal comprises clean speech and noise. The system estimates a time-varying magnitude gain using only past and present input information, then applies the noisy phase to the enhanced magnitude.
2. PROBLEM FORMULATION
- The formulation assumes microphone signals are represented in the short-time Fourier transform domain.The observed noisy signal is modeled as clean speech plus noise in that domain.Source: §2 · p. 1
- The system seeks a time-varying STFT-magnitude gain that recovers the clean-speech magnitude as closely as possible.The gain is estimated from transformed and normalized noisy-signal features using a DNN.Source: §2 · pp. 1–2
- For real-time processing, the gain may depend only on past and present input information.This expresses the causal constraint imposed on the gain computation.Source: §2 · p. 2
- The system applies the noisy signal’s phase to the estimated enhanced magnitude to obtain the enhanced signal.The DNN parameters are denoted by Θ; f transforms STFT magnitudes and g normalizes them.Source: §2 · p. 2
3. STATE-OF-THE-ART ONLINE NOISE REDUCTION
Online noise reduction methods include statistical classical approaches and DNN-based systems that use MSE objectives. RNNoise combines RNN gain estimation with pitch filtering and has reported strong objective and subjective quality.
3. STATE-OF-THE-ART ONLINE NOISE REDUCTION
- A classical online method minimizes clean-versus-enhanced STFT-magnitude or log-STFT-magnitude MSE under uncorrelated complex-Gaussian clean-speech and noise assumptions [6, 14].More advanced noise and speech-presence-probability models can be incorporated, but retaining speech quality while suppressing highly non-stationary noise remains challenging.Source: §3 · p. 2
- Recent DNN methods typically drop statistical distribution assumptions and optimize an MSE loss by stochastic gradient descent.MSE between clean and enhanced STFT magnitudes is among the most popular loss functions.Source: §3 · p. 2
- RNNoise estimates a smoothed-energy gain function with an RNN and interpolates spectral details using pitch filtering.Experiments report strong objective and subjective quality for its enhanced speech.Source: §3 · p. 2
4. PROPOSED METHODS
The proposed system uses normalized spectral features and a compact recurrent network to enhance speech one frame at a time. Its losses separate speech distortion from noise reduction, with an SNR-weighted variant adjusting their relative importance by utterance SNR.
4.1. Feature representation
- The proposed features combine STFT magnitude or log-power spectra with global, frequency-dependent, or frequency-independent normalization.Frequency-dependent online normalization uses smoothed running statistics; the experiments outside feature comparisons use τ = 3s.Source: §4.1 · p. 2
4.2. Learning machine
- A three-layer GRU and sigmoid fully connected output map one noisy spectral frame to a magnitude gain function.The authors omit convolution layers and instead explore temporal modeling through training with different sequence lengths, features, and losses.Source: §4.2 · p. 2
4.3. Loss functions
- The loss design starts with regular MSE, then separates speech distortion from noise reduction to control their relative importance.The speech term is computed over speech-active frames; training requires clean speech and noise, while enhancement uses noisy phase to reconstruct the waveform.Source: §4.3 · pp. 2–3
- The SNR-weighted loss adjusts speech-distortion weighting by utterance SNR; β marks equal weighting and the point of greatest weighting sensitivity.Source: §4.3 · p. 3, Figure 1 · p. 3
5. EXPERIMENTAL RESULTS AND DISCUSSIONS
The selected fixed-weight system outperformed tested classical and DNN-based online methods on all objective metrics, while listening-test preferences differed from objective predictions. The experiments also found gains from longer training segments up to five seconds and modest advantages for fixed over SNR weighting.
5.2. Results & Discussions
- Performance improved noticeably as training segments grew to five seconds, with diminishing improvement beyond that length.Each minibatch contained one minute of noisy speech; the authors recommend segments of at least five seconds.Source: Table 1 · p. 3, §5.2 · pp. 3–4
5.1. Corpora & Experimental setup
- Training and evaluation used 84 hours each of clean speech and noise, with 18 hours of noisy evaluation speech and nine noise types absent from training.The evaluation set included highly non-stationary noises such as munching, multi-talker babble, and keyboard typing.Source: §5.1 · p. 3
- The study compared three baselines using PESQ, STOI, cepstral distance, and SI-SDR, alongside listening ratings from 20 crowd-sourced listeners per clip.The baselines were a classical LSA enhancer, RNNoiseI, and RNNoise257.Source: §5.1 · p. 3
5.2. Results & Discussions
- Fixed weighting was slightly better than SNR weighting across all metrics; STOI and CD shared optima at α = 0.65 and β = 18.2 dB.The optimal SNR weights for all metrics were concentrated around 20 dB.Source: Figure 3 · p. 4, §5.2 · p. 4
- All selected proposed systems significantly outperformed RNNoiseI in listening tests, while listeners preferred a lower α than objective measures predicted.Below α = 0.35, the authors observed noticeable speech distortion alongside greater noise suppression.Source: Table 2 · p. 4, §5.2 · p. 4
- The selected system, trained with fixed speech-distortion weighting at α = 0.35, outperformed all tested classical and DNN-based methods on every objective metric.The comparison includes baseline online systems and an oracle Wiener filter as an upper bound.Source: Table 3 · p. 4, §5.2 · p. 4
6. CONCLUSIONS
The compact RNN speech-enhancement method outperforms competitive online methods in objective and subjective tests, while examining training choices that affect enhancement quality.
6. CONCLUSIONS
- Both objective and subjective tests show the method outperforms other competitive online methods.The method is a real-time speech-enhancement approach based on a compact recurrent neural network.Source: §6 · p. 4
- The approach uses a compact recurrent neural network trained with a simple MSE-based speech-distortion-weighted loss.Source: §6 · p. 4
- Feature normalization techniques and sequence lengths affect the objective quality of enhanced speech.Source: §6 · p. 4
- Fixed-weighted and SNR-weighted loss coefficients provide control over the amount of speech distortion.Source: §6 · p. 4
- Future work will explore time-varying speech-distortion weighting and its influence on subjective and objective speech quality.Source: §6 · p. 4