PaperlayerView exact revision on arXiv

Source-linked AI summary

A Hybrid DSP/Deep Learning Approach to Real-Time Full-Band Speech Enhancement

Jean-Marc Valin

arXiv:1709.08243v3cs.SDeess.AS

TL;DR

Noise suppression remains difficult to tune, while existing deep-learning approaches can be too complex for low-latency full-band operation. This paper combines deep learning for critical-band gain estimation with DSP pitch filtering, achieving significantly higher quality than a conventional spectral estimator at low enough complexity for real-time 48 kHz processing.

  • Problem

    Noise suppression depends on difficult-to-design and difficult-to-tune estimators, while many deep-learning approaches are unsuitable for low-complexity, low-latency full-band operation.

  • Method

    A hybrid system uses deep learning to estimate 22 ideal critical-band gains and DSP techniques, including a pitch filter, for complementary suppression tasks.

  • Results

    The approach provides significantly higher quality than a pure signal-processing approach while requiring around 40 Mflops for real-time operation.

  • Takeaways & Limitations

    The low complexity and low latency make the approach suitable for mobile or embedded devices and video-conferencing systems.

Abstract

from arXiv · show

Despite noise suppression being a mature area in signal processing, it remains highly dependent on fine tuning of estimator algorithms and parameters. In this paper, we demonstrate a hybrid DSP/deep learning approach to noise suppression. A deep neural network with four hidden layers is used to estimate ideal critical band gains, while a more traditional pitch filter attenuates noise between pitch harmonics. The approach achieves significantly higher quality than a traditional minimum mean squared error spectral estimator, while keeping the complexity low enough for real-time operation at 48 kHz on a low-power processor.

I. INTRODUCTION

Traditional noise suppression relies on multiple estimators that are difficult to design and tune. The paper proposes a low-complexity hybrid approach combining deep learning with established signal-processing components for real-time full-band speech enhancement.

  • Traditional algorithms combine noise spectral estimation, voice activity detection, and spectral estimation, all of which require accurate tuning.
  • The proposed system targets low-complexity, low-latency noise suppression for real-time applications such as videoconferencing at 48 kHz.
  • Deep learning replaces traditionally difficult-to-tune estimators while proven signal-processing techniques remain in the hybrid pipeline.
  • The paper reports acceptable complexity and better quality than conventional approaches.
  • The system applies recurrent-neural-network gains to a low-resolution spectral envelope and uses a pitch comb filter for finer inter-harmonic suppression.

A. Band structure

The approach reduces neural-network output dimensionality by estimating gains over 22 triangular critical bands rather than individual frequency bins. These bounded gains are interpolated across frequency bins to control the spectrum efficiently.

  • Band structure: 48 kHz processing would require about 400 frequency-bin outputs for 20 ms frames, making direct spectral-magnitude estimation too complex.
  • Band structure: The method assumes speech and noise spectral envelopes are sufficiently flat to support coarser-than-bin resolution.
  • Band structure: The spectrum is divided into 22 triangular bands following an approximation of the Bark scale, with at least four bins at low frequencies.
  • Band structure: Ideal critical-band gains are bounded between 0 and 1, requiring only 22 network output values.
  • Band structure: For each band, gain is based on the ratio of clean-speech energy to noisy-input energy, then interpolated onto individual frequency bins.

B. Pitch filtering

The pitch-filtering stage compensates for the coarse spectral resolution of Bark-derived gains by attenuating noise between pitch harmonics. It uses per-band pitch correlations and a heuristic filtering coefficient, balancing suppression against distortion.

  • B. Pitch filtering: Bark-derived bands cannot model fine spectral details, so the coarse gain computation does not suppress noise between pitch harmonics.A pitch comb filter addresses this limitation by targeting inter-harmonic noise.
  • B. Pitch filtering: The filter operates in the frequency domain using a pitch-delayed signal and a per-band coefficient α_b.It computes X(k) + α_bP(k) and then renormalizes the result.
  • B. Pitch filtering: The pitch correlation p_b is computed from weighted frequency-domain energy and cross-correlation terms for each band.For a single band, this is equivalent to time-domain pitch correlation.
  • B. Pitch filtering: The heuristic sets α_b=1 when p_b≥g_b, α_b=0 when g_b=1, and α_b=0 when p_b=0.These constraints respectively preserve strong periodicity, avoid distortion in clean bands, and avoid enhancing bands without pitch.

C. Feature extraction

The system extracts Bark-scale spectral and pitch-correlation features for the recurrent network. The feature design preserves absolute level information, making it sensitive to signal amplitude and channel frequency response.

  • C. Feature extraction: The input uses the noisy log spectrum in the same Bark-derived bands as the output, transformed into 22 Bark-frequency cepstral coefficients.The feature set also includes temporal derivatives of the first 6 BFCCs and the first 6 DCT coefficients of pitch correlation across bands.
  • C. Feature extraction: The features retain the first cepstral coefficient and omit cepstral mean normalization to track the absolute noise level.This differs from features typically used in speech recognition.
  • C. Feature extraction: Figure 3 shows the neural-network layer types, activation function, and unit count for each layer.The architecture includes feed-forward fully connected layers and recurrent layers.
  • C. Feature extraction: Because cepstral mean normalization is omitted, the features are sensitive to absolute signal amplitude and channel frequency response.The paper states that this sensitivity is addressed in Sec. III-A.

III. DEEP LEARNING ARCHITECTURE

The neural network follows the structure of traditional noise-suppression algorithms while using recurrent layers for the modeled processing components. It is deliberately compact, and an auxiliary VAD output adds little complexity while improving training.

  • III. DEEP LEARNING ARCHITECTURE: The network has 4 hidden layers and 215 total units, with its largest layer containing 96 units.The architecture is based on the assumption that three recurrent layers correspond to basic noise-suppression components, though the network can deviate from this structure.
  • III. DEEP LEARNING ARCHITECTURE: Increasing the number of units does not significantly improve the quality of noise suppression.This supports the paper’s compact architecture choice.
  • III. DEEP LEARNING ARCHITECTURE: The auxiliary VAD output adds only 24 weights and improves training by encouraging its corresponding GRU to discriminate speech from noise.The VAD output is included even though it is not strictly necessary.

A. Training data

Training data is created by mixing clean multilingual speech with diverse noises at varied levels, then augmenting frequency responses and signal amplitudes. The process expands 10 hours of source data into 140 hours of noisy speech.

  • A. Training data: The training set combines McGill TSP and NTT multilingual speech databases with computer-fan, office, crowd, transportation, and construction noises.The speech data includes French and English, while the NTT database covers 21 languages.
  • A. Training data: Noise is mixed at different levels to create a wide range of signal-to-noise ratios, including clean-speech and noise-only segments.This provides varied training conditions for the gain targets.
  • A. Training data: Independent filtering of speech and noise in each training example augments frequency responses because cepstral mean normalization is not used.The augmentation is intended to improve robustness to frequency-response variation.
  • A. Training data: Varying the final level of the mixed signal provides robustness to signal amplitude.The filter coefficients used for augmentation are randomly selected within a specified range.
  • A. Training data: 140 hours of noisy speech are generated from 6 hours of speech and 4 hours of noise using varied gains, filters, and resampling frequencies from 40 kHz to 54 kHz.The data-generation process uses combinations of gains and filters to expand the original recordings.

B. Optimization process

The network is trained with a perceptually motivated gain loss rather than binary cross-entropy, while the VAD uses standard cross-entropy. The loss exponent controls suppression aggressiveness, with γ = 1/2 providing a practical trade-off.

  • The gain loss is designed to balance excessive and insufficient attenuation according to perceptual effects.Binary cross-entropy is unsuitable because it does not match the perceptual effect of gain errors.
  • The exponent γ controls how aggressively the system suppresses noise.
  • γ = 1/2 provides a good trade-off and corresponds to minimizing mean-squared error on energy raised to the power 1/4.
  • The network’s VAD output is trained with the standard cross-entropy loss using Keras and TensorFlow.

C. Gain smoothing

Gain smoothing addresses overly dry output by limiting gain decay across frames, preserving a controlled reverberation level. The decay factor λ = 0.6 corresponds to a 135 ms reverberation time.

  • Limiting gain decay across frames remedies output that can sound overly dry and lack sufficient reverberation.
  • Figure 4 illustrates noisy, denoised, and clean speech spectrograms for babble noise at 15 dB SNR over 0–12 kHz.
  • The smoothed gains are computed from the current estimated gains while limiting their decay relative to the previous frame.
  • λ = 0.6 is equivalent to a reverberation time of 135 ms.

IV. COMPLEXITY ANALYSIS

The implementation is designed for low memory and computational cost. Its total complexity is around 40 Mflops, and measured CPU use remains modest for 48 kHz single-channel suppression.

  • 87,503 weights represent the neural networks’ 215 units, and 8-bit quantization preserves performance while fitting the weights in CPU L2 cache.
  • 175,000 floating-point operations per frame correspond to 17.5 Mflops for real-time neural-network processing.
  • 40 Mflops is the algorithm’s total complexity, including neural-network processing, FFTs, IFFT, and pitch search.
  • A non-vectorized implementation uses around 1.3% of one Haswell i7-4800MQ core for 48 kHz single-channel suppression, versus 14% on a 1.2 GHz ARM Cortex-A53.
  • A comparison 16 kHz approach requires 12.5 million weights and 1600 Mflops, with about 800 MB/s memory bandwidth even after quantization.

V. RESULTS

The system is evaluated on held-out speech and noise against the SpeexDSP MMSE suppressor. The reported results show significantly improved quality, especially for non-stationary noise, with the demonstration and implementation made available publicly.

  • The evaluation uses speech and noise data not included in training and compares against the SpeexDSP MMSE-based noise suppressor.
  • Figure 5 evaluates PESQ MOS-LQO quality for babble, car, and street noise.
  • The objective evaluation reports significant quality improvement from deep learning, especially for non-stationary noise types.
  • The quality improvement is also confirmed by casual listening, and Figure 4 shows an example of the suppression effect.
  • An interactive real-time JavaScript demonstration and BSD-licensed implementation are available.
Loading 1709.08243v3…