Source-linked AI summary
Real Time Speech Enhancement in the Waveform Domain
Alexandre Defossez, Gabriel Synnaeve, Yossi Adi
TL;DR
Speech enhancement must remove diverse background noise while operating online with minimal lag on commodity hardware. This paper adapts DEMUCS into a causal raw-waveform encoder-decoder with skip-connections, joint waveform and spectrogram losses, and waveform augmentations. Across standard evaluations, it matches state-of-the-art performance, improves ASR in noisy conditions, and benefits from targeted reverberation and loss/augmentation choices.
Problem
Speech enhancement needs to remove noise while meeting real-time, low-lag requirements on commodity communication devices.
Method
The paper adapts DEMUCS into a causal raw-waveform encoder-decoder with skip-connections, waveform and spectrogram losses, and raw-waveform augmentations.
Results
The method matches current state-of-the-art performance across objective and subjective metrics while operating directly on raw waveforms.
Takeaways & Limitations
Enhanced samples improve ASR performance in noisy conditions, including without retraining the acoustic model.
Takeaways & Limitations
Baseline MOS values differ from the challenge because evaluation used 100 examples per category and ITU-T P.835 instead of P.808.
Abstract
from arXiv · showhide
We present a causal speech enhancement model working on the raw waveform that runs in real-time on a laptop CPU. The proposed model is based on an encoder-decoder architecture with skip-connections. It is optimized on both time and frequency domains, using multiple loss functions. Empirical evidence shows that it is capable of removing various kinds of background noise including stationary and non-stationary noises, as well as room reverb. Additionally, we suggest a set of data augmentation techniques applied directly on the raw waveform which further improve model performance and its generalization abilities. We perform evaluations on several standard benchmarks, both using objective metrics and human judgements. The proposed model matches state-of-the-art performance of both causal and non causal methods while working directly on the raw waveform.
1. Introduction
Speech enhancement aims to improve speech quality by removing noise, but real-time, low-lag operation on commodity hardware remains important for practical applications. The paper adapts DEMUCS into a causal waveform model and evaluates its losses, augmentations, and enhancement effects using objective, subjective, and ASR measures.
- Motivation: Real-time, low-lag speech enhancement is important for calls, hearing aids, and ASR on communication devices.Recorded conversational speech commonly contains noise that can hinder intelligibility.
- Motivation: Traditional noise-model approaches struggle with non-stationary and babble noise, motivating neural enhancement methods.These noise types greatly degrade human-speech intelligibility.
- Approach: The proposed causal DEMUCS model processes waveform audio with convolutions and LSTMs faster than real time on one laptop CPU core.Its frame size is 40ms and stride is 16ms.
- Approach: The model uses U-Net-like skip-connections, waveform and spectrogram losses, and raw-waveform frequency masking and reverberation augmentations.The target is the clean speech waveform, optimized with L1 loss complemented by a spectrogram-domain loss.
- Evaluation: Objective and subjective evaluations are combined because existing speech-enhancement metrics correlate imperfectly with human judgments.The study also ablates losses and augmentations and analyzes enhancement artifacts using ASR word error rates.
- Results: The method is comparable to state-of-the-art performance across metrics while operating directly on the raw waveform.Enhanced samples also improve ASR under noisy conditions.
2. Model
DEMUCS is adapted as a causal, monaural encoder-decoder that maps noisy raw waveforms to estimated clean speech. Its architecture combines convolutional encoding and decoding, sequence modeling, skip-connections, resampling, and joint waveform–multi-resolution STFT objectives.
- Problem formulation: The task models noisy audio as x = y + n and seeks an enhancement function f such that f(x) ≈ y.The setup is monaural and supports utterances with varying lengths.
- Architecture: DEMUCS is adapted from music source separation into a causal speech-enhancement architecture for real-time processing.The architecture is described visually in Figure 1a.
- Architecture: The model uses a multilayer convolutional encoder-decoder, U-Net skip-connections, and sequence modeling over the encoder output.Its principal design parameters include depth L, hidden channels H, kernel size K, stride S, and resampling factor U.
- Encoder: The encoder transforms the raw waveform into a latent representation z through successive convolutional layers.Each encoder layer uses a kernel size K and stride S, with channel counts determined by H and layer depth.
- Architecture: Figure 1 depicts noisy speech entering the causal model and clean speech emerging, with arrows denoting U-Net skip-connections.The figure also distinguishes encoder and decoder layers and notes that input/output resampling by U is omitted.
- Sequence modeling and decoder: A unidirectional LSTM causally transforms the latent representation, while the decoder maps it to an estimated clean signal and receives encoder skip connections.The decoder uses transposed convolutions and outputs one channel in the final layer.
- Resampling: Upsampling by U before encoding and downsampling afterward improves accuracy, with sinc interpolation included in end-to-end training.The resampling is part of the model pipeline rather than a separate preprocessing step.
- Objective: Training combines waveform L1 loss with a multi-resolution STFT loss composed of spectral-convergence and magnitude terms.The multi-resolution loss sums STFT losses computed with different parameters, including FFT bins 512, 1024, and 2048.
3. Experiments
The experiments evaluate causal and non-causal DEMUCS models on speech enhancement, reverberation, ablations, real-time processing, and noisy-condition ASR. Results show competitive enhancement quality, practical streaming performance, and improved ASR without retraining.
- Evaluation setup: The evaluation covers objective and subjective speech-quality measures on the Valentini and DNS benchmarks, plus ablations and noisy-condition ASR tests.Subjective evaluation uses ITU-T P.835 MOS ratings across distortion, background-noise intrusiveness, and overall quality.
- Valentini benchmark: DEMUCS matched the current state-of-the-art model across objective and subjective Valentini measures while operating directly on raw waveforms without extra training data.It also outperformed the other causal and non-causal baselines by a significant margin.
- DNS benchmark: Partial dereverberation improved overall ratings on artificial-reverb recordings, while two-source reverb simulation improved ratings on real recordings.The remove-reverb model performed poorly on the reverb category because of dereverberation artifacts; partial dereverberation did not improve real recordings.
- Ablation study: The STFT loss and time-shift augmentation produced the largest ablation gains, whereas remix augmentation contributed only modestly.The ablation used the causal DEMUCS model and reported PESQ and STOI.
- Real-time evaluation: 0.8 RTF was achieved by the H=48 model on a single CPU core, supporting real-time use under streaming conditions.On a quad-core Intel i5, the H=48 and H=64 models achieved RTFs of 0.6 and 1.05, respectively.
- Effect on ASR models: 51% of the WER lost to noise at SNR 0 was recovered, with average recovery of 41% on dev-clean and 31% on dev-other.The acoustic model was not retrained on denoised data, testing enhancement as a black-box preprocessing step.
4. Related Work
Traditional enhancement methods estimate spectral quantities or masks, while recent DNN approaches address difficult noise conditions and increasingly support real-time operation. The paper positions causal waveform enhancement among these spectral and neural alternatives.
- Traditional methods: Traditional speech enhancement estimates an enhanced magnitude spectrum or an ideal binary mask used to enhance that spectrum.These approaches operate primarily through spectral representations.
- DNN methods: DNN-based methods include waveform-loss-trained frequency masks, multi-objective losses, and recurrent neural networks for speech enhancement.The cited approaches represent several ways to combine neural modeling with spectral or waveform objectives.
- Causal methods: Causal prior work includes convolutional recurrent spectral models, weighted-loss real-time systems, and MMSE noise-power-spectrum tracking.The cited literature contains both causal and non-causal enhancement methods.
5. Discussion
The study adapts DEMUCS into a causal waveform speech enhancer that runs in real time on consumer CPUs. It achieves state-of-the-art benchmark performance, improves subjective scores through targeted augmentation, and benefits noisy-condition ASR without retraining.
- Discussion: DEMUCS was converted from a music source-separation architecture into a causal speech enhancer operating in real time on consumer-level CPUs.The model was evaluated on standard speech-enhancement benchmarks.
- Discussion: Reverberation augmentation using two sources and partial dereverberation significantly improved subjective evaluations.These augmentation strategies were tested under real reverberant conditions with the DNS dataset.
- Discussion: The model improved noisy-condition ASR performance without retraining the acoustic model.This supports using enhancement as a preprocessing stage for noisy speech recognition.