Source-linked AI summary
A Wavenet for Speech Denoising
Dario Rethage, Jordi Pons, Xavier Serra
TL;DR
Speech denoising commonly relies on magnitude spectrograms and thereby discards phase information. The paper adapts Wavenet into a supervised, non-causal, non-autoregressive raw-audio model with target-field prediction, and reports preference over Wiener filtering while reducing time-complexity. A stated limitation is difficulty handling sudden interferences such as city-traffic honks.
Problem
Magnitude-spectrogram front-ends used in speech denoising discard potentially valuable phase information.
Method
The paper proposes a supervised discriminative Wavenet adaptation using non-causal convolutions, raw-audio predictions, target fields, and an energy-conserving regression loss.
Results
Perceptual tests show the proposed model’s estimates are preferred over Wiener-filtering estimates based on magnitude spectrograms.
Takeaways & Limitations
Target-field prediction and convolutional processing reduce time-complexity and support variable-length, one-shot denoising.
Takeaways & Limitations
The model remains unable to deal with sudden interferences such as honks in city traffic.
Abstract
from arXiv · showhide
Currently, most speech processing techniques use magnitude spectrograms as front-end and are therefore by default discarding part of the signal: the phase. In order to overcome this limitation, we propose an end-to-end learning method for speech denoising based on Wavenet. The proposed model adaptation retains Wavenet's powerful acoustic modeling capabilities, while significantly reducing its time-complexity by eliminating its autoregressive nature. Specifically, the model makes use of non-causal, dilated convolutions and predicts target fields instead of a single target sample. The discriminative adaptation of the model we propose, learns in a supervised fashion via minimizing a regression loss. These modifications make the model highly parallelizable during both training and inference. Both computational and perceptual evaluations indicate that the proposed method is preferred to Wiener filtering, a common method based on processing the magnitude spectrogram.
1 Introduction
Speech denoising commonly uses magnitude spectrograms, which discard phase information. This motivates adapting Wavenet to learn directly from raw audio.
- Magnitude-spectrogram front-ends are standard in speech denoising but discard potentially valuable phase information.
- Raw-audio neural models can represent multiple non-discrete levels of speech structure rather than focusing explicitly on separate levels.
- The study adapts Wavenet to learn multi-scale hierarchical representations from raw audio for speech denoising.
- Earlier raw-audio approaches, including feed-forward, generative-adversarial, and Bayesian Wavenet models, reported better results than magnitude-spectrogram counterparts.
- The paper presents Wavenet, proposed architectural modifications, experiments on architectural parameters, and concluding contributions.
2 Wavenet
Wavenet is an autoregressive audio model that generates samples sequentially using causal dilated convolutions and related architectural components. Its sequential generation creates a major time-complexity limitation.
- Wavenet predicts the next audio sample from previous samples and generates sequences by feeding generated samples back into the model.
- Gated units: Gated units control each layer’s activation contribution through sigmoidal gates, convolutional filters, and element-wise multiplication.
- Causal, dilated convolutions: Causal dilated convolutions use length-2 filters with exponentially increasing dilation factors, expanding the receptive field with depth.
- Skip connections: Skip connections facilitate training deep models and pass features from multiple hierarchical levels directly to final layers.
- Context stacks: Context stacks deepen the network by repeating layers up to a maximum dilation factor without proportionally expanding the receptive field.
- Time-complexity: Sequential sample generation is a significant drawback because it is non-parallelizable.
3 Wavenet for Speech Denoising
The speech-denoising adaptation removes Wavenet’s causal autoregression, predicts raw-audio target fields, and uses supervised regression with an energy-conserving loss. The architecture supports flexible, parallel denoising while using conditioning and augmentation for practical robustness.
- Speech denoising estimates clean speech st from a mixed signal mt = st + bt containing speech and background noise.
- Non-causality: Non-causal convolutions use future context and symmetric padding, removing Wavenet’s autoregressive causal structure for denoising.
- Predictions: The model predicts raw audio directly and uses real-valued predictions instead of Wavenet’s discrete softmax output.
- Energy-conserving loss: The energy-conserving loss has two terms and trains the model to account for both speech and background-noise components.
- Discriminative model: The discriminative model is trained supervised by minimizing a regression loss rather than explicitly modeling an output probability distribution.
- Target field prediction: Target-field prediction produces many samples per forward pass, reducing redundant overlapping computations and improving parallel inference efficiency.
- Conditioning: Speaker conditioning uses speaker identities 1–28 plus an all-zero code for unknown speakers, with zero-coded examples also serving as augmentation.
- Augmentation and inference: Noise-only augmentation addresses difficulty producing silence, while fully convolutional processing permits variable-length and one-shot denoising, including up to 25s on specified hardware.
4 Experiments
Experiments evaluate the Wavenet denoiser across dataset, architecture, loss, conditioning, and augmentation choices using computed and subjective MOS measures. The selected configuration combines speaker conditioning, energy-conserving loss, a 1601-sample target field, and 10% noise-only augmentation based on computed measures and informal listening.
- Experimental setup: The dataset combines Voice Bank speech from 30 speakers with environmental sounds from DEMAND, using 28 speakers for training and 2 for testing.Recordings were studio-quality, sampled at 48 kHz, and subsampled for the experiments.
- Evaluation: Evaluation uses SIG, BAK, and OVL computed MOS predictors, each ranging from 1–5 and targeting signal distortion, noise intrusiveness, and overall quality.Subjective evaluation additionally used 33 participants and 20 samples, comparing noisy, clean, Wiener-filtered, and Wavenet-denoised speech.
- Configuration studies: The basic setup with 0% noise-only augmentation performs best across all computed metrics, while 10% augmentation produces pleasant silence during nonspeech moments without degrading the signal.Using 20% augmentation lowers computed MOS ratings and does not improve denoised-sample quality in listening tests.
- Configuration studies: Longer target fields are crucial for significant denoising; small fields produce fuzzier audio and impractically long inference because of redundant computations.Results for smaller target fields were computed on the 20-sample perceptual test set because of inference-time constraints.
- Configuration studies: Speaker conditioning yields marginal but consistent improvements, while the energy-conserving loss slightly outperforms standard L1 loss with barely noticeable auditory improvement.The energy-conserving loss is related to the parameterless background-noise subtraction and is left for future study with a more powerful noise estimator.
- Comparison and selection: Against Wiener filtering, the proposed method preserves comparable speech quality while removing background noise more effectively; the selected model uses conditioning, energy-conserving loss, a 1601-sample target field, and 10% augmentation.The best-sounding configuration was chosen from computed MOS measures together with informal listening because those measures alone did not identify a clear winner.
5 Conclusion
The paper presents a non-causal, non-autoregressive Wavenet adaptation for speech denoising that reduces time-complexity while preserving effective raw-audio modeling. It supports target-field prediction and variable-length, one-shot denoising, but remains limited with sudden interferences.
- The discriminative model removes Wavenet’s autoregression and uses final 3x1 filters to maintain adjacent signal continuity.Removing autoregression reduces time-complexity, but no longer enforces temporal continuity directly.
- Target-field prediction further reduces time-complexity and significantly improves performance compared with single-sample prediction.The convolutional architecture also supports variable-length audio and one-shot denoising across differing hardware memory availability.
- Real-valued outputs removed artifacts introduced by the discrete softmax loss, while raw-audio processing permits domain-knowledge-motivated alternative costs.
- Noise-only augmentation helps the model produce silence, but it remains unable to handle sudden interferences such as city-traffic honks.The paper reports this limitation despite the augmentation strategy.
- The denoising pipeline inherently estimates both speech and background noise because noise can be obtained by subtracting the speech estimate from the mixture.
- The model denoises signals from unseen noise conditions and speakers without speech-specific constraints, while significantly reducing time-complexity.The approach relies on architectural improvements and data augmentation rather than speech-specific constraints.
- Perceptual tests preferred the model’s estimates over Wiener filtering, supporting raw-audio hierarchical modeling instead of magnitude-spectrogram front-ends.