Source-linked AI summary
Improved Speech Enhancement with the Wave-U-Net
Craig Macartney, Tillman Weyde
TL;DR
The paper asks whether Wave-U-Net can improve speech enhancement beyond predominantly time-frequency approaches. It applies the architecture directly to mixture waveforms and finds improved objective performance over the state of the art on VCTK, with fewer layers sufficient than for music vocal separation.
Problem
Most audio source-separation methods use time-frequency representations, motivating evaluation of a time-domain approach for speech enhancement.
Method
The paper applies Wave-U-Net, a one-dimensional U-Net with downsampling and upsampling blocks, to separate mixture waveforms into source waveforms.
Results
Wave-U-Net is reported as the most effective speech-enhancement model, with performance peaking around the 10- and 9-layer variants.
Takeaways & Limitations
The results indicate potential for further exploring Wave-U-Net and time-domain speech enhancement.
Takeaways & Limitations
The authors identify further adaptation of model size and parameters, and expansion to multi-channel and multi-source separation, as future work.
Abstract
from arXiv · showhide
We study the use of the Wave-U-Net architecture for speech enhancement, a model introduced by Stoller et al for the separation of music vocals and accompaniment. This end-to-end learning method for audio source separation operates directly in the time domain, permitting the integrated modelling of phase information and being able to take large temporal contexts into account. Our experiments show that the proposed method improves several metrics, namely PESQ, CSIG, CBAK, COVL and SSNR, over the state-of-the-art with respect to the speech enhancement task on the Voice Bank corpus (VCTK) dataset. We find that a reduced number of hidden layers is sufficient for speech enhancement in comparison to the original system designed for singing voice separation in music. We see this initial result as an encouraging signal to further explore speech enhancement in the time-domain, both as an end in itself and as a pre-processing step to speech recognition systems.
1 Introduction
The paper applies Wave-U-Net, a time-domain audio source-separation architecture, to speech enhancement and positions it within related voice-separation tasks. The paper then presents its organization across related work, architecture, experiments, and conclusions.
- Speech enhancement improves intelligibility and quality by extracting human voice while suppressing additive noise.
- The paper applies Wave-U-Net to speech enhancement and reports results better than the current state of the art.
- Audio source separation methods commonly use time-frequency representations rather than operating directly in the time domain.
- The paper covers related work, the Wave-U-Net architecture, experiments and comparisons, then conclusions and future perspectives.
2 Related work
Related work includes predominantly time-frequency source-separation methods and newer time-domain speech-enhancement models. Wave-U-Net combines these architectural developments with a one-dimensional U-Net design.
- U-Net architectures have achieved state-of-the-art results for music source separation and speech dereverberation on magnitude spectrograms.
- SEGAN performs speech enhancement and denoising directly in the time domain using encoder-decoder pathways and skip connections.
- Wavenet for Speech Denoising uses non-causal conditional input, parallel sample prediction, and dilated convolutions to incorporate context.
3 Wave-U-Net for Speech Enhancement
For speech enhancement, Wave-U-Net uses a one-dimensional U-Net with downsampling and upsampling blocks to estimate source waveforms directly from a mixture waveform.
- Wave-U-Net uses a one-dimensional U-Net architecture with downsampling and upsampling blocks.
- At each network level, Wave-U-Net halves the time resolution through its downsampling and upsampling design.
- The network separates a mixture waveform into K source waveforms; monaural speech enhancement uses K = 2 and C = 1.
- A final 1D convolution predicts each source sample, followed by tanh to constrain estimates to the interval (−1, 1).
4 Experiments
Experiments evaluate Wave-U-Net speech enhancement on VCTK using objective measures and comparisons with established enhancement methods. Results identify the strongest model variation and show that fewer layers can suffice than in music separation.
- 4.1 Datasets: VCTK provides clean and noisy speech, downsampled from 48kHz to 16kHz, with 30 speakers and noise mixtures for training and testing.
- 4.1 Datasets: 11,572 training samples cover 40 noise conditions formed from 10 noise types and four SNRs.
- 4.1 Datasets: The 824-item test set mismatches training speakers, noise types, and SNRs, using two unseen speakers.
- 4.3 Results: Evaluation compares enhanced speech with clean references using PESQ, CSIG, CBAK, COVL, and SSNR.
- 4.3 Results: The 10-layer fine-tuned Wave-U-Net is the overall best-performing model and is reported as the most effective speech-enhancement model.
- 4.3 Results: Performance peaks around the 10- and 9-layer models, while fine-tuning meaningfully affects mainly CSIG.
5 Conclusions
The Wave-U-Net is particularly effective for speech enhancement, improving over the state of the art without significant adaptation or parameter tuning. Further work should investigate the source of its advantage and adapt model settings to the task.
- The Wave-U-Net is particularly effective at speech enhancement and improves over the state of the art by a good margin.
- The improvement occurs without significant adaptation or parameter tuning, indicating potential for speech enhancement.
- The possible advantage from aliasing-avoiding upsampling remains to be investigated, alongside adapting model size, filter sizes, and audio-source settings.