Source-linked AI summary
Self-supervised Learning for Speech Enhancement
Yu-Che Wang, Shrikant Venkataramani, Paris Smaragdis
TL;DR
Speech enhancement often requires paired noisy mixtures and clean targets, which can be difficult to obtain and may not reflect unseen test conditions. The paper learns a representation from clean speech and trains on noisy mixtures to map them toward clean speech without paired labels. The reported experiments show improvement over spectral subtraction on DAPS and demonstrate efficacy in indoor and street-noise settings.
Problem
Supervised speech enhancement requires paired noisy mixtures and clean targets, which can be difficult to obtain, and unseen test conditions may mismatch training conditions.
Source: §1 · p. 1Method
Results
The DAPS experiments report consistent improvement over spectral subtraction across all metrics, and the approach demonstrates efficacy in ambient indoor environments and street noises.
Source: §3.3 · p. 5, §4 · p. 5Takeaways & Limitations
The approach maps noisy speech mixtures to clean sounds autonomously and without clean targets in the reported indoor and street-noise settings.
Source: §4 · p. 5The approach assumes access to clean examples representing the source, though they may be unrelated to the mixture recordings.
Source: §2 · p. 2
Abstract
from arXiv · showhide
Supervised learning for single-channel speech enhancement requires carefully labeled training examples where the noisy mixture is input into the network and the network is trained to produce an output close to the ideal target. To relax the conditions on the training data, we consider the task of training speech enhancement networks in a self-supervised manner. We first use a limited training set of clean speech sounds and learn a latent representation by autoencoding on their magnitude spectrograms. We then autoencode on speech mixtures recorded in noisy environments and train the resulting autoencoder to share a latent representation with the clean examples. We show that using this training schema, we can now map noisy speech to its clean version using a network that is autonomously trainable without requiring labeled training examples or human intervention.
1. Introduction
Supervised speech enhancement depends on paired noisy and clean examples, which can be difficult or costly to obtain and may not match unseen test conditions. The paper proposes self-supervised learning to map noisy mixtures to clean speech using clean examples and noisy recordings without paired labels.
1. Introduction
- Supervised enhancement trains on paired noisy mixtures and corresponding clean speech, but clean targets can be difficult or expensive to obtain.The introduction also notes that these networks do not autonomously learn to denoise mixtures in ambient environments.Source: §1 · p. 1
- Previously unseen test conditions can mismatch training conditions, and recorded test mixtures cannot then be used to improve the model.Source: §1 · p. 1
- Style-transfer approaches use unpaired clean sounds but jointly train two autoencoders, while weakly supervised approaches may require source-activity information obtained through human listening or clean examples.The described alternatives also may not work when test conditions differ from training conditions.Source: §1 · pp. 1–2
- Self-supervised learning uses relationships among training examples to construct paired labels, but had not been explored for audio applications including speech enhancement.Source: §1 · p. 2
- The paper learns a clean-speech representation, then uses it with noisy recordings to learn a mixture-to-clean mapping without human intervention.Source: §1 · p. 2
2. Self-Supervision for Speech Enhancement
The method learns a clean-speech representation from a small clean dataset, then trains on ambient mixtures to map them into the clean representation without paired examples. Its design combines two autoencoders with reconstruction and cycle-consistency objectives, and is intended to support autonomous training in ambient environments.
2. Self-Supervision for Speech Enhancement
- The method’s setup assumes access to a small clean-speech dataset, while the clean examples can differ from the speakers and utterances in the mixtures.The clean dataset represents the source to be isolated.Source: §2 · p. 2
- The approach uses clean examples to train a CAE, then trains an MAE on ambient mixtures to share the CAE’s latent space without paired training examples.The clean examples may be unrelated to the mixtures.Source: §2 · p. 2
2.1. Network Architecture
- The architecture uses clean-speech and mixture autoencoders; after training, the mixture encoder followed by the clean decoder produces enhanced speech.The clean autoencoder reconstructs clean magnitude spectrograms, while the MAE is trained to share its latent representation.Source: §2.1 · pp. 2–3, Figure 1 · p. 2
2.2.1. TRAINING THE CAE
- The CAE is trained to minimize the discrepancy between clean input spectrogram C and its reconstruction, with a KL term encouraging a zero-mean normal latent representation.The stated reconstruction discrepancy uses the L2 norm.Source: §2.2.1 · p. 3
2.2.2. TRAINING THE MAE
- The MAE objective combines mixture reconstruction, cycle-consistency, noise-example, and KL terms to train its representation alongside the CAE’s.Cycle consistency checks both mixture reconstruction and closeness of latent representations before and after the cycle; noise-only inputs are trained to remain noise while their clean estimate is silence.Source: §2.2.2 · p. 3
2.3. Advantages of Self-supervision
- The stated advantages include training without paired data, adapting with mixtures from deployment environments, autonomous enhancement training, and reusing the pretrained CAE across speech environments.After CAE training, further training does not require clean speech examples; the authors also identify not transporting data between deployment locations as a security advantage.Source: §2.3 · p. 4
3. Experiments
The experiments evaluate the self-supervised speech enhancement model on indoor recordings and street noise, including comparisons across training noise percentages and against available baselines.
3.1. Experimental Setup
- The experiments evaluate magnitude spectrograms with PESQ and composite CSIG, CBAK, and COVL metrics; the CAE and MAE use convolutional autoencoders.The spectrograms use a 1024-sample window and DFT, a 256-sample hop, and a Hann window.Source: §3.1 · p. 4
3.2.1. EXPERIMENT 1: DAPS
- The DAPS experiment trains the CAE, trains the MAE, and tests on disjoint script and speaker segments, with speakers distributed evenly by gender.The dataset contains 100 clean and noisy recordings from 10 male and 10 female speakers reading five scripts.Source: §3.2.1 · p. 4
3.3. Results and Discussion
- On DAPS, all SSE versions outperform spectral subtraction across metrics, and higher pure-noise training percentages further improve performance.The table compares SSE models trained with 0%, 30%, and 50% pure-noise examples; higher scores are better.Source: Table 1 · p. 5, §3.3 · p. 5
- On BBC street-noise mixtures, higher pure-noise training percentages improve CBAK and PESQ, while gains are greater at higher mixture SNRs.Mixture metrics, rather than spectral subtraction, serve as the comparison baseline for these non-stationary street recordings.Source: Table 2 · p. 5, §3.3 · p. 5
3.2.2. EXPERIMENT 2: BBC SOUND EFFECTS
- The BBC experiment combines DAPS speech with London and Paris street noises at 5 and 10 dB SNR, using test speakers and utterances not encountered with training noise.The same noise files are used to construct the mixture and test segments.Source: §3.2.2 · pp. 4–5
4. Conclusion
The paper develops a self-supervised approach to single-channel speech enhancement: a clean-speech autoencoder learns a latent representation shared with a noisy-mixture autoencoder. Experiments demonstrate the approach in indoor ambient environments and with street noise.
4. Conclusion
- A clean-speech autoencoder learns a latent representation that is shared with an autoencoder trained on noisy speech mixtures.The shared representation supports mapping noisy mixtures to clean sounds without clean targets.Source: §4 · p. 5
- The resulting system maps noisy mixtures to clean sounds autonomously and without clean targets.Source: §4 · p. 5
- Experiments demonstrate the approach in ambient indoor environments and in the presence of street noises.Source: §4 · p. 5