Source-linked AI summary
Statistical Speech Enhancement Based on Probabilistic Integration of Variational Autoencoder and Non-Negative Matrix Factorization
Yoshiaki Bando, Masato Mimura, Katsutoshi Itoyama, Kazuyoshi Yoshii, Tatsuya Kawahara
TL;DR
Supervised speech enhancement can deteriorate in unknown noisy environments, while conventional statistical models can distort speech by relying on unrealistic linear or sparsity assumptions. The paper combines a VAE speech prior with an NMF noise model and estimates clean speech through posterior inference. It reports that VAE-NMF outperformed the conventional supervised DNN-based method in unseen noisy environments.
Problem
Supervised DNN enhancement can deteriorate in unknown noisy environments, while conventional statistical methods can cause considerable distortion because speech spectrograms do not satisfy their linearity or sparsity assumptions.
Source: §1 · p. 1Method
VAE-NMF combines a clean-speech VAE prior with an NMF noise model and uses MCMC-based Bayesian inference to estimate clean speech while adapting to noisy environments.
Source: §1 · p. 1, §3 · p. 2, §3.4 · p. 3Results
VAE-NMF outperformed the conventional supervised DNN-based method in unseen noisy environments.
Source: §5 · p. 4Takeaways & Limitations
The experiments support VAE-NMF as a semi-supervised speech-enhancement approach for unseen noisy environments.
Source: §5 · p. 4In DEMAND's cafe condition, conversational background speech was enhanced when target-speech power was relatively small because speech was estimated independently at each frame.
Source: §4.2 · p. 4
Abstract
from arXiv · showhide
This paper presents a statistical method of single-channel speech enhancement that uses a variational autoencoder (VAE) as a prior distribution on clean speech. A standard approach to speech enhancement is to train a deep neural network (DNN) to take noisy speech as input and output clean speech. Although this supervised approach requires a very large amount of pair data for training, it is not robust against unknown environments. Another approach is to use non-negative matrix factorization (NMF) based on basis spectra trained on clean speech in advance and those adapted to noise on the fly. This semi-supervised approach, however, causes considerable signal distortion in enhanced speech due to the unrealistic assumption that speech spectrograms are linear combinations of the basis spectra. Replacing the poor linear generative model of clean speech in NMF with a VAE---a powerful nonlinear deep generative model---trained on clean speech, we formulate a unified probabilistic generative model of noisy speech. Given noisy speech as observed data, we can sample clean speech from its posterior distribution. The proposed method outperformed the conventional DNN-based method in unseen noisy environments.
1. INTRODUCTION
Supervised DNN enhancement needs paired clean and noisy training data and can deteriorate in unknown environments, while conventional statistical models can distort speech through unrealistic assumptions. The paper combines a VAE speech model with an NMF noise model in a unified probabilistic framework.
1. INTRODUCTION
- Supervised DNN methods require many paired clean and noisy signals and often deteriorate in unknown noisy environments.Examples include mapping noisy spectrograms to clean spectrograms or predicting ideal ratio masks.Source: §1 · p. 1
- Conventional statistical methods rely on linear representation or sparsity assumptions that speech spectrograms do not satisfy in reality, resulting in considerable signal distortion.The introduction describes NMF and RPCA as examples of these methods.Source: §1 · p. 1
- VAEs learn probability distributions over complex data and provide a principled way to infer latent representations of observed and new data.The paper contrasts this with GANs, which are described as generally generating more realistic data.Source: §1 · p. 1
- The proposed model combines a VAE-based speech prior with an NMF-based noise model and uses Bayesian inference to adapt their latent representations and noise parameters to observations.Inference uses an MCMC sampler initialized by the VAE encoder, allowing adaptation to unseen speech and noise spectra.Source: §1 · p. 1
2. RELATED WORK
The related work reviews supervised DNN speech enhancement and introduces VAEs as probabilistic generative models. It describes their latent-variable formulation and variational training objective.
2. RELATED WORK
- The section reviews DNN-based enhancement and introduces the VAE as a framework for learning a dataset probability distribution.The VAE section develops its latent-variable formulation and training objective.Source: §2 · p. 2, §2.2 · p. 2
2.1. DNN-based speech enhancement
- DNN enhancement commonly trains on simulated noisy speech with clean speech as the target; related methods also estimate masks or use speech and noise networks.The review notes that these methods generally use speech and noise training data, while a speech-only approach may remain constrained by an NMF model.Source: §2.1 · p. 2
2.2. Variational autoencoder
- A VAE models a dataset with Gaussian latent variables and a decoder that generates each sample conditionally from its latent representation.Its encoder approximates the latent posterior, and the model is trained to maximize a variational lower bound on log marginal likelihood.Source: §2.2 · p. 2
3. STATISTICAL SPEECH ENHANCEMENT BASED ON COMBINATION OF VAE AND NMF
VAE-NMF combines a VAE-based speech model with an NMF-based noise model to represent noisy speech probabilistically. Its stages define the speech prior, mixture model, VAE pre-training, Bayesian inference, and speech reconstruction.
3.1. VAE-based speech model
- Frame-wise latent variables Z represent speech characteristics, while a VAE-trained nonlinear function defines the speech spectrum variance.The model places a standard Gaussian prior on Z and assumes zero-mean complex Gaussian speech spectra.Source: §3.1 · pp. 2–3
3.2. Generative model of mixture signals
- The mixture model represents each observed complex spectrogram as speech plus noise, using a VAE prior for speech and a low-rank NMF prior for noise.Its phase-independent exponential likelihood on the power spectrogram corresponds to minimizing Itakura-Saito divergence.Source: §3.2 · p. 3
3.3. Pre-training of VAE-based speech model
- VAE pre-training seeks speech-decoder parameters that maximize clean-speech marginal likelihood, approximated with a variational mean-field posterior and optimized by SGD.The approximation ignores the speech spectrogram's phase term.Source: §3.3 · p. 3
3.4. Bayesian inference of VAE-NMF
- Bayesian inference approximates p(W, H, Z|X) with MCMC, sampling W and H from variational conditional posteriors and updating Z with a Metropolis method.The Z conditional posterior is hard to derive analytically, so the method proposes frame-wise latent samples using a Gaussian proposal.Source: §3.4 · p. 3
3.5. Reconstruction of complex speech spectrogram
- The enhanced speech is reconstructed with Wiener filtering by maximizing the conditional posterior p(S|X, W, H, Z).The method uses the mean values of sampled W, H, and Z in the reconstruction equation.Source: §3.5 · p. 3
4. EXPERIMENTAL EVALUATION
The evaluation compared VAE-NMF with DNN-IRM and RPCA on speech mixed with noise captured in actual environments. VAE-NMF performed especially well on unseen DEMAND noise, with a weaker result for conversational-speech noise in the cafe condition.
4.1. Experimental settings
- Evaluation used CHiME-3 development speech and DEMAND mixes in four environments each, with average/input SNRs of 5.8/5.0 dB, scored by SDR.The CHiME-3 development set had 410 simulated noisy utterances per environment; the DEMAND evaluation used 20 per environment.Source: §4.1 · p. 4, §4 · p. 4
- The VAE speech prior was trained on about 15 hours of WSJ-0 clean speech; DNN-IRM was trained using CHiME-3 noisy data with noise recorded in the evaluated environments.The VAE latent dimension was 10, and DNN-IRM used five hidden layers to predict ideal ratio masks.Source: §4.1 · p. 4
4.2. Experimental results
- On DEMAND, VAE-NMF outperformed all methods in every condition; on CHiME-3 it beat RPCA everywhere and DNN-IRM on BUS, while DNN-IRM had the highest average SDR.DNN-IRM's performance deteriorated significantly on DEMAND because its CHiME-3 training did not include the DEMAND noise.Source: Table 2 · p. 4, Table 1 · p. 4, §4.2 · p. 4
- VAE-NMF had lower SDR in DEMAND's cafe condition, where conversational background speech was enhanced during frames with relatively weak target speech.The authors suggest a VAE-based speech model with time dependencies, such as a variational recurrent neural network, as a possible extension.Source: §4.2 · p. 4
5. CONCLUSION
VAE-NMF combines a VAE speech model with an NMF noise model, using the speech model as a prior and MCMC to estimate clean speech while adapting the noise model. The authors report that it outperformed a conventional supervised DNN method in unseen noisy environments.
5. CONCLUSION
- VAE-NMF uses a VAE speech prior and an NMF noise model, estimating clean speech with an MCMC sampler while adapting the noise model to noisy environments.Only the speech model is trained in advance, using clean speech.Source: §5 · p. 4
- VAE-NMF outperformed the conventional supervised DNN-based method in unseen noisy environments.Source: §5 · p. 4
- The authors identify multichannel enhancement using a VAE speech model and phase-aware linear mixing model, and GAN-based speech-model training, as future directions.Source: §5 · p. 4