PaperlayerView exact revision on arXiv

Source-linked AI summary

Supervised and Unsupervised Speech Enhancement Using Nonnegative Matrix Factorization

Nasser Mohammadiha, Paris Smaragdis, Arne Leijon

arXiv:1709.05362v1cs.SDcs.LG

TL;DR

The paper addresses monaural speech enhancement when supervised methods need noise-specific models in advance, while unsupervised methods often face difficult noise estimation. It develops Bayesian NMF systems combining BNMF with HMM-based noise classification and online noise-model learning. Experiments report substantially better performance than competing speech-enhancement methods, without requiring difficult a priori noise information.

  • Problem

    Monaural speech enhancement must reduce interference, but supervised methods require a model for each noise type trained in advance, while noise PSD estimation is difficult for non-stationary noise.

  • Method

    The paper uses Bayesian NMF with temporal priors, an HMM for simultaneous noise classification and enhancement, and online learning of a noise BNMF model from the noisy mixture.

  • Results

    The proposed BNMF-based systems outperform the Wiener filter and an MMSE STSA estimator with super-Gaussian priors by a high margin.

  • Takeaways & Limitations

    BNMF-based enhancement can operate without a priori knowledge of the underlying noise type while supporting both classified and online unsupervised systems.

Abstract

from arXiv · show

Reducing the interference noise in a monaural noisy speech signal has been a challenging task for many years. Compared to traditional unsupervised speech enhancement methods, e.g., Wiener filtering, supervised approaches, such as algorithms based on hidden Markov models (HMM), lead to higher-quality enhanced speech signals. However, the main practical difficulty of these approaches is that for each noise type a model is required to be trained a priori. In this paper, we investigate a new class of supervised speech denoising algorithms using nonnegative matrix factorization (NMF). We propose a novel speech enhancement method that is based on a Bayesian formulation of NMF (BNMF). To circumvent the mismatch problem between the training and testing stages, we propose two solutions. First, we use an HMM in combination with BNMF (BNMF-HMM) to derive a minimum mean square error (MMSE) estimator for the speech signal with no information about the underlying noise type. Second, we suggest a scheme to learn the required noise BNMF model online, which is then used to develop an unsupervised speech enhancement system. Extensive experiments are carried out to investigate the performance of the proposed methods under different conditions. Moreover, we compare the performance of the developed algorithms with state-of-the-art speech enhancement schemes using various objective measures. Our simulations show that the proposed BNMF-based methods outperform the competing algorithms substantially.

I. INTRODUCTION

Single-channel speech enhancement seeks cleaner speech for applications such as hearing aids and recognition, but conventional methods face noise-modeling and temporal-dependency challenges. This paper develops supervised and unsupervised NMF-based systems using Bayesian priors, HMM classification, and online noise-dictionary learning.

  • Single-channel enhancement estimates clean speech from noisy recordings to improve quality and reduce listening effort in applications including hearing aids and speech recognition.
  • Unsupervised methods require noise power spectral density estimation, which is challenging when background noise is non-stationary.
  • Supervised methods generally produce higher-quality enhanced speech by training signal models for specific noise types, but require suitable prior information.
  • Basic NMF can struggle when speech and noise basis matrices are similar, including in multitalker babble, while also ignoring important temporal dependencies.
  • The proposed supervised BNMF system uses temporal dependencies as Bayesian priors and combines BNMF with an HMM to classify environmental noise while enhancing speech.
  • The proposed unsupervised approach learns the noise basis matrix online from the noisy mixture using a causal dictionary-learning scheme.

II. REVIEW OF STATE-OF-THE-ART NMF-BASED SPEECH ENHANCEMENT

NMF-based speech enhancement combines learned speech and noise representations to estimate clean speech, with extensions addressing temporal dynamics, Bayesian regularization, and unknown or changing noise.

  • Basic NMF: NMF represents a spectrogram using nonnegative basis and coefficient matrices, typically by minimizing a divergence with optional regularization.Regularization can encourage sparsity or temporal dependence.
  • Supervised NMF enhancement: Supervised denoising learns speech and noise basis matrices in advance, concatenates them, and estimates clean speech using Wiener-type filtering.The noisy spectrogram is modeled under the additive approximation y = s+n.
  • Supervised NMF enhancement: NMF approaches can capture long-term signal levels automatically, avoiding the additional gain modeling required by some HMM- and codebook-based methods.
  • Temporal and unsupervised extensions: Prior work incorporates temporal structure through regularization, hidden Markov models, dynamical systems, or speech-pause-based online noise learning.These approaches include constrained NMF, nonnegative HMMs, causal filtering, and batch unsupervised algorithms.

III. SPEECH ENHANCEMENT USING BAYESIAN NMF

Bayesian NMF models spectrogram magnitudes with latent Poisson variables and Bayesian nonnegative factors. Variational inference estimates the posterior factors iteratively, providing the basis for the paper’s enhancement extensions.

  • Probabilistic NMF model: BNMF models each spectrogram entry through latent Poisson variables whose contributions sum to the observed magnitude.The observed spectrogram is treated as scaled and rounded integer-valued data in practice.
  • Bayesian formulation: In Bayesian NMF, the basis and coefficient matrices are random variables governed by gamma prior distributions with hyperparameters.The hyperparameters include gamma-distribution shape and scale parameters.
  • Variational inference: Variational Bayes approximates the posterior distributions of the factors by iteratively maximizing a lower bound under an independence assumption.The exact Bayesian inference is described as difficult, motivating the variational approximation.
  • Variational inference: The iterative BNMF updates alternate between posterior distributions for latent variables and gamma-distributed basis and coefficient factors until convergence.Latent-variable posteriors are multinomial, while basis and coefficient posteriors are gamma distributions.

A. BNMF-HMM for Simultaneous Noise Classification and Reduction

BNMF-HMM combines Bayesian NMF speech and noise models within an HMM whose states represent noise types. It classifies noise from the noisy input while estimating enhanced speech, avoiding the need to specify the noise type beforehand.

  • Model structure: Each HMM state corresponds to one trained noise type, while all states share a universal speaker-independent speech BNMF model.The model also uses a long-term SNR estimate for enhancement.
  • Model structure: Speech and noise basis matrices are concatenated under an additive noisy-signal model to define state-conditional BNMF output densities.The state-conditional likelihood integrates over the basis matrix and NMF coefficients.
  • Bayesian enhancement: Variational Bayes estimates posterior distributions for NMF coefficients while temporal dependencies construct informative coefficient priors.During enhancement, the basis posterior is held time-invariant and equal to its trained value.
  • Bayesian enhancement: The method derives a state-dependent MMSE estimate of speech DFT magnitudes and reconstructs the time-domain signal using noisy phase information.The estimator is obtained from posterior quantities calculated by variational Bayes.
  • Noise classification: Noise classification uses posterior HMM state probabilities computed from the noisy input and smoothed over time to reduce fluctuations.This classification is integrated with enhancement rather than requiring a separate noise-selection procedure.

B. Online Noise Basis Learning for BNMF

Online BNMF learns the noise basis matrix causally from the noisy mixture and then enhances speech using the adapted model. A sliding-window buffer balances adaptation speed, initialization quality, and overfitting risk.

  • Online adaptation: The online method learns the noise basis matrix directly from the noisy mixture and uses it with BNMF enhancement and a universal offline speech model.Unlike earlier clean-data dictionary learning, the proposed adaptation is causal and mixture-based.
  • Online adaptation: A slowly varying prior encourages the adapted noise basis to retain previous patterns unless the noise spectrogram changes substantially.The prior is formed from the preceding noise-dictionary estimate with a high shape parameter.
  • Online adaptation: A local sliding buffer stores recent noisy DFT frames, selects low-energy frames without a voice activity detector, and updates a main noise buffer.KL-NMF on the main buffer initializes the noise-basis posterior means before variational Bayes continues to convergence.
  • Trade-offs and demonstration: N1 = 50, N2 = 15, and q = 5 are used in experiments, with larger main buffers reducing overfitting risk but slowing basis adaptation.The approach is reported to be independent of the underlying SNR.
  • Trade-offs and demonstration: At 0 dB SNR, a learned single noise basis vector adapts to changes in two sinusoidal noise spectra, while adaptation introduces around 15 frames of latency.The latency is upper-bounded by 2N2−q frames, and prior-noise effects decay gradually through the sliding window.
  • Trade-offs and demonstration: Updating the basis every short-time frame was not robust enough to noise-type changes because the priors had to trade off adaptation against single-frame overfitting.The simulations found this trade-off difficult to achieve reliably.

C. Informative Priors for NMF Coefficients

The BNMF methods recursively update NMF-coefficient priors using temporal information, with a smoothing factor selected according to long-term SNR. Stronger smoothing is used under noisier conditions.

  • Recursive priors: Prior distributions for NMF coefficients are recursively updated by widening the previous posterior while retaining temporal dependencies and signal non-stationarity.The prior mean at time t combines the previous prior mean with the previous posterior expectation.
  • Recursive priors: The smoothing factor α controls how strongly prior means retain earlier values, making each prior dependent on previously observed noisy data.The recursive update links the current prior to all preceding observations through repeated application.
  • Prior parameterization: The prior shape parameter represents uncertainty and corresponds to the signal’s non-stationarity, with speech and noise allowed different parameter settings.The paper reports speech training values of φ^(s) = 3–5 but notes that an uninformative speech prior can perform better in some conditions.
  • SNR-dependent smoothing: Low SNRs favor strong smoothing (α →1) to reduce unwanted fluctuations, whereas high SNRs favor milder smoothing (α →0).The experiments therefore use an SNR-dependent smoothing factor represented by an empirical α-SNR curve.

IV. EXPERIMENTS AND RESULTS

The experiments compare supervised and unsupervised NMF-based enhancement under settings with and without training data for particular noise types. The evaluation uses TIMIT speech and a universal speaker-independent BNMF speech model.

  • Experimental design: The evaluation is divided into supervised experiments with training data for each noise type and unsupervised experiments lacking training data for some noise types.This setup tests the proposed systems under both matched and mismatched noise-model availability.
  • Experimental design: All signals are down-sampled to 16 kHz, and the core TIMIT test set contains 192 sentences for noise-reduction evaluation.The DFT uses 512-sample frames with 50% overlapped Hann windows.
  • Experimental design: All BNMF methods use a universal speaker-independent speech model containing 60 basis vectors learned from TIMIT training data.Long-term SNR estimates and the empirical α-SNR curve provide SNR-dependent prior smoothing.
  • Experimental design: The study also evaluates causal extensions of a previously noncausal whole-spectrogram NMF method for practical comparison.One evaluated extension is a supervised frame-by-frame approach using universal and noise-specific NHMM models.

A. Noise Reduction Using a-Priori Learned NMF Models

The supervised experiments compare BNMF-based enhancement variants with established NMF and STSA-GenGamma methods using BSS-Eval, PESQ, and SegSNR. BNMF variants achieved the strongest overall improvements, while classification allowed BNMF-HMM to approach oracle performance without prior noise-type specification.

  • Experimental setup: Five NMF-based algorithms were evaluated on factory, babble, and city traffic noises, including BNMF-HMM, General-model BNMF, oracle variants, and Oracle NHMM.The experiments used three noise types and compared supervised model configurations with different assumptions about noise-type knowledge.
  • Evaluation measures: BSS-Eval measured SDR, SIR, and SAR, with SDR and SIR reported as improvements and results averaged across noise types.SDR reflects overall enhanced-speech quality, while SIR and SAR correspond to noise reduction and speech distortion, respectively.
  • BSS-Eval results: Oracle BNMF achieved the best performance, closely followed by BNMF-HMM across all three BSS-Eval measures.The close results indicate that the HMM-based classifier successfully selected suitable noise models without requiring the noise type in advance.
  • BSS-Eval results: Except for Oracle ML, the NMF-based techniques outperformed STSA-GenGamma, while ML-NMF provided particularly poor noise reduction at high input SNRs.The reported findings were also supported by informal listening tests.
  • Method comparisons: The similar SDR values of Oracle NHMM and General-model BNMF concealed a trade-off: NHMM preserved speech with less noise suppression, whereas General-model BNMF showed the reverse pattern.Comparing BNMF-HMM with General-model BNMF also supported the advantage of using multiple small noise-dependent models.
  • Perceptual and segmental results: BNMF-based methods produced the highest SegSNR and PESQ improvements, whereas NHMM was less successful in improving perceptual speech quality.SegSNR was evaluated within the range [−10dB, 30dB].

B. Experiments with Unsupervised Noise Reduction

The unsupervised experiments evaluate online noise-model learning across six noise types and compare it with BNMF-HMM, Wiener filtering, STSA-GenGamma, and Online NHMM. Online BNMF achieved the strongest overall results, particularly when test noises were absent from the offline training models, while also running faster than BNMF-HMM.

  • Experimental setup: Six noise types were evaluated: factory, babble, city traffic, highway traffic, ocean, and hammer.Signals were concatenated before processing, and the BNMF-HMM comparison used only offline models for factory, babble, and city traffic.
  • Experimental setup: Online BNMF and Online NHMM learned 30 noise basis vectors, using a smaller dictionary than supervised methods to reduce overfitting from limited observations.The supervised methods used 100 basis vectors for each noise, whereas online methods adapted from a small number of observations.
  • Objective results: Online BNMF outperformed all other systems in BSS-Eval measures while introducing the least speech distortion and only moderate noise reduction.Wiener filtering and STSA-GenGamma achieved greater noise reduction but introduced more output artifacts.
  • Objective results: Online NHMM outperformed Wiener and STSA-GenGamma at low input SNRs for SDR but performed worst among competitors at high input SNRs.Online NHMM also provided the least noise suppression among the evaluated methods.
  • Perceptual and segmental results: Online BNMF outperformed BNMF-HMM by a large margin for SegSNR and PESQ, extending its advantage beyond the BSS-Eval comparison.The comparison is reported in Fig. 10.
  • Noise-dependent behavior: For the first three noise types, BNMF-HMM was marginally better, but Online BNMF significantly outperformed it for highway traffic, ocean, and hammer noises.The largest difference occurred for hammer noise because the pre-trained models represented it poorly.
  • Computational complexity: Online BNMF ran twice as fast as three-state BNMF-HMM and approximately five times faster than real time in the reported Matlab implementation.The timing used a 3.8 GHz Intel CPU and 2 GB RAM without parallel processing.

V. CONCLUSIONS

The paper concludes that Bayesian NMF improves speech enhancement by incorporating temporal dependencies and supports both classified and online noise-model strategies. Across the reported simulations, these BNMF-based systems surpassed competing enhancement methods without requiring difficult prior noise information.

  • Conclusions: The paper developed BNMF-based speech enhancement systems, including BNMF-HMM for simultaneous noise classification and enhancement and Online BNMF for online noise-model learning.Both systems operate without requiring the underlying noise type in advance.
  • Conclusions: ML-NMF with a universal speaker-independent speech model did not outperform state-of-the-art approaches, whereas temporal priors and optimal MMSE filters substantially improved NMF-based performance.The conclusion attributes the improvement to these modeling and estimation choices.
  • Conclusions: Online BNMF was faster than BNMF-HMM and superior when the underlying noise type was absent from the training data.The simulations also found that the suggested systems outperformed Wiener filtering and super-Gaussian-prior STSA estimation by a high margin.
Loading 1709.05362v1…