PaperlayerView exact revision on arXiv

Source-linked AI summary

Speech Enhancement and Dereverberation with Diffusion-based Generative Models

Julius Richter, Simon Welker, Jean-Marie Lemercier, Bunlong Lay, Timo Gerkmann

arXiv:2208.05830v3eess.AScs.LGcs.SD

TL;DR

Speech enhancement must handle varied acoustic conditions while avoiding distortions from discriminative mappings. This work develops a diffusion-based generative framework with a task-adapted stochastic process and improved network architecture, achieving competitive matched-condition performance, strong mismatched-condition generalization, and dereverberation gains.

  • Problem

    Speech enhancement methods must recover clean speech across diverse acoustic conditions, but training data cannot cover all conditions and some discriminative approaches introduce unpleasant distortions.

  • Method

    The method uses a diffusion-based generative model in the complex STFT domain, with a drift term directing the forward process toward noisy or reverberant speech and an adapted network architecture.

  • Results

    The proposed method performs on par with competitive discriminative methods under matched conditions, outperforms all baselines across metrics under mismatched conditions, and significantly outperforms discriminative baselines for dereverberation.

  • Takeaways & Limitations

    A single diffusion-based framework can train individual models for different distortion types while generalizing strongly across speech-enhancement corpora.

  • Takeaways & Limitations

    The forward process retains a slight mismatch with the reverse process; increasing stiffness to reduce it may make reverse sampling unstable.

Abstract

from arXiv · show

In this work, we build upon our previous publication and use diffusion-based generative models for speech enhancement. We present a detailed overview of the diffusion process that is based on a stochastic differential equation and delve into an extensive theoretical examination of its implications. Opposed to usual conditional generation tasks, we do not start the reverse process from pure Gaussian noise but from a mixture of noisy speech and Gaussian noise. This matches our forward process which moves from clean speech to noisy speech by including a drift term. We show that this procedure enables using only 30 diffusion steps to generate high-quality clean speech estimates. By adapting the network architecture, we are able to significantly improve the speech enhancement performance, indicating that the network, rather than the formalism, was the main limitation of our original approach. In an extensive cross-dataset evaluation, we show that the improved method can compete with recent discriminative models and achieves better generalization when evaluating on a different corpus than used for training. We complement the results with an instrumental evaluation using real-world noisy recordings and a listening experiment, in which our proposed method is rated best. Examining different sampler configurations for solving the reverse process allows us to balance the performance and computational speed of the proposed method. Moreover, we show that the proposed method is also suitable for dereverberation and thus not limited to additive background noise removal. Code and audio examples are available online, see https://github.com/sp-uhh/sgmse.

I. INTRODUCTION

The paper develops diffusion-based generative models for speech enhancement and extends an earlier approach with a stronger network, broader theory, and expanded evaluation. It addresses limitations of discriminative enhancement methods through generative modeling of clean speech and tests generalization across corpora and recording conditions.

  • I. INTRODUCTION: Discriminative speech-enhancement models directly map noisy speech to clean targets but may generalize poorly and produce speech distortions.Training data cannot cover all acoustic conditions, and some methods produce unpleasant distortions that can outweigh noise-reduction benefits.
  • I. INTRODUCTION: Generative approaches learn a prior over clean speech and use it to infer clean signals from noisy or reverberant inputs.The learned prior captures inherent spectral and temporal speech structure.
  • I. INTRODUCTION: Diffusion models gradually transform data into noise and train a neural network to iteratively invert that process through score-based modeling.The SDE formulation connects the diffusion process with score matching and defines a reverse process based on the score function.
  • I. INTRODUCTION: The method remains generative in objective while using clean/noisy speech pairs to learn conditional generation for enhancement.The paper distinguishes supervised training from the generative nature of learning a clean-speech prior.
  • I. INTRODUCTION: The proposed work improves a previous complex-STFT diffusion model with an image-processing network architecture and extends its theoretical and empirical analysis.The authors investigate sampler configurations and add cross-dataset, real-world instrumental, and listening evaluations.

A. Data representation

The method represents speech in the complex STFT domain and uses a conditional SDE whose drift moves clean speech toward the noisy or reverberant observation while diffusion adds Gaussian noise. A learned score model then solves the reverse SDE to reconstruct clean speech from a corrupted initialization.

  • A. Data representation: The model estimates clean real and imaginary spectrogram components from noisy or reverberant speech in the complex-valued STFT domain.Complex coefficients preserve a representation in which additive Gaussian noise matches the denoising signal model.
  • A. Data representation: Magnitude-domain diffusion is avoided because additive Gaussian noise can produce physically undefined negative amplitudes.The complex spectral representation therefore better matches the assumed additive-noise model.
  • A. Data representation: The complex spectrogram is compressed and scaled to compensate for heavy-tailed STFT amplitudes and emphasize lower-energy frequency components.The compression exponent can bring out components such as unvoiced fricatives, while scaling roughly normalizes amplitudes to [0, 1].
  • B. Stochastic Process: The forward SDE gradually corrupts clean speech with Wiener-process noise while its y-dependent drift moves the process mean toward noisy or reverberant speech.The drift is the task-adaptation mechanism, while the diffusion coefficient controls injected Gaussian white noise.
  • B. Stochastic Process: The reverse SDE uses a neural score model conditioned on the current state, corrupted speech, and continuous time to generate clean speech iteratively.Inference starts from a strongly corrupted version of the noisy speech and proceeds from t = T to t = 0.

C. Training objective

The score model is trained by sampling arbitrary diffusion states from a Gaussian perturbation kernel and matching its output to the kernel score with an unweighted L2 objective.

  • The perturbation kernel enables direct sampling of x_t at arbitrary t given clean speech x_0 and noisy speech y.The SDE’s Gaussian-process structure provides closed-form mean and variance expressions for this sampling.
  • Training samples random t, a dataset pair (x_0, y), complex Gaussian noise z, and x_t from the perturbation kernel.The time is sampled uniformly from [t_ε, T] to avoid the numerical instability near t = 0.
  • The model is optimized with an unweighted L2 loss between its output and the perturbation-kernel score.The expectation is approximated by sampling the random variables at each training step and applying stochastic gradient descent.
  • Because the mean term cancels, y enters the loss only through the score model input rather than through explicit environmental-noise estimation.The model therefore is not tasked with estimating any portion of the environmental noise directly.

D. Interpretation and limitations

The forward process forms a continuum from clean speech to strongly corrupted noisy speech, while the reverse model reconstructs clean speech from a distribution centered near the noisy input. A finite-time mismatch between forward and reverse terminal distributions creates a trade-off between matching them and maintaining reverse-process stability.

  • Interpretation: The forward process evolves distributions from essentially clean speech at t = 0 to strongly corrupted noisy speech at t = T.The diffusion coefficient starts with σ(0)^2 = 0 and increases exponentially, masking sample-specific noisy-speech characteristics at T.
  • Interpretation: The reverse process learns this distribution family starting from a high-variance state and produces a low-variance estimate of clean speech.The forward process supplies a strong clean-speech prior through the learned reverse process and a weakly informative noisy-speech prior at its terminal end.
  • Limitations: The forward terminal distribution is slightly mismatched with the reverse process’s initial distribution because µ(x_0, y, T) ≠ y at finite T.The mismatch can be reduced by increasing γ or σ_max, which smooths the respective density functions.
  • Limitations: Increasing γ may destabilize the reverse process by moving the mean rapidly toward y and leaving environmental-noise removal concentrated in the final steps.The trade-off is illustrated by the SNR evolution of the process mean for different γ values.

III. NUMERICAL SDE SOLVERS

The reverse SDE is discretized over N time intervals and solved with predictor-corrector samplers or a deterministic probability-flow ODE. These alternatives provide numerical routes for generating enhanced speech from noisy or reverberant mixtures.

  • Discretization: The continuous reverse process is approximated by partitioning [0, T] into N equal intervals of width Δt = T/N.The resulting discrete states run from x_T through x_0, and Euler-Maruyama is a common single-step solver.
  • Predictor-corrector samplers: Predictor-corrector samplers combine a single-step reverse-SDE solver with numerical optimization such as annealed Langevin dynamics.The predictor iterates through the reverse SDE, while the corrector follows each predictor iteration.
  • Probability-flow ODE: The probability-flow ODE provides a deterministic process with the same marginal density p_t(x_t) as the stochastic reverse SDE.Enhancement is performed by solving this ODE, and the paper evaluates this class of solvers.
  • Probability-flow ODE: The paper specifically uses a fourth-order Runge-Kutta method with a fifth-order error estimator for the ODE.

IV. NETWORK ARCHITECTURE

The score model adapts NCSN++ to complex spectrograms by representing real and imaginary components as separate channels within a time-conditioned multi-resolution architecture. Progressive input and output growth supplements the U-Net’s feature-processing paths.

  • Complex-spectrogram adaptation: NCSN++ estimates both real and imaginary score components by treating the complex spectrogram input as separate channels.This allows the model to generate complex spectrograms of clean speech.
  • Multi-resolution architecture: The architecture uses a multi-resolution U-Net with skip connections, progressive growing, residual blocks, and a bottleneck layer.The network’s upsampling, downsampling, and bottleneck stages contain multiple residual blocks in series.
  • Time conditioning: Fourier embeddings map the diffusion time t to an M-dimensional vector integrated into every residual block.This supplies the network with information about the current progression of the diffusion process.
  • Progressive growing: Progressive growing provides downsampled input to feature maps in the contracting path and progressively growing output in the expansive path.Shared weights are used for the downsampling operation at each resolution, and the design was successful in stabilizing high-resolution image generation.

V. EXPERIMENTS

The experiments evaluate speech enhancement and dereverberation using two speech-enhancement datasets, a simulated reverberation dataset, and complementary objective metrics.

  • Datasets: Two datasets, WSJ0-CHiME3 and VB-DMD, enable cross-dataset testing on unseen noise types and recording conditions.WSJ0-REVERB is additionally created for training and evaluating dereverberation.
  • Datasets: WSJ0-CHiME3 mixes WSJ0 speech with CHiME3 noise at uniformly sampled SNRs from 0 to 20 dB across all splits.
  • Datasets: VB-DMD uses eight real-recorded and two artificial DEMAND noises, with training SNRs of 0, 5, 10, and 15 dB and different test SNRs.Test mixtures use SNRs of 2.5, 7.5, 12.5, and 17.5 dB.
  • Datasets: WSJ0-REVERB convolves WSJ0 speech with simulated room impulse responses and uses matched anechoic targets for dereverberation.Room reverberation is sampled with T60 between 0.4 and 1.0 seconds.
  • Metrics: Evaluation combines full-reference metrics for processed-versus-clean comparisons with non-intrusive metrics for recordings without clean references.The metric suite includes POLQA, PESQ, ESTOI, SI-SDR/SI-SIR/SI-SAR, DNSMOS, SIG/BAK/OVRL, and WVMOS.
  • Metrics: PESQ ranges from 1 to 4.5, ESTOI from 0 to 1, and SI-SDR, SI-SIR, and SI-SAR are measured in dB with higher values indicating better performance.

C. Listening Experiment

The study supplements instrumental evaluation with a MUSHRA listening experiment and describes the signal representation and training configuration used for the proposed method.

  • Listening Experiment: Ten participants rated the overall quality of twelve randomly sampled WSJ0-CHiME3 test examples reconstructed by the compared algorithms.The experiment used the webMUSHRA framework.
  • Implementation: Audio sampled at 16 kHz is converted to a complex-valued STFT with 256 frequency bins and 256 randomly selected time frames.The transform uses a 510-sample window, 128-sample hop length, and periodic Hann window.
  • Implementation: The stochastic process uses σmin = 0.05, σmax = 0.5, and γ = 1.5, selected through grid-search hyperparameter optimization.
  • Implementation: Training runs for 160 epochs with distributed data parallelism, Adam at learning rate 10^-4, effective batch size 32, and exponential moving-average weights.Training uses four Quadro RTX 6000 GPUs and takes about one day.

E. Sampler settings

Sampler experiments examine performance–computation trade-offs across sampler type, reverse-step count, and corrector step size, using VB-DMD for hyperparameter selection.

  • Sampler type: One corrector step improves the PC sampler, while two steps add no further performance; the selected configuration has RTF 1.77 because it doubles function evaluations.The ODE sampler can reach RTF 0.46 with 14 average function evaluations, although PC performs better in both metrics.
  • Number of reverse steps N: At N = 30 reverse steps, PESQ and SI-SDR show no further performance increase, while SI-SDR stagnates earlier than PESQ.
  • Step size in corrector: Choosing r = 0.5 maximizes PESQ while retaining a good SI-SDR value, reflecting a compromise between the two metrics.
  • Baselines: The proposed method is compared with four generative and four discriminative baselines, with selected VB-DMD models obtained pretrained from their authors.

VI. RESULTS

SGMSE+ improves speech-enhancement performance across matched and mismatched conditions, competing with discriminative methods while generalizing better across datasets. Listening tests also rate it highest, although very low-SNR inputs can produce vocalizing artifacts.

  • SGMSE+ outperforms all other generative methods across every metric in the matched WSJ0-CHiME3 condition.
  • 0.75 POLQA and 0.68 PESQ improvements are reported for SGMSE+ over the previous SGMSE model.
  • SGMSE+ keeps up with discriminative methods and surpasses them in POLQA, SI-SIR, and DNSMOS.
  • SGMSE+ outperforms all other methods in every metric under mismatched training conditions, showing strong cross-dataset generalization.
  • The POLQA distributions for SGMSE+ remain relatively similar between matched and mismatched conditions, unlike those of Conv-TasNet and MetricGAN+.
  • Ten participants rated SGMSE+ highest on average in the MUSHRA experiment, and the method remained fairly robust under mismatched training.
  • At very low input SNRs, SGMSE+ occasionally produces poorly articulated vocalizing artifacts or speech-like sounds in regions without original speech.

B. Dereverberation

SGMSE+ performs strongly on single-channel dereverberation, particularly under the challenging low-DRR conditions evaluated. Its improved network architecture yields a larger benefit for dereverberation than for additive-noise enhancement.

  • B. Dereverberation: SGMSE+ performs particularly well on instrumental dereverberation metrics compared with all evaluated baseline models.
  • B. Dereverberation: At an average input DRR of -9 dB, discriminative approaches distort the target while SGMSE+ reconstructs high-quality speech in most cases.
  • B. Dereverberation: The improved network architecture benefits SGMSE+ more substantially for dereverberation than for additive background-noise removal.
  • B. Dereverberation: SGMSE+ does not produce the vocalized artifacts observed in low-SNR speech-enhancement experiments when applied to single-channel dereverberation.

C. Evaluation on real data

On 300 real-world noisy recordings from the DNS Challenge 2020 test set, the proposed method was evaluated without clean references using non-intrusive speech-quality metrics. The authors report superior performance, while noting that these metrics may miss vocalizing artifacts or phonetic confusions.

  • 300 DNS Challenge 2020 test-set files were used for real-world evaluation without clean speech references.Evaluation therefore relied on non-intrusive perceptual speech-quality metrics.
  • The proposed method performed superior to the evaluated models on non-intrusive real-world speech-quality metrics.The authors hypothesize that training to generate clean speech contributes to this outcome.
  • Non-intrusive metrics may not capture vocalizing artifacts or phonetic confusions in the enhanced estimates.These artifacts were observed in very adverse conditions elsewhere in the evaluation.
  • The framework was also evaluated for speech dereverberation, extending its use beyond additive background-noise removal.Real reverberant examples from the MC-WSJ-AV dataset were included among the listening examples.
  • Different reverse-process sampling strategies provide a performance–computational-speed trade-off at test time.Future sampling techniques could reduce diffusion steps and computational complexity further.
Loading 2208.05830v3…