Source-linked AI summary
Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain
Simon Welker, Julius Richter, Timo Gerkmann
TL;DR
The paper addresses limitations of discriminative speech enhancement and prior diffusion formulations by developing score-based generative modeling in the complex STFT domain. It derives an SDE-based training task that avoids environmental-noise distribution assumptions and remains purely generative. The method improves SI-SDR over DiffuSE and CDiffuSE while producing more natural-sounding reconstructions with fewer artifacts.
Problem
Discriminative enhancement methods may generalize poorly to unseen acoustic conditions, while prior diffusion training can retain discriminative characteristics by estimating environmental noise.
Method
The paper trains an SDE-based score model with a complex-valued neural network directly on complex STFT coefficients, using noisy speech as conditioning rather than estimating environmental noise.
Results
About 5 dB and 3 dB SI-SDR improvements are reported over DiffuSE and CDiffuSE, respectively, with more natural-sounding reconstructions and fewer artifacts.
Takeaways & Limitations
Directly enhancing amplitude and phase in the complex STFT domain avoids phase retrieval and supports speech enhancement with fewer artifacts than previous methods.
Takeaways & Limitations
The forward process assumes circularity and a scaled identity covariance matrix, which would be a strong assumption for modeling real-world additive noise.
Abstract
from arXiv · showhide
Score-based generative models (SGMs) have recently shown impressive results for difficult generative tasks such as the unconditional and conditional generation of natural images and audio signals. In this work, we extend these models to the complex short-time Fourier transform (STFT) domain, proposing a novel training task for speech enhancement using a complex-valued deep neural network. We derive this training task within the formalism of stochastic differential equations (SDEs), thereby enabling the use of predictor-corrector samplers. We provide alternative formulations inspired by previous publications on using generative diffusion models for speech enhancement, avoiding the need for any prior assumptions on the noise distribution and making the training task purely generative which, as we show, results in improved enhancement performance.
1. Introduction
Speech enhancement seeks clean speech from noisy recordings, but discriminative methods may generalize poorly beyond finite training conditions. This work proposes SDE-based score modeling directly in the complex STFT domain to improve generative enhancement.
- Speech enhancement estimates clean speech from recordings affected by acoustic noise.
- Discriminative methods minimize distances between clean and noisy speech but may generalize poorly to unseen noise, reverberation, and signal-to-noise conditions.
- Generative models learn clean speech distributions as priors for inference from noisy inputs, including spectral and temporal structure.
- The proposed method uses an SDE-based score-generative process in the complex STFT domain, enhancing amplitude and phase directly without phase retrieval.
- Experiments report good enhancement performance and fewer speech distortions than the baseline method.
2. Background
Score-based generative models corrupt data with increasing noise and learn a score function to reverse that corruption. An SDE formulation provides a reverse-time process that can be solved for generative sampling.
- SGMs progressively corrupt training data with increasing noise and train a score model to reverse the corruption.The score estimates ∇x log p_data(x), and Langevin dynamics can draw samples after training.
- The SDE formalism defines the process through drift, diffusion, conceptual progression coordinate, infinitesimal steps, and a Wiener process.The SDE progression coordinate is not the audio or STFT time axis.
- Every forward SDE has a corresponding reverse SDE, whose score term is approximated by a learned time-dependent score model.
- Solving the reverse SDE with a solver procedure provides the basis for score-based generative modeling with SDEs.
3. Proposed method
The proposed method uses a continuous SDE to define a purely generative diffusion process for speech enhancement in the complex STFT domain. It combines a mean-reverting drift with increasing Gaussian corruption, trains by denoising score matching, and samples enhanced speech with predictor-corrector methods.
- 3.1. A stochastic process for speech enhancement: The forward process is Gaussian and supports efficient sampling because its perturbation kernel is characterized by a closed-form mean and variance.The stiffness parameter γ is selected empirically so the mean approaches noisy speech closely at the terminal time.
- 3.1. A stochastic process for speech enhancement: The proposed process combines an Ornstein-Uhlenbeck-style drift toward noisy speech with a variance-exploding diffusion term.The drift exponentially pulls the mean from clean speech toward y, while the diffusion increasingly corrupts features with Gaussian noise.
- 3.2. Data representation: The method works directly with clean and noisy signals represented as complex one-sided STFTs, retaining amplitude and phase information.The representation uses complex coefficients, with the frequency dimension determined by the DFT length and the time dimension by the number of frames.
- 3.2. Data representation: An amplitude transform compresses heavy-tailed STFT magnitudes and approximately maps them into [0, 1] before diffusion modeling.The paper chooses α = 0.5 and β = 3, then applies the forward and backward Gaussian processes in the transformed domain.
- 3.3. Training task: Denoising score matching trains the network to estimate the score from samples drawn at uniformly sampled diffusion times.For enhancement, sampling begins from a prior centered at the noisy spectrogram, reverses the SDE with diffusion sampling and annealed Langevin correction, and ends with inverse transformation and inverse STFT.
- 3.1. A stochastic process for speech enhancement: The SDE avoids estimating environmental noise directly, so the noisy signal serves only as conditioning while the model estimates artificially added Gaussian noise.This contrasts with CDiffuSE, whose objective includes a term that trains the network to remove environmental noise at each step.
- 3.4. Speech enhancement procedure: Each U-Net encoder/decoder block processes real and imaginary components separately for activations and normalization while using natural complex algebra for complex layers.The architecture is designed for complex-valued STFT-domain processing.
4. Experimental setup
The experiments use VoiceBank-DEMAND with complex STFT inputs, a modified DCUNet-based complex U-Net, and SI-SDR, SI-SIR, and SI-SAR evaluation against noisy speech and baseline systems.
- 4.1. Model: The model is a modified Deep Complex U-Net with timestep embeddings inserted into all encoder and decoder blocks.The architecture contains 3.56M parameters overall and encodes t using 128 random Fourier feature embeddings passed through complex-valued layers.
- 4.2. Dataset: Training and testing use the standardized VoiceBank-DEMAND dataset with clean and noisy audio normalized by the maximum absolute clean-speech value.The dataset follows the setup used by the DiffuSE baseline.
- 4.1. Model: The modified DCUNet parameter table specifies encoder and decoder input/output channels, kernel sizes, strides, dilations, and frequency-time axis tuples.These quantities define the architecture's layer configurations.
- 4.2. Dataset: Each input is converted to a complex one-sided STFT using DFT length F = 512, hop length 128, and a periodic Hann window.The resulting spectrograms are randomly cropped to 256 STFT time frames during each epoch.
- 4.3. Training: The network is trained for 325 epochs with Adam, learning rate 10^-4, batch size 32, and an exponential moving average used for sampling.The SDE uses γ = 1.5, σmin = 0.05, σmax = 0.5, and tε = 0.03; two baseline models are also trained.
- 4.4. Evaluation: Evaluation reports SI-SDR, SI-SIR, and SI-SAR relative to noisy speech and the baseline method.PESQ is omitted because the cited standard requires at least 3.2 s of active speech in the reference.
5. Results
SGMSE outperforms DiffuSE and CDiffuSE on SI-SDR while balancing interference suppression and artifact reduction. Spectrogram comparisons indicate fewer voice distortions and better preservation of speech structure, at some expense of noise reduction.
- Evaluation setup: Table 2 reports average performance against DiffuSE and the noisy mixture, including raw outputs and m = (0.8x̂ + 0.2y) values.The table reports mean test-set results with 95% confidence intervals; the m values follow prior work.
- Quantitative results: 4.6 dB and 3.0 dB SI-SDR improvements over DiffuSE and CDiffuSE, respectively, are reported for SGMSE raw output.The comparison uses the raw model output rather than the interpolated m output.
- Quantitative results: SGMSE achieves the most favorable balance between increasing SI-SAR and decreasing SI-SIR, resulting in the best SI-SDR.The three compared methods share this opposing SI-SAR/SI-SIR trend.
- Qualitative comparison: SGMSE exhibits fewer voice distortions than CDiffuSE at some expense of noise reduction.This comparison is drawn from highlighted regions in an example VoiceBank-DEMAND spectrogram.
- Qualitative comparison: Both SGMSE and CDiffuSE remove environmental noise, while SGMSE better preserves high-frequency fricatives and natural formant structure.The qualitative comparison uses power spectrograms of clean, noisy, and estimated speech.
6. Conclusion
The paper introduces an SDE-based score-generative approach for speech enhancement in the complex STFT domain. It reports more natural-sounding reconstructions and fewer artifacts than prior methods, with improved SI-SDR but some reduced noise removal.
- Contribution: The work designs a novel SDE-based stochastic process for score-based speech enhancement in the complex STFT domain.The authors identify it as the first application of SGMs in the complex time-frequency domain.
- Findings: SGMSE produces more natural-sounding reconstructions with fewer artifacts than previous methods, at some expense of noise removal.The conclusion attributes the combined trade-off to improved SI-SDR relative to DiffuSE and CDiffuSE.
- Findings: The reported SI-SDR improvement is about 5 dB over DiffuSE and 3 dB over CDiffuSE.These are approximate conclusion-level comparisons with the prior methods.
- Future work: Further investigation of other SDEs, data representations, and DNN architectures is identified as a promising research direction.The conclusion also mentions an extended journal article in preparation.