Source-linked AI summary
Conditional Diffusion Probabilistic Model for Speech Enhancement
Yen-Ju Lu, Zhong-Qiu Wang, Shinji Watanabe, Alexander Richard, Cheng Yu, Yu Tsao
TL;DR
Speech enhancement remains challenging, and vanilla diffusion models assume Gaussian noise despite realistic enhancement noise often being non-Gaussian. The paper proposes CDiffuSE, which conditions diffusion and reverse processes on noisy speech, and reports stronger generative-model performance and greater robustness to unseen noise characteristics.
Problem
Speech enhancement remains challenging, while vanilla diffusion models assume Gaussian noise that does not match realistic non-Gaussian noise conditions.
Method
CDiffuSE incorporates observed noisy speech into conditional diffusion and reverse processes, adapting the model to real noise characteristics.
Results
CDiffuSE achieves state-of-the-art performance among generative speech-enhancement approaches and maintains strong performance when regression-based competitors collapse.
Takeaways & Limitations
CDiffuSE generalizes better to speech data with noise characteristics not observed during training than non-generative approaches.
Takeaways & Limitations
Deriving the conditional forward process while preserving the original diffusion process is difficult because noisy speech depends on clean speech through y = x0 + n.
Abstract
from arXiv · showhide
Speech enhancement is a critical component of many user-oriented audio applications, yet current systems still suffer from distorted and unnatural outputs. While generative models have shown strong potential in speech synthesis, they are still lagging behind in speech enhancement. This work leverages recent advances in diffusion probabilistic models, and proposes a novel speech enhancement algorithm that incorporates characteristics of the observed noisy speech signal into the diffusion and reverse processes. More specifically, we propose a generalized formulation of the diffusion probabilistic model named conditional diffusion probabilistic model that, in its reverse process, can adapt to non-Gaussian real noises in the estimated speech signal. In our experiments, we demonstrate strong performance of the proposed approach compared to representative generative models, and investigate the generalization capability of our models to other datasets with noise characteristics unseen during training.
1. INTRODUCTION
Speech enhancement remains difficult despite progress from discriminative and generative deep-learning approaches. This work investigates diffusion probabilistic models for enhancement, addressing their mismatch with realistic non-Gaussian noise by conditioning the model on observed noisy speech.
- Speech enhancement supports immersive audio, robust speech recognition, assistive hearing, and speaker recognition but remains an unsolved audio-processing problem.
- Deep-learning enhancement methods include discriminative models trained against clean speech and generative models that model the clean-speech distribution.
- Diffusion probabilistic models progressively add Gaussian noise to clean data and reverse the process by predicting and removing that noise.
- Vanilla diffusion assumes Gaussian noise, whereas realistic speech-enhancement conditions commonly involve non-Gaussian noise that violates this assumption.
- The proposed conditional formulation incorporates observed noisy data into diffusion and reverse processes and derives corresponding training objectives.
2. DIFFUSION PROBABILISTIC MODEL
The diffusion probabilistic model uses a forward Markov chain to transform clean speech into isotropic Gaussian noise and a learned reverse chain to reconstruct clean speech. Training uses an evidence lower bound because the reverse-process marginal likelihood is intractable.
- Diffusion Process: A T-step diffusion model contains a forward process indexed from t=0 to T and a reverse process indexed from T back to 0.
- Diffusion Process: The forward process starts from clean speech x0 and defines same-length variables xt through a fixed Markov chain.
- Diffusion Process: Each forward step adds Gaussian noise controlled by βt, gradually converting clean speech into xT with isotropic Gaussian distribution N(0, I).
- Diffusion Process: The marginal forward distribution q(xt|x0) can be derived by substituting the stepwise Gaussian model and marginalizing intermediate variables.
- Reverse Process: The reverse process transforms xT sampled from N(0, I) back to x0 using a learnable Markov distribution pθ.
- Reverse Process: Because the reverse marginal likelihood is intractable, training uses an evidence lower bound objective.
- Reverse Process: The model estimates Gaussian noise with ϵθ, then uses the estimate to define the reverse-step mean while keeping the variance fixed.
3. CONDITIONAL DIFFUSION PROBABILISTIC MODEL
The conditional diffusion probabilistic model incorporates noisy speech into both diffusion and reverse processes. Its interpolation-based formulation and conditional reverse process provide a theoretically justified way to estimate Gaussian and non-Gaussian noise while recovering clean speech.
- 3.1. Conditional Diffusion Processes: The conditional diffusion process combines clean speech x0 and noisy speech y through an interpolation weight mt that increases from 0 to approximately 1.This shifts the mean of xt from clean speech toward noisy speech across diffusion steps.
- 3.1. Conditional Diffusion Processes: The conditional formulation generalizes the original diffusion model and becomes equivalent to it under the specified variance condition.The authors use this analytical relationship to establish a connection with the original diffusion process.
- 3.2. Conditional Reverse Processes: The reverse process conditions on xt and y, predicting xt−1 with a mean formed from xt, y, and estimated noise.The estimated noise model represents a combination of Gaussian and non-Gaussian noise.
- 3.3. Coefficient Estimation by Optimizing ELBO: The ELBO objective determines the reverse-process variance and coefficients used to combine xt, y, and the estimated noise.The derivation matches the conditional reverse distribution to the corresponding conditional diffusion distribution.
- 3.4. CDiffuSE Training and Sampling Algorithm: During training, the model samples clean/noisy speech pairs and Gaussian noise, constructs xt with the interpolation rule, and optimizes the noise-estimation objective.The sampling algorithm then iteratively predicts and subtracts combined noise to return the clean signal x0.
- 3.4. CDiffuSE Training and Sampling Algorithm: CDiffuSE removes the assumption that real noise is Gaussian, avoiding the mismatch between diffusion and reverse processes present in the earlier supportive reverse process.The proposed method instead incorporates the observed noisy signal into the model formulation.
4. EXPERIMENTS
Experiments evaluate CDiffuSE on VoiceBank-DEMAND and under noise-condition shifts using comparisons with diffusion, generative, and regression-based speech-enhancement models. CDiffuSE improves over DiffuSE and other time-domain generative models on nearly all reported metrics, and is more resilient than regression-based competitors on CHiME-4.
- Results on VoiceBank-DEMAND: Large DiffuSE and CDiffuSE models outperform their corresponding base models in Table 1.The table reports VoiceBank results for both model sizes using the supportive reverse process.
- Results on VoiceBank-DEMAND: CDiffuSE improves on all metrics over the DiffuSE baseline by directly accessing noisy data during reverse diffusion.The authors attribute this access to compensating for input-noise characteristics.
- Results on VoiceBank-DEMAND: CDiffuSE outperforms SEGAN, SE-Flow, and DSEGAN on every metric except CBAK, with particularly significant improvement in PESQ.These comparisons use time-domain generative speech-enhancement models on VoiceBank.
- Results on CHiME-4: On matched VoiceBank conditions, regression-based models such as Demucs and Conv-TasNet outperform CDiffuSE, the strongest generative model in Table 2.The broader comparison indicates that generative speech-enhancement models lag behind regression-based counterparts in this setting.
- Results on CHiME-4: Under the VoiceBank-to-CHiME-4 domain shift, CDiffuSE degrades less than regression-based competitors and obtains the best CHiME-4 test scores.Demucs, Conv-TasNet, and WaveCRN experience significant performance drops, sometimes below unprocessed data.
5. CONCLUSION
The paper concludes that CDiffuSE explicitly explores noise characteristics from noisy input and adapts better to non-Gaussian real-world noise. It reports state-of-the-art results among generative approaches and stronger robustness to unseen noise characteristics than non-generative methods.
- Conclusion: CDiffuSE explicitly explores noise characteristics from the noisy input to adapt to non-Gaussian noise statistics.The authors describe it as a conditional diffusion probabilistic model for real-world speech enhancement.
- Conclusion: CDiffuSE achieves state-of-the-art results compared with other generative speech-enhancement approaches.This is the paper's reported generative-model comparison.
- Conclusion: CDiffuSE generalizes better than non-generative approaches to speech data with noise characteristics not observed during training.The authors report that it maintains strong performance when Demucs and Conv-TasNet collapse.