Source-linked AI summary
Cold Diffusion for Speech Enhancement
Hao Yen, François G. Germain, Gordon Wichern, Jonathan Le Roux
TL;DR
Speech enhancement faces degradations that may be non-Gaussian, while a conditional diffusion objective can still assume white-Gaussian noisy speech. This work applies cold diffusion with unfolded training and reports stronger performance than other diffusion-based enhancement models, narrowing but not closing the gap with discriminative systems.
Problem
Realistic speech noise may be non-Gaussian, while the CDiffuSE objective still assumes noisy speech follows a standard white-Gaussian distribution.
Source: §1 · p. 1, §2.2 · p. 2Method
The paper defines noisy speech as the terminal point of a deterministic degradation from clean speech and uses unfolded training to account for multiple degradation and restoration steps.
Source: §3.1 · p. 2, §3.2 · p. 3Results
Takeaways & Limitations
The reported systems have not yet achieved the best overall results, and the authors identify different backbones, advanced losses, and data augmentation as compatible avenues for closing the remaining gap.
Source: §5 · p. 4CDiffuSE’s objective assumes that noisy speech follows a standard white-Gaussian distribution, which may not hold for speech enhancement.
Source: §2.2 · p. 2
Abstract
from arXiv · showhide
Diffusion models have recently shown promising results for difficult enhancement tasks such as the conditional and unconditional restoration of natural images and audio signals. In this work, we explore the possibility of leveraging a recently proposed advanced iterative diffusion model, namely cold diffusion, to recover clean speech signals from noisy signals. The unique mathematical properties of the sampling process from cold diffusion could be utilized to restore high-quality samples from arbitrary degradations. Based on these properties, we propose an improved training algorithm and objective to help the model generalize better during the sampling process. We verify our proposed framework by investigating two model architectures. Experimental results on benchmark speech enhancement dataset VoiceBank-DEMAND demonstrate the strong performance of the proposed approach compared to representative discriminative models and diffusion-based enhancement models.
1. INTRODUCTION
Speech enhancement commonly uses discriminative noisy-to-clean mappings, while cold diffusion’s broader family of degradations motivates an approach suited to non-Gaussian noise. The paper applies cold diffusion with unfolded training and reports improved results on VoiceBank-DEMAND.
- Speech enhancement methods include discriminative noisy-to-clean mappings and generative models that learn clean-speech distributions as priors, including GANs, VAEs, and flow-based models.Diffusion-based speech approaches include supportive reverse sampling, noisy-conditioned CDiffuSE, complex-STFT diffusion, and score-based enhancement for 55 distortions [26].Source: §1 · p. 1
- Cold diffusion [27] considers degradations beyond additive Gaussian noise, including blur, masking, and downsampling, motivating its use for non-Gaussian speech noise.The paper expects this framework to recover clean speech from arbitrary noise degradations.Source: §1 · p. 1
- On VoiceBank-DEMAND, the proposed system outperforms existing diffusion-based enhancement models and substantially shrinks the gap with discriminative models.The paper notes that its systems do not yet achieve the best overall results.Source: §1 · p. 1
- Unfolded training lets the network account for multiple degradation and restoration steps, and the paper proposes it to improve cold-diffusion performance.The paper presents its work as the first study of cold diffusion for additive degradations and speech enhancement.Source: §1 · p. 1
2. RELATED METHODS
The related methods section describes cold diffusion as learning to invert a severity-indexed degradation, with iterative sampling for more severe corruption. It also reviews conditional diffusion for speech enhancement and notes that its objective retains a white-Gaussian assumption.
2.1. Cold Diffusion
- Cold diffusion trains a restoration operator to approximately invert a degradation operator at intermediate severity levels, using a minimization objective.For audio, the norm can be the L1 norm.Source: §2.1 · p. 2
- For severe degradations, sampled reconstruction iteratively restores and re-degrades the signal as severity decreases; an improved sampling step is more tolerant to restoration errors [27].A single reconstruction step may suffice for small degradations, but can perform poorly for more severe ones.Source: §2.1 · p. 2
2.2. Conditional Diffusion Probabilistic Model
- CDiffuSE incorporates noisy data into conditional diffusion and uses a linear interpolation between clean and noisy data for the Gaussian-chain mean.Under this assumption, it estimates both Gaussian and non-Gaussian noise during the reverse process.Source: §2.2 · p. 2
- Despite conditioning on noisy spectrograms, CDiffuSE’s objective still assumes noisy speech follows a standard white-Gaussian distribution, which may not hold for speech enhancement.Source: §2.2 · p. 2
3. COLD DIFFUSION FOR SPEECH ENHANCEMENT
The method defines speech enhancement as deterministic degradation from clean speech to a noisy endpoint, then restores it through cold-diffusion sampling. Unfolded training and two backbone architectures adapt the framework to this task.
3.1. Degradation and Sampling Process
- The degraded sample is a deterministic interpolation between clean x0 and noisy xT, with αt decreasing from 1 to 0 across severity levels.Source: §3.1 · p. 2
- Sampling uses a degradation anchored around the current xt, which performs better than the alternative DxT(ˆx0,s) formulation.Source: §3.1 · pp. 2–3
3.2. Unfolded Training for Cold Diffusion
- Unfolded training degrades and restores the first prediction in a second step, training both estimates against clean speech to expose the model to its own reconstruction errors.The authors argue that this better matches iterative sampling and makes the model more tolerant of restoration errors.Source: §3.2 · p. 3
3.3. Model Structure
- DiffWave directly estimates clean waveforms with a Tanh output, while DCCRN adds diffusion-step embeddings throughout its encoder and decoder blocks.Source: §3.3 · p. 3, Figure 1 · p. 3
4. EXPERIMENTS
The experiments evaluate cold diffusion on VoiceBank-DEMAND with two backbones, comparing training variants against diffusion-based and discriminative systems. Unfolded CD improves on the original framework and narrows the performance gap to discriminative methods.
4.1. Dataset
- VoiceBank-DEMAND contains speech from 30 speakers and 10 noise types, with 28 speakers for training and two for testing; evaluation uses PESQ, CSIG, CBAK, and COVL.Training uses four SNRs from 0 to 15 dB, while testing uses four SNRs from 2.5 to 17.5 dB.Source: §4.1 · pp. 3–4
4.2. Model Setting and Training Procedure
- The study tests unconditioned DiffWave (2.3M parameters) and DCCRN (about 5.6M), using 50 diffusion steps and selecting checkpoints by validation PESQ.Training uses L1 loss, and results include direct reconstruction and 50-step improved sampling.Source: §4.2 · p. 4
4.3. Results
- Using the alternative degradation operator leads to slightly worse performance.Source: §4.3 · p. 4
- With DiffWave, CD beats DiffuSE and Base CDiffuSE on all metrics; Unfolded CD further beats Large CDiffuSE with fewer sampling steps and no conditioning.Source: §4.3 · p. 4, Table 1 · p. 4
- Unfolded CD improves on CD across both backbones and metrics, and on same-backbone discriminative systems except for DiffWave CBAK.The systems do not yet match top-performing methods overall.Source: §4.3 · p. 4, Table 1 · p. 4
5. CONCLUSION
The conclusion presents cold diffusion as a speech-enhancement framework and unfolded training as a way to improve it. Results improve on other diffusion-based methods and narrow, but do not close, the gap with discriminative systems.
- Cold diffusion can outperform other diffusion-based enhancement models, and unfolded training improves the original framework.Source: §5 · p. 4
- The systems do not yet achieve the best overall results; the authors identify different backbones, training losses, and data augmentation as compatible future directions.Source: §5 · p. 4
- The study establishes in-domain performance, while out-of-domain robustness remains a future direction; the authors attribute CDiffuSE’s reported robustness to noisy-input conditioning.Source: §5 · p. 4