PaperlayerView exact revision on arXiv

Source-linked AI summary

StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation

Jean-Marie Lemercier, Julius Richter, Simon Welker, Timo Gerkmann

arXiv:2212.11851v2eess.AScs.LGcs.SD

TL;DR

Speech restoration must handle corruption while balancing predictive distortion, generative artifacts, and diffusion models’ computational cost. The paper proposes stochastic regeneration, using a predictive estimate to guide a diffusion model. The approach preserves high sample quality, avoids unwanted artifacts, and reduces computational complexity by an order of magnitude without sacrificing quality.

  • Problem

    Diffusion models can produce artifacts in adverse conditions and require many reverse-diffusion network evaluations, while predictive models may distort target speech.

  • Method

    StoRM combines a predictive initial estimate with diffusion-based regeneration guided by that estimate.

  • Results

    StoRM improves ASR word-error-rate performance by a relative factor of 19% compared with SGMSE+M and reduces computational complexity by an order of magnitude without sacrificing quality.

  • Takeaways & Limitations

    Predictive guidance enables fewer diffusion steps while retaining high-quality samples and avoiding vocalizing artifacts and phonetic confusions.

Abstract

from arXiv · show

Diffusion models have shown a great ability at bridging the performance gap between predictive and generative approaches for speech enhancement. We have shown that they may even outperform their predictive counterparts for non-additive corruption types or when they are evaluated on mismatched conditions. However, diffusion models suffer from a high computational burden, mainly as they require to run a neural network for each reverse diffusion step, whereas predictive approaches only require one pass. As diffusion models are generative approaches they may also produce vocalizing and breathing artifacts in adverse conditions. In comparison, in such difficult scenarios, predictive models typically do not produce such artifacts but tend to distort the target speech instead, thereby degrading the speech quality. In this work, we present a stochastic regeneration approach where an estimate given by a predictive model is provided as a guide for further diffusion. We show that the proposed approach uses the predictive model to remove the vocalizing and breathing artifacts while producing very high quality samples thanks to the diffusion model, even in adverse conditions. We further show that this approach enables to use lighter sampling schemes with fewer diffusion steps without sacrificing quality, thus lifting the computational burden by an order of magnitude. Source code and audio examples are available online (https://uhh.de/inf-sp-storm).

I. INTRODUCTION

Speech restoration recovers clean speech from corruption, but predictive and diffusion approaches trade off distortion, artifacts, and computational cost. StoRM combines both approaches, using a predictive estimate to guide efficient diffusion-based regeneration.

  • Motivation: Real-world speech is corrupted by background noise, interfering speakers, room acoustics, and codec degradation.Speech restoration aims to recover clean speech from the corrupted signal.
  • Predictive and generative restoration: Predictive models learn a single deterministic mapping from corrupted speech, whereas generative models can produce multiple valid estimates from the target distribution.Diffusion models are generative approaches that use a forward process and a learned reverse process for conditional restoration.
  • Limitations of diffusion models: Diffusion models require repeated neural-network evaluations during reverse diffusion, creating a substantial computational burden.Several reverse-diffusion steps are needed because each step calls the score-estimation network.
  • Limitations of diffusion models: Diffusion models can generate vocalizing artifacts and phonetic confusions, while predictive models may smooth away fine-grained speech detail through regression to the mean.These effects motivate combining predictive and generative modeling for difficult restoration conditions.
  • Proposed approach: StoRM first predicts a restored signal and then uses that estimate to guide a diffusion model that produces a final clean-speech estimate.The proposed approach is evaluated for low-SNR speech enhancement and dereverberation, including ASR results and ablations of sampling efficiency, predictor mismatch, and training strategy.
  • Diffusion framework: Score-based diffusion models define restoration through a forward diffusion process, a score-function estimator, and a sampling method for inference.The process operates on complex spectrogram representations, with diffusion time τ indexing stochastic-process progress rather than signal time.

A. Predictive artifacts for images and spectrograms

Predictive speech-restoration models can smooth away fine-grained target detail and distort speech, while residual-based diffusion refinement may depend strongly on the task and predictor. These limitations motivate stochastic regeneration as an alternative refinement strategy.

  • Predictive artifacts: L2-trained predictive models learn the posterior mean E[x|y], which can poorly represent intricate posterior distributions through regression to the mean.This may produce outputs that do not lie on the posterior data manifold.
  • Predictive artifacts: In spectrograms, predictive smoothing removes fine-grained detail in quiet regions and speech onsets or offsets.The spectrogram can be viewed as an image, with distortions corresponding to loss of low-luminosity detail and edges.
  • Stochastic refinement: Stochastic refinement combines a predictive estimate with a diffusion model that learns an ideal residue and adds its estimate back to the predictor output.The approach is intended to correct predictive-stage distortions while retaining efficient inference.
  • Residual limitations: Residual distributions can lack a consistent structure: dereverberation residues degrade formants, while denoising residues show no clear structure compared with clean speech.Their distribution depends strongly on the predictive model and restoration task.
  • Residual limitations: Pointwise entropy may make residual learning easier but does not capture global structures that can facilitate training.The authors therefore question whether residual learning provides the most suitable generative target.
  • Residual limitations: At low SNRs, the residual refinement starting point has very low a priori SNR, making the reverse-process learning difficult.This motivates the proposed stochastic regeneration process.

C. Stochastic regeneration

StoRM cascades a predictive denoiser with a diffusion model that regenerates clean speech from the predictive estimate. This two-stage design aims to correct predictive distortions while reducing unwanted generative artifacts and diffusion cost.

  • C. Stochastic regeneration: StoRM cascades predictive model Dθ with generative diffusion model Gϕ to regenerate clean speech from a distorted predictive estimate.The diffusion model regenerates clean speech based on the predictive output, unlike stochastic refinement, which operates on noisy residual cues.
  • C. Stochastic regeneration: The diffusion stage removes residual noise and restores distorted target cues from the initial estimate Dθ(y).The resulting starting point has high a priori SNR when predictive distortions and residual noise are much smaller than the original corruption.
  • C. Stochastic regeneration: The inference process first produces Dθ(y), adds Gaussian noise to obtain xT, and solves the reverse diffusion SDE to sample x0 from p(x|Dθ(y)).This is the inference process illustrated in Figure 4.
  • C. Stochastic regeneration: StoRM training combines denoising score matching with supervised regularization matching the initial predictor output to target speech.The balance term α is empirically set to 1.
  • C. Stochastic regeneration: StoRM remains generative because it can output realistic samples belonging to a posterior distribution despite conditioning on the predictive estimate.The paper uses Figure 5 to contrast predictive, generative, and stochastic-regeneration inference.

A. Data

The study uses simulated noisy, reverberant, and speech-corrupted datasets, with StoRM inference described through a predictor followed by diffusion sampling. The data include WSJ0-, TIMIT-, CHiME3-, and VoiceBank/DEMAND-based conditions.

  • A. Data: The StoRM inference algorithm outputs a clean speech estimate using predictor-corrector diffusion steps.The supplied algorithm excerpts identify corrector and Euler–Maruyama prediction stages.
  • A. Data: CHiME3 mixtures combine a randomly selected noise file with clean speech at SNRs uniformly sampled between -6 and 14 dB.
  • A. Data: TIMIT+Chime is generated similarly to WSJ0+Chime and supports ASR evaluation because oracle word annotations are available.
  • A. Data: VoiceBank/DEMAND uses VCTK clean speech and recorded or artificial DEMAND noises at specified training, validation, and test SNRs.Training and validation use 0, 5, 10, and 15 dB; test conditions use 2.5, 7.5, 12.5, and 17.5 dB.
  • A. Data: WSJ0+Reverb is created by convolving WSJ0 utterances with simulated room impulse responses.Rooms sample T60 between 0.4 and 1.0 seconds, producing average DRR around -9 dB and measured T60 of 0.91 s.
  • A. Data: StoRM uses a two-stage path through the posterior mean before generating a posterior sample more likely to occupy high-density regions.

B. Hyperparameters and training configuration

Experiments use standardized STFT representations, diffusion settings, a lighter NCSN++M backbone, alternative predictors, and task-specific baselines. Training uses Adam, exponential moving averages, early stopping, and model-specific losses.

  • B. Hyperparameters and training configuration: Utterances are represented with 16 kHz STFTs using a 510-sample window, 128-sample hop, square-root Hann window, and magnitude warping.Training sequences contain 256 STFT frames, approximately 2 seconds, and are normalized by the noisy utterance maximum absolute value.
  • B. Hyperparameters and training configuration: Unless otherwise stated, reverse diffusion uses N = 50 time steps with a predictor-corrector sampling scheme.The fixed parameters include γ = 1.5, σmin = 0.05, σmax = 0.5, T = 1, and τϵ = 0.03.
  • B. Hyperparameters and training configuration: The lighter NCSN++M backbone has roughly 27.8M parameters instead of 65M without significant speech-enhancement degradation.
  • B. Hyperparameters and training configuration: StoRM additionally conditions score estimation on the initial prediction Dθ(y), noisy spectrogram y, and current diffusion estimate xτ.ConvTasNet and GaGNet are also tested as unseen alternative initial predictors during inference.
  • B. Hyperparameters and training configuration: WSJ0-based experiments compare StoRM with SGMSE+M, NCSN++M, and GaGNet, while VoiceBank/DEMAND adds several predictive and generative baselines.
  • B. Hyperparameters and training configuration: Models use Adam with learning rate 10^-4, effective batch size 16, exponential-moving-average weights, and early stopping with patience 10.Generative models use denoising score matching, while predictive methods use mean-square error on complex spectrograms.

C. Evaluation metrics

Evaluation combines intrusive speech-quality, intelligibility, distortion, ASR, and listening-test measures. The listening test compares predictive, generative, and StoRM systems on balanced WSJ0-based samples.

  • C. Evaluation metrics: Intrusive evaluation measures speech quality with PESQ, intelligibility with ESTOI, and distortion components with SI-SDR, SI-SIR, and SI-SAR.
  • C. Evaluation metrics: ASR performance is evaluated with pretrained QuartzNet Base-en and word error rate computed using dynamic programming with jiwer.
  • C. Evaluation metrics: A medium-scale MUSHRA listening test with 9 participants rates overall quality across speech distortion, residual distortions, and potential artifacts.Participants rate 10 samples with one overall-quality score.
  • C. Evaluation metrics: The MUSHRA comparison includes NCSN++M, SGMSE+, and StoRM, using noisy mixtures as low anchors and inputs with 10 dB higher SNR as supplementary anchors.Samples are balanced across gender and task, with speaker exclusivity within each task.

V. EXPERIMENTAL RESULTS AND DISCUSSION

StoRM combines predictive interference removal with generative sample quality across denoising and dereverberation benchmarks. It also slightly outperforms other generative models on VoiceBank/DEMAND.

  • A. Comparison to baselines: Predictive NCSN++M and GaGNet provide stronger interference removal and intelligibility, while SGMSE+M provides higher PESQ and WV-MOS.This quality–fidelity gap is stronger for dereverberation than denoising.
  • A. Comparison to baselines: StoRM produces very high quality samples like SGMSE+M while achieving approximately NCSN++M-level interference removal.For dereverberation, StoRM outperforms both comparators on all reported metrics.
  • B. VoiceBank/DEMAND: StoRM slightly outperforms other generative models on ESTOI, WV-MOS and SI-SDR on VoiceBank/DEMAND.The benchmark uses predominantly positive SNRs around 10 dB, making it less challenging than WSJ0+Chime.

B. Efficient sampling

StoRM maintains near-optimal quality with substantially fewer reverse-diffusion steps and no corrector, reducing computation while preserving enhancement quality. The reported examples also show correction of artifacts and reverberation under adverse inputs.

  • B. Efficient sampling: 10 steps preserve StoRM performance near its optimum, whereas SGMSE+M degrades rapidly as reverse-diffusion steps decrease.The comparison uses the same NCSN++M architecture for denoising on WSJ0+Chime.
  • B. Efficient sampling: 4.5 · 10^11 MAC·s−1 yields near-optimal StoRM quality with 20 steps and no corrector, versus 2.1 · 10^12 MAC·s−1 for optimal SGMSE+M.StoRM also outperforms optimal SGMSE+M with 10 steps and no corrector, reducing computational complexity by a full order of magnitude.
  • Qualitative examples: StoRM corrects speech distortions without vocalizing artifacts at −0.9 dB input SNR and restores formant structure without residual reverberation at 1.06 s input T60.These examples compare processed utterances from WSJ0+Chime and WSJ0+Reverb.
  • Mismatched conditions: StoRM and SGMSE+M maintain performance better than NCSN++M when trained on VoiceBank/DEMAND and tested on WSJ0+Chime⋆.The mismatched condition uses input SNRs from 0 to 20 dB.

D. Generalization to mismatched predictors

StoRM remains effective when its inference-time predictor differs from the training predictor, and listening tests favor it over purely predictive and generative alternatives. Its ASR performance also improves over SGMSE+M.

  • D. Generalization to mismatched predictors: StoRM performance with GaGNet as the inference-time predictor is equivalent to using the training predictor NCSN++M.With ConvTasNet, performance is poorer but close to the NCSN++M-predictor configuration.
  • E. ASR results: StoRM improves WER by 19% relative to SGMSE+M and slightly outperforms NCSN++M on TIMIT+Chime speech enhancement for ASR.The passage attributes the improvement to correcting vocalizing artifacts and phonetic confusions.
  • C. Listening test: Participants rated StoRM higher on average than both NCSN++M and SGMSE+M in the MUSHRA listening test.NCSN++M was rated slightly better than SGMSE+M, consistent with listeners weighting residual distortions and artifacts strongly.

G. Ablation studies

The ablations show that StoRM benefits from conditioning the score network on both noisy speech and the predictor estimate, and from supervised regularization during training. These choices support the paper’s broader conclusion that stochastic regeneration can preserve quality while reducing diffusion cost.

  • Conditioning of the score network: Using both noisy speech and the denoised predictor estimate as conditioning inputs performs best overall, whereas using only noisy speech is detrimental.The predictor estimate supplies information about the original distortions needed during early denoising steps; instrumental metrics slightly favor “Both” over “PostDenoiser.”
  • Training strategies: Jointly training the predictor and score network slightly improves denoising results compared with alternative training strategies.The most favorable strategy combines joint training with the supervised criterion J (Sup).
  • Training strategies: Removing supervised regularization causes a dramatic performance drop when the predictor is randomly initialized and trained only with the score-matching criterion J (DSM).The authors attribute this to the larger search space and lack of regularization.
  • Training strategies: The preferred training strategy encourages the predictor to produce speech-like outputs while allowing it to adapt its representation for regeneration.This balances a clean-speech target with flexibility for the diffusion stage.
  • Overall conclusion: An order-of-magnitude reduction in computational complexity is achieved without sacrificing quality compared with the original diffusion model.The conclusion attributes this efficiency to fewer reverse-diffusion steps and avoiding Annealed Langevin Dynamics correction.
Loading 2212.11851v2…