PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Speech Enhancement with Score-based Diffusion

Joan Serrà, Santiago Pascual, Jordi Pons, R. Oguz Araz, Davide Scaini

arXiv:2206.03065v2cs.SDcs.LGeess.AS

TL;DR

Speech enhancement research has largely focused on background noise, although real recordings contain many other distortions and universal enhancement lacks established test sets and metrics. The paper presents UNIVERSE, combining score-based diffusion with mixture-density conditioning, and reports expert-listener gains over 12 state-of-the-art approaches plus high-quality enhancement in 4–8 diffusion steps.

  • Problem

    Speech enhancement has traditionally targeted background noise, while recordings contain many other distortions and universal enhancement lacks established test sets or metrics.

    Source: §1 · p. 1, §4.1 · p. 7

  • Method

    UNIVERSE pairs a score-based diffusion generator of clean speech with a conditioning network using auxiliary mixture density network heads.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    12 state-of-the-art approaches were outperformed in expert-listener evaluation, and high-quality enhancement was achieved with 4–8 diffusion steps.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results support a universal, generative framing of speech enhancement across many distortions, evaluated subjectively by expert listeners.

    Source: §1 · p. 2

    Because data-driven models depend on training data characteristics, target languages and use cases should be sufficiently represented before deployment.

    Source: §5 · p. 10

Abstract

from arXiv · show

Removing background noise from speech audio has been the subject of considerable effort, especially in recent years due to the rise of virtual communication and amateur recordings. Yet background noise is not the only unpleasant disturbance that can prevent intelligibility: reverb, clipping, codec artifacts, problematic equalization, limited bandwidth, or inconsistent loudness are equally disturbing and ubiquitous. In this work, we propose to consider the task of speech enhancement as a holistic endeavor, and present a universal speech enhancement system that tackles 55 different distortions at the same time. Our approach consists of a generative model that employs score-based diffusion, together with a multi-resolution conditioning network that performs enhancement with mixture density networks. We show that this approach significantly outperforms the state of the art in a subjective test performed by expert listeners. We also show that it achieves competitive objective scores with just 4-8 diffusion steps, despite not considering any particular strategy for fast sampling. We hope that both our methodology and technical contributions encourage researchers and practitioners to adopt a universal approach to speech enhancement, possibly framing it as a generative task.

1 INTRODUCTION

Speech enhancement has broadened from denoising toward realistic recordings with many kinds of degradation. The paper proposes a universal approach that addresses 55 distortions using score-based diffusion and conditioning networks.

1 INTRODUCTION

  • Speech denoising has shifted from filtering focused on stationary noise to neural networks that generalize across noise types, while distribution shift and real-world recordings remain challenges.The authors also note an open question about whether two-stage generation or re-generation can perceptually outperform purely supervised approaches in realism and voice artifacts.Source: §1 · p. 1
  • 55 distortions across 10 families define the paper’s universal enhancement scope, including clipping, bandwidth reduction, codec artifacts, and silent gaps.The authors argue these neglected distortions occur in everyday recordings and amateur-produced audio.Source: §1 · p. 2
  • The end-to-end system combines a clean-speech generator built on score-based diffusion with a conditioner using auxiliary out-of-path mixture density networks; expert listeners found it outperformed the state of the art across all considered distortions.The authors found the diffusion steps needed for high-quality universal speech enhancement to be on par with the fastest diffusion-based neural vocoders, without specific tuning.Source: §1 · p. 2

2 RELATED WORK

Prior diffusion work spans generative audio and speech denoising, while universal enhancement remains a broader challenge. This paper positions generative enhancement as a promising approach and evaluates it with expert listeners.

2 RELATED WORK

  • Diffusion models have been applied to unconditional or weakly conditioned image generation and to audio tasks including speech synthesis, text-to-speech, bandwidth extension, and drum synthesis.The cited audio applications include Chen et al. (2021), Kong et al. (2021), Jeong et al. (2021), Popov et al. (2021), Lee & Han (2021), and Rouard & Hadjeres (2021).Source: §2 · p. 2
  • Earlier diffusion-based speech-denoising studies improved over their considered baselines but did not reach state-of-the-art objective scores.This work builds on WaveGrad and introduces modifications and concepts for universal enhancement across an unprecedented number of distortions.Source: §2 · p. 2
  • The authors report that their generative approach outperforms existing supervised and adversarial approaches in expert-listener evaluation, addressing a gap in subjective evidence.Earlier work suggested generative methods could help with clipping, bandwidth removal, and silent gaps because those degradations require generating absent signal, but had not convincingly demonstrated this through human judgment.Source: §2 · p. 2

3 DIFFUSION-BASED UNIVERSAL SPEECH ENHANCEMENT

The authors train a score-based diffusion enhancer on paired clean and synthetically distorted speech, then develop a two-network conditioning design for 55 distortion types. Their development experiments motivate multi-resolution conditioning, auxiliary losses, and a larger model for final evaluation.

3.1 METHODOLOGY

  • The training set contains 1,500 h of clean speech, programmatically paired with distortions from 55 types across 10 families.The speech spans about 1.2 M utterances, thousands of speakers, more than 10 languages, and over 50 recording conditions.Source: §3.1 · p. 3
  • Evaluation combines four objective metrics into COMP and compares enhanced outputs with existing systems in a preference test by 22 expert listeners.The objective measures include COVL, STOI, WARP-Q, and a SESQA-based metric.Source: §3.1 · p. 3

3.2 BASE APPROACH

  • The base method trains a variance-exploding score network and samples with noise-consistent Langevin dynamics, using distorted speech as the conditioning approaches’ only input.The conditioning signal c is paired with clean speech x0 during training; sampling uses N uniformly discretized time steps.Source: §3.2 · pp. 3–4
  • UNIVERSE combines a UNet-like generator with a conditioning network that extracts waveform and log-mel cues and supplies multi-resolution conditioning signals.The generator also receives noise-level signals; the conditioning network uses a two-layer GRU and convolutional decoder blocks.Source: §3.2 · p. 4, Figure 1 · p. 4

3.3 DEVELOPING UNIVERSE

  • Scaling to 189 M parameters and 2.5 M iterations yielded large objective and subjective improvement; this model was used for final evaluation with 50 diffusion steps.Training took less than 14 days using 8 Tesla V100 GPUs.Source: Table 1 · p. 6, §3.3 · p. 7
  • 4.75 COMP was reached after adding recurrent layers; earlier baselines exposed coarse-mel limitations, signal bypass, and varying scores across conditioning designs.Adding log-mel input to the conditioner improved on B3, while recurrent layers raised COMP to 4.75 in S2.Source: §3.3 · pp. 5–6
  • 0.41 COMP increment came from moving loss computation out of the signal path and using MDNs; adding waveform loss produced the largest objective-score improvement.Multi-resolution conditioning and decoupled heads also improved objective scores, with the latter improving subjective quality.Source: §3.3 · pp. 6–7
  • Additional pitch, harmonicity, VAD, and loudness targets gave marginal but consistently heard improvement; frozen-conditioner training scored worse, while noise-level input and auxiliary losses were important.The authors interpret the frozen-conditioner result as evidence of coupling between generator and conditioner.Source: §3.3 · p. 7

4 RESULTS

UNIVERSE is preferred over existing approaches across the tested distortion combinations. Ablations compare generative and denoising-only variants, while sampling tests report quality at high speed with few steps.

4.1 COMPARISON WITH EXISTING APPROACHES

  • Because established test sets and metrics for distortions beyond denoising are unavailable, the comparison uses expert preference and reports objective scores on two common denoising test sets.The authors note that common objective metrics may not measure enhancement of other distortions reliably.Source: §4.1 · pp. 7–8
  • UNIVERSE significantly outperforms 12 existing approaches and is preferred by expert listeners across all considered distortions.The subjective test covers different combinations of enhancement tasks.Source: Table 2 · p. 8, §4.1 · p. 8

4.2 VARIATIONS AND INSIGHTS

  • Listeners preferred the generative model over an otherwise capacity-matched regression version 88% of the time, particularly regarding speech distortion and artifacts.The regression version used MSE and STFT losses for direct waveform prediction; its conditioner retained MDN losses.Source: §4.2 · p. 8

4.3 SPEED-QUALITY TRADEOFF

  • 4–8 denoising steps achieve good-quality synthesis at over 10 times real-time speed on GPU, without a specialized fast-sampling strategy.The result holds for a variety of ϵ values using plain geometric scheduling; Figure 3 plots COMP against Tesla V100 GPU real-time factor as N and ϵ vary.Source: Figure 3 · p. 9, §4.3 · p. 9

4.2 VARIATIONS AND INSIGHTS

  • 43 vs. 57% preference was a slight, non-significant advantage for adding extra data and distortions to the additive-noise-only model.The comparison trained that model on 500 hours of additive-noise mixtures and evaluated it only on additive noise.Source: §4.2 · pp. 8–9
  • Listeners preferred ideal target recordings 86% of the time; informal listening identified very loud noise and strong, usually long reverberation as especially problematic.The authors conclude that UNIVERSE still has room for improvement.Source: §4.2 · p. 9

5 CONCLUSION

UNIVERSE treats speech enhancement as a universal task, using score-based diffusion with MDN conditioning; the authors report expert-listener gains over 12 state-of-the-art approaches and high-quality enhancement in 4–8 diffusion steps. The appendices describe the reverse diffusion process and its practical sampling schedule, while the conclusion cautions that deployment depends on training-set representation.

5 CONCLUSION

  • UNIVERSE combines score-based diffusion with MDN conditioning, outperforms 12 state-of-the-art approaches in an expert-listener test, and achieves high-quality enhancement in 4–8 diffusion steps.The authors also state that they did not use a particular fast-sampling strategy.Source: §5 · p. 9
  • Before deployment, target languages and use cases should be sufficiently represented in training data because data-driven models depend strongly on training-data characteristics.The authors report including multiple languages and recording conditions in their training set.Source: §5 · p. 10

A.1 THEORY

  • The reverse-time SDE uses the score of the time-dependent distribution to simulate samples from the prior back toward clean speech.The forward process progressively adds noise until samples become indistinguishable from a tractable prior; the reverse process uses the score function to recover data-distribution samples.Source: §A.1 · p. 15

A.2 IN PRACTICE

  • In practice, the authors use a variance-exploding schedule and consistent annealed sampling, discretizing time into N uniform steps and tuning a noise-progression parameter that varies with N.They use σ_t as the score-network input and condition generation on a signal derived from distorted speech.Source: §A.2 · pp. 15–16

B IMPLEMENTATION DETAILS

UNIVERSE jointly trains generator and conditioner encoder-decoder networks, with multiscale speech conditioning and auxiliary mixture-density losses. Its implementation uses specialized convolutional blocks and substantial training runs.

Architecture overview

  • The jointly trained generator estimates scores of perturbed speech distributions, while the conditioner supplies the signal for synthesizing speech with preserved speaker characteristics.Both are encoder-decoder architectures with convolutional blocks, conditioning signals, and skip connections.Source: B IMPLEMENTATION DETAILS · p. 17

Convolutional block

  • Convolutional blocks use a residual core, with optional prior or posterior parts; downsampling expands channels and upsampling reduces them.The core has one width-5 convolution followed by two width-3 convolutions; residual sums and skip connections are weighted by 1/√r.Source: B IMPLEMENTATION DETAILS · p. 17, Figure 4 · p. 17

Conditioner network

  • The conditioner combines downsampled waveform features with 80 mel-band features, then supplies hierarchical conditioning signals to the generator across sampling rates.Its encoder-decoder processes the signal down to 100 Hz and back to the original rate, with a bidirectional GRU in the middle.Source: B IMPLEMENTATION DETAILS · p. 18

Generator network

  • The generator uses UNet-like skip connections and a bidirectional GRU, and receives both noise-level embeddings and multirate conditioning from distorted speech.Conditioner signals enter corresponding generator decoder blocks from 100 Hz through 16 kHz.Source: B IMPLEMENTATION DETAILS · p. 18

Training objective

  • Training minimizes denoising score-matching loss plus waveform and feature negative log-likelihoods from auxiliary MDNs, making the model fully generative without sampling during training or validation.The MDNs model clean waveform and feature distributions using three diagonal-covariance multivariate Gaussian components.Source: B IMPLEMENTATION DETAILS · pp. 18–19

Optimization

  • The implementation trains a roughly 49-million-parameter model for 1 million updates, while the final 189-million-parameter model uses 2.5 million iterations.The final model uses batch size 64 and trains in under 14 days on eight Tesla V100 GPUs.Source: B IMPLEMENTATION DETAILS · p. 19

C DATA AND DISTORTIONS

Training pairs come from clean speech and programmatically generated distortions spanning 55 types in 10 families. The target speech is further processed through an enhancement chain before training.

Data

  • The training data pairs clean speech with distorted versions, using 3.5–5.5-second chunks from recordings spanning speakers, languages, prosody, and recording conditions.Distortions that introduce delays are automatically aligned at the sample level.Source: C DATA AND DISTORTIONS · p. 19

Input

  • The input pipeline randomly combines one to five distortions from 55 types grouped into 10 families, with perceptually noticeable parameter bounds.Table 3 groups the considered distortion types by family.Source: C DATA AND DISTORTIONS · p. 19, Table 3 · p. 20

Target

  • Rather than using the original clean recording as target, the authors apply a four-step enhancement chain because the model makes no assumptions about distortion type and targets should be high quality.The chain includes denoising, deplosive processing, de-essing, and dynamic equalization.Source: C DATA AND DISTORTIONS · p. 19
  • Passing clean recordings through the target chain is reported to discourage reliance on input characteristics and to give target speech a more homogeneous character or ‘signature sound’.The authors particularly note low-level input characteristics.Source: C DATA AND DISTORTIONS · pp. 19–20

D EVALUATION

The evaluation passage describes objective measures used to assess development steps and compare with speech-denoising state of the art.

D.1 OBJECTIVE MEASURES

  • COVL, STOI, WARP-Q, and SESQA are used to aid assessment of the development steps presented in the main paper.COVL and STOI use pysepm2 default parameters; WARP-Q uses its default parameters, and SESQA is a reference-based version of a reference-free quality measure.Source: §D.1 · p. 21
  • PESQ, COVL, and STOI are also used for comparison with state-of-the-art speech-denoising methods in Table 4.PESQ is calculated using pysepm.Source: §D.1 · p. 21
  • The passage specifies default-parameter implementations for the reported objective measures, including the package versions’ dates for COVL, STOI, and WARP-Q.The dates given are June 30, 2021 for pysepm2 and March 14, 2021 for WARP-Q.Source: §D.1 · p. 21

D.2 SUBJECTIVE TEST

The subjective test compares UNIVERSE with 15 approaches using expert listeners’ forced-choice preferences. The authors report pairwise preferences and adjust significance tests for multiple comparisons.

D.2 SUBJECTIVE TEST

  • The test included 12 existing approaches and three UNIVERSE ablations, using at least 25 speech pairs per system.Pairs judged too easy to enhance were removed, never more than five per approach.Source: §D.2 · p. 21
  • 22 expert listeners each made two forced-choice judgments per approach, for 30 choices across 15 approaches.Triplets were randomly selected and ordered; each contained the distorted input and two enhanced versions.Source: §D.2 · p. 21
  • Preference percentages compare UNIVERSE only with the corresponding competitor; differing materials and tasks prevent ranking systems across table rows.Significance was assessed with a binomial test at p < 0.05 and Holm-Bonferroni adjustment.Source: §D.2 · p. 21

E.1 DENOISING TASK WITH OBJECTIVE METRICS

For objective denoising benchmarks, the authors trained a task-specific UNIVERSE and evaluated standard metrics on two established datasets. Its expectation variant reached state-of-the-art values across both datasets and all metrics except one COVL result.

E.1 DENOISING TASK WITH OBJECTIVE METRICS

  • Table 4 reports narrow- and wide-band PESQ, while COVL is computed using wide-band PESQ.Source: §E.1 · p. 21
  • The denoising-specific model was evaluated on Voicebank-DEMAND and IS20 DNS-Challenge using PESQ, COVL, and STOI.A denoising-specific version avoids scoring UNIVERSE for additional enhancements beyond denoising.Source: §E.1 · p. 22
  • The expectation variant averages 10 waveforms sampled with the same conditioning signal, boosting standard metrics while audibly lowering high-frequency presence.The authors attribute the metric boost probably to sensitivity to small misalignments and low frequencies.Source: §E.1 · p. 22
  • UNIVERSE-Denoise-E achieved state-of-the-art values on both datasets and all metrics except Voicebank-DEMAND COVL, where it remained competitive among top systems.The table compares generative approaches, regression/adversarial approaches, and UNIVERSE variants.Source: Table 4 · p. 23, §E.1 · p. 23

E.2 SPEED-QUALITY TRADE-OFF

The speed-quality analysis varies the synthesis parameters and compares real-time processing speed with objective quality on the validation set.

E.2 SPEED-QUALITY TRADE-OFF

  • Figure 6 plots real-time factor on a Tesla V100 GPU against validation-set objective metrics while varying denoising iterations N and hyper-parameter ϵ.RTF is processing time divided by recording duration; values below 1 indicate faster-than-real-time processing.Source: §E.2 · p. 23, Figure 6 · p. 24
Loading 2206.03065v2…