PaperlayerView exact revision on arXiv

Source-linked AI summary

DriftSE: Speech Enhancement with Generative Drifting

Liang Xu, Diego Caviedes-Nozal, W. Bastiaan Kleijn, Longfei Felix Yan, Rasmus Kongsgaard Olsson

arXiv:2609.12252v1cs.SDcs.AIeess.AS

TL;DR

Iterative generative speech enhancement can require many function evaluations, limiting real-time use. DriftSE aligns generated speech with clean speech through latent distribution drifting, using dual semantic and acoustic latents and discarding drifting at inference. It achieves one-step enhancement with state-of-the-art word error rates across four evaluated datasets.

  • Problem

    Score-based speech enhancement reconstructs clean speech through iterative reverse trajectories, typically requiring 10–100 function evaluations and causing prohibitive latency for real-time use.

    Source: §I · p. 1

  • Method

    DriftSE trains a generator with latent distribution drifting, combining semantic and acoustic representations and discarding the drifting process at inference for one-step generation.

    Source: §I · p. 1, D. The Empirical Drifting Field · p. 2, A. One-Step Generation · p. 3

  • Results

    DriftSE operates strictly at 1 NFE and achieves state-of-the-art word error rates across all four evaluated datasets in causal and non-causal settings.

    Source: §I · p. 1, §VI · p. 10

  • Takeaways & Limitations

    DriftSE supports unpaired cross-dataset training and enhancement across different generator backbones in offline and real-time causal configurations.

    Source: §I · p. 1

Abstract

from arXiv · show

We propose DriftSE, a novel one-step generative framework for speech enhancement formulated as a latent distribution equilibrium problem. During training, the drifting field aligns the generator's pushforward distribution with the clean speech manifold through drifting in a latent domain. During inference, the drifting process is discarded, enabling one-step generation. We establish that its enhancement quality depends fundamentally on the choice of latent representation. Semantic latents preserve phonetic structure but fail to capture physical acoustic cues, whereas acoustic latents reconstruct the physical signal but risk linguistic hallucination. Therefore, we introduce dual-latent drifting, performing parallel drifting in both semantic and acoustic latents to simultaneously preserve phonetic intelligibility and acoustic fidelity. Additionally, we demonstrate that DriftSE enables fully unpaired training by aligning latent distributions rather than exact point-wise targets. Consequently, DriftSE facilitates cross-dataset learning in the absence of paired noisy-clean samples. Moreover, DriftSE exhibits broad architectural flexibility across different generator backbones. Extensive evaluations on additive denoising and convolutive dereverberation demonstrate robust one-step enhancement across both offline and real-time causal settings. Notably, DriftSE achieves state-of-the-art word error rates across all four evaluated datasets while strictly operating at 1 NFE. Code and audio examples are available online.

I. INTRODUCTION

Iterative score-based enhancement can impose prohibitive latency, motivating trajectory-free one-step generation. DriftSE uses latent distribution drifting and extends it to dual-latent, unpaired, and causal settings.

I. INTRODUCTION

  • Score-based generative models restore speech through iterative reverse trajectories, typically requiring 10–100 function evaluations that can make real-time enhancement prohibitively slow.Offline systems can use the complete signal, whereas streaming systems rely on causal processing for real-time interaction.Source: §I · p. 1
  • Cascades and distillation accelerate generation, but their continuous trajectory modeling motivates trajectory-free paradigms that natively enable 1-NFE generation.DriftSE discards its training-time drifting field at inference to enable one-step offline additive denoising.Source: §I · p. 1
  • DriftSE extends single-latent offline denoising to additive noise and reverberation, introducing dual-latent drifting, unpaired training, and varied generator backbones.The authors report robust one-step enhancement in offline and causal settings, with state-of-the-art WERs across four evaluated datasets.Source: §I · p. 1

II. BACKGROUND

  • Drifting Models frame generation as a dynamic equilibrium process that transports a pushforward distribution toward target data, formalized as a continuous Wasserstein gradient flow.The cited formulation connects this evolution to a smoothed KL energy functional.Source: §II · p. 2, B. Wasserstein Gradient Flow · p. 2

A. Pushforward Distribution

  • A generator maps samples from a source distribution to observations, inducing a pushforward distribution; for enhancement, noisy speech is the source and clean speech is the target.Training seeks to make the generated distribution converge to the target distribution.Source: A. Pushforward Distribution · p. 2

B. Wasserstein Gradient Flow

  • The Wasserstein-flow transport uses an implicit JKO update that evaluates velocity at an unknown future distribution, making the discretization computationally intractable.The flow follows steepest descent of smoothed KL energy, with Gaussian smoothing preventing infinite KL divergence between disjoint supports.Source: B. Wasserstein Gradient Flow · p. 2

C. Generative Drifting

Generative drifting replaces the intractable implicit update with an explicit Euler approximation. The generator is trained to match a drifted latent target, while drifting computations are reserved for training.

C. Generative Drifting

  • An explicit Euler approximation freezes transport velocity at the current distribution, bypassing the intractable implicit JKO update.The resulting update directly shifts the current pushforward mass toward an approximate successor distribution.Source: C. Generative Drifting · p. 2
  • The drifting target advances a generated sample's latent representation along the transport velocity field.The latent representation is defined by applying a pretrained encoder to the generator output.Source: C. Generative Drifting · p. 2
  • Training optimizes the generator so its encoded output matches the empirical drifted target, using stop-gradient; the empirical field absorbs the step size so h = 1.Drifting computations are discarded at inference, enabling one-step generation.Source: C. Generative Drifting · p. 2

D. The Empirical Drifting Field

The empirical drifting field estimates transport between target and model distributions in latent space. Its antisymmetric construction balances target attraction against model repulsion using kernel-weighted local mean shifts.

D. The Empirical Drifting Field

  • A valid empirical field must define a transport direction that vanishes at equilibrium and explicitly depends on target and model distributions.The field is constructed to be antisymmetric, so the target and model contributions cancel at equilibrium.Source: D. The Empirical Drifting Field · p. 2
  • Raw-waveform Euclidean distance can be dominated by signal energy and poorly reflect linguistic or acoustic similarity, motivating latent-space drifting.The target term attracts toward high-density data regions, while the model term repels from model-clustered regions.Source: D. The Empirical Drifting Field · p. 2
  • Both forces are kernel-weighted mean shifts toward local latent-space centers of mass, with local normalizers preventing the field from vanishing in low-density regions.The attractive and repulsive terms are combined into a unified field.Source: D. The Empirical Drifting Field · pp. 2–3

E. Drifting as Score Matching

Drifting aligns generated and target distributions through latent-space transport, whose field has a score-difference interpretation on smoothed densities. DriftSE applies this framework to speech enhancement and uses a single forward pass for generation.

E. Drifting as Score Matching

  • With a Gaussian kernel, the drifting field equals the scaled difference between target and model scores on smoothed densities.This identifies the field with an instantaneous score-based corrective direction.Source: E. Drifting as Score Matching · p. 3

III. METHOD

  • DriftSE formulates speech enhancement as a latent distribution equilibrium problem, extending previous work.Source: §III · p. 3

A. One-Step Generation

  • The mapping network operates on the complex STFT to model magnitude and phase, generating enhanced speech in one forward pass from degraded speech and scaled noise.Source: A. One-Step Generation · p. 3

B. Generative Latent Drifting

  • Latent-space alignment provides meaningful transport directions because raw waveform or STFT distances may poorly reflect speech structure.The latent spaces are selected so local neighborhoods capture underlying speech structure.Source: B. Generative Latent Drifting · p. 3

1) Latent Representations:

  • Dual-latent drifting aligns semantic and acoustic representations in parallel, combining phonetic integrity with physical environmental cues.The method also benchmarks joint semantic-acoustic latents.Source: 1) Latent Representations · p. 3

2) Empirical Latent Drifting:

Empirical latent drifting estimates transport directions from pooled generated and clean frames. The resulting drifted targets train the generator across one or more latent spaces.

  • The empirical drifting field replaces theoretical expectations with sample averages over pooled mini-batch frames.Source: 2) Empirical Latent Drifting · p. 3
  • The kernel-weighted field repels generated frames from model-output density and attracts them toward high-density regions of the clean manifold.Source: 2) Empirical Latent Drifting · p. 3
  • Training minimizes the discrepancy between each generated frame and its drifted target, with latent losses balanced by weights λ_n.For N = 1, the objective reduces to single-latent drifting; for N > 1, it aligns distributions across latent spaces.Source: 2) Empirical Latent Drifting · p. 3
  • The overview shows parallel latent drifting during training, while inference discards the encoders and drifting fields for 1-NFE generation.Source: Figure 1 · p. 4

22: // Inference Phase

The inference excerpt specifies sampling Gaussian noise and fixing the noise level before generating enhanced test speech.

  • Inference samples Gaussian noise and fixes γ_inf before generating enhanced test speech.Source: §22 · p. 4

C. Unpaired Training

DriftSE supports fully unpaired training by computing latent drift without time-aligned noisy-clean utterance pairs. Its latent representations provide complementary semantic, acoustic, and joint properties.

C. Unpaired Training

  • Batch-wise latent drifting requires no time-aligned utterance pairs, allowing degraded and clean examples to be drawn independently from different datasets.Source: C. Unpaired Training · p. 4

A. Latent Representations

  • DriftSE can use arbitrary pretrained encoders, whose frame features construct the parallel latent distributions; extracted layers are weighted equally.Source: A. Latent Representations · p. 4, Table I · p. 5
  • Semantic latents prioritize phonetic structure, acoustic latents capture broad environmental patterns, and joint latents combine phonetic structure with acoustic detail.Source: A. Latent Representations · p. 4

B. Datasets

The evaluation spans speech denoising and dereverberation benchmarks, plus mismatched-corpus unpaired training. It combines reference-based and reference-free speech-quality measures with system-level computational profiling.

B. Datasets

  • EARS-WHAM and VoiceBank-DEMAND (VB-DMD) evaluate denoising, with VB-DMD testing on unseen noise types.EARS-WHAM SNRs are sampled from −2.5 to 17.5 dB; VB-DMD training uses SNRs of 0, 5, 10, and 15 dB, and its test SNRs are 2.5, 7.5, 12.5, and 17.5 dB.Source: B. Datasets · p. 5
  • Dereverberation uses EARS-Reverb simulated from 2,313 RIRs and WSJ0-REVERB with simulated room responses and matched-geometry anechoic targets.EARS-Reverb draws RIRs from seven datasets with RT60 ≤2 s; WSJ0-REVERB uses T60 values from 0.4 to 1.0 s.Source: B. Datasets · p. 5
  • Unpaired experiments draw noisy utterances from VB-DMD or WSJ0-REVERB and clean speech from the mismatched DNS2020 corpus.DNS2020 contains 500 hours of clean speech from 2,150 speakers.Source: B. Datasets · p. 5

C. Evaluation Metrics

  • Reference-free perceptual evaluation uses DiMOS, WVMOS, NISQA, and SCOREQ to predict speech quality without a ground-truth reference.These predictors avoid the time-alignment and phase-matching constraints of intrusive metrics such as SI-SDR.Source: C. Evaluation Metrics · p. 5
  • Reference-based evaluation includes WER, PESQ, SI-SDR, and ESTOI; these measure linguistic preservation, perceptual quality, waveform fidelity, and intelligibility, respectively.Intrusive metrics require a time-aligned clean reference.Source: C. Evaluation Metrics · p. 5
  • Computational profiling reports trainable parameters, neural-network forward passes per enhanced signal, and multiply-accumulate operations per one-second segment.For iterative models, GMACs are calculated as the per-step baseline multiplied by NFE.Source: C. Evaluation Metrics · pp. 5–6

D. Implementation Details

DriftSE is instantiated with offline and causal backbones and trained using frozen encoders and multiscale latent features. Across denoising and dereverberation, its reported results include competitive WER and perceptual quality.

D. Implementation Details

  • Offline backbones are NCSN++ U-Net and non-causal TF-GridNet; causal settings use SFMUnet and TF-GridNet with 32 ms algorithmic delay.The models concatenate real and imaginary components of the scaled Gaussian noise and degraded STFT observation as channel-wise early fusion.Source: D. Implementation Details · p. 6
  • Training uses frozen pretrained encoders, uniformly weighted multiscale latent features, 2-second crops, and AdamW for 100 epochs.The STFT uses a 512-point FFT, hop size 256, and Hann window; training uses batch size 8 and learning rate 5 × 10−4.Source: D. Implementation Details · p. 6

A. Main Results

  • 14.33% WER on offline NCSN++ and 18.67% on causal SFMUnet make dual-latent drifting best across both U-Net settings; causal TF-GridNet with WavCube reaches 18.11%.Dual-latent drifting outperforms both single-latent configurations across the two U-Net backbones.Source: A. Main Results · p. 6
  • Dual-latent drifting raises denoising PESQ from 2.39 to 2.46 offline and from 2.05 to 2.13 causally; DriftSE’s SCOREQ of 3.85 exceeds ROSE-CD’s 3.50.ROSE-CD has a higher PESQ of 2.81; causal WavCube PESQ is 2.21 with SFMUnet and 2.26 with TF-GridNet.Source: A. Main Results · p. 6
  • 8.91% WER with offline NCSN++ dual latents beats FM-Euler5 (11.40%), while causal TF-GridNet with WavCube reaches 8.85% against SFM-LRK5 (15.90%).Source: A. Main Results · p. 6
  • Dereverberation TF-GridNet with WavCube reaches PESQ 2.43, while causal WavCube results exceed SFM-LRK5 on NISQA (4.07 vs. 3.67) and DiMOS (4.16 vs. 3.68).Offline NCSN++ with dual latents reports SCOREQ 3.61, compared with 2.69 for SGMSE+ and 2.66 for ROSE-CD.Source: A. Main Results · p. 6

B. Latent Ablation

Latent choice shapes DriftSE’s balance between phonetic integrity and acoustic fidelity. Dual-latent drifting improves dereverberation results, while denoising and generalization tests show how the balance varies by task and objective.

B. Latent Ablation

  • On high-SNR VB-DMD, acoustic encoders had the group’s highest WERs (11.44% for PANNs; 9.33% for BEATs), while dual latents improved fidelity but not WER over DistilHuBERT.With BEATs, PESQ rose from 3.00 to 3.07 and SI-SDR from 15.6 dB to 16.3 dB; WER remained 7.80%.Source: B. Latent Ablation · pp. 6–7
  • In causal denoising, SFMUnet with DistilHuBERT achieved PESQ 2.70 vs. 2.72 and DiMOS 4.05 vs. 3.88 against SFM-Euler4, using 24.6M parameters at 1 NFE versus 52.5M at 5 NFE.DistilHuBERT was also highly comparable to HuBERT across metrics, with 13× fewer parameters.Source: B. Latent Ablation · p. 7
  • On WSJ0-REVERB, WavLM+PANNs reduced WER from 5.31% to 4.01%, while dual-latent configurations raised PESQ from 2.18 to 2.37 and SI-SDR from −5.2 dB to −3.8 dB.Across backbones, TF-GridNet with WavLM+PANNs achieved 2.14% WER and 2.53 PESQ with 1.69M parameters, outperforming the 60-step SGMSE+ baseline’s 4.38% WER.Source: B. Latent Ablation · p. 7
  • The phoneme-accuracy plot inversely tracks downstream WER; WavCube reached 5.02% WER, but WavLM+PANNs achieved the best NCSN++ WER at 4.01%.Acoustic latents show lower phoneme accuracy and higher WER, while semantic latents show the reverse pattern.Source: B. Latent Ablation · pp. 7–8, Figure 2 · p. 8

C. From Memorization to Generalization

  • DriftSE generalizes best across in-domain VB-DMD and cross-domain EARS-WHAM, while RegressSE memorizes most and generalizes worst.Regression variants raise intrusive PESQ and SI-SDR, consistent with those metrics rewarding point-to-point memorization while drifting prioritizes latent generalization.Source: C. From Memorization to Generalization · p. 8

D. Unpaired Training

The conclusion reports one-step DriftSE with state-of-the-art word error rates across additive-noise and convolutive-reverberation evaluations, including offline and causal backbones. In unpaired training, latent distribution alignment captures acoustic structure, while missing utterance-level phonetic correspondence limits semantic fidelity.

D. Unpaired Training

  • DriftSEU trains on independently sampled degraded and clean speech, aligning aggregate frame distributions rather than enforcing point-to-point temporal trajectories.The training maps VB-DMD or WSJ0-REVERB speech to an independent DNS2020 clean-speech distribution.Source: D. Unpaired Training · pp. 8–9
  • Unpaired DriftSEU scored NISQA 4.22 and SCOREQ 3.77 for denoising, and NISQA 4.35 and SCOREQ 3.05 for dereverberation, with close acoustic-space alignment.For dereverberation, both models had a larger centroid offset, but DriftSEU tracked DriftSE almost identically: 2.250 vs. 2.248.Source: D. Unpaired Training · p. 9, Figure 3 · p. 9
  • Unpaired DriftSEU WER increased from 7.09% to 20.55% for denoising and from 3.20% to 15.27% for dereverberation; missing phonetic ground truth induces hallucinations.The authors conclude utterance-level pairing remains essential for semantic fidelity.Source: D. Unpaired Training · p. 9

VI. CONCLUSION

  • DriftSE achieves state-of-the-art word error rates across additive-noise and convolutive-reverberation evaluations while generalizing across offline and causal backbones at one-step inference.The conclusion also reports competitive perceptual quality.Source: §VI · p. 10
Loading 2609.12252v1…