Source-linked AI summary
Unsupervised Speech Enhancement via Drifting
Diego Caviedes-Nozal, Liang Xu, Rasmus Kongsgaard Olsson, W. Bastiaan Kleijn
TL;DR
Unpaired speech enhancement trains on separate degraded and clean collections, but standard drifting can lose the input’s linguistic content and speaker identity. The paper introduces input-conditioned drifting to re-tether outputs to inputs while retaining the pull of clean speech. It reports improved single-step enhancement and partial transfer to dereverberation, where the quality gap remains open.
Problem
Unpaired training provides separate degraded and clean collections, while standard drifting’s input-decoupled objective can lose linguistic content and speaker identity.
Source: §1 · p. 1Method
Input-conditioned drifting uses anchor and key encoders to re-tether the target to the input without paired data.
Source: §1 · p. 1Results
Single-step generative enhancement reduces WER below the input while improving quality; the reported ECAPA speaker-identity score is 0.879 against 0.888 for the input.
Source: §4 · p. 4Takeaways & Limitations
On dereverberation, content measures improve on the input, but rendering quality and speaker similarity do not.
Source: §4 · p. 4On dereverberation, the reported gain is confined to content, while SCOREQ and speaker similarity do not improve.
Source: §4 · p. 4
Abstract
from arXiv · showhide
This paper addresses unsupervised speech enhancement in the unpaired setting using drifting methods, where training relies on separate collections of degraded and clean audio without corresponding pairs. While recent drifting approaches enable unpaired training, they do so at a heavy cost: because the objective optimizes only a marginal prior over clean speech, the enhancer gradually loses the input's linguistic content and speaker identity. To fix this, we introduce input-conditioned drifting. We preserve the pull of the clean corpus while re-tethering the output to the degraded input via two mechanisms: an anchor encoder supplies the missing likelihood by pulling toward the input's features, and a key encoder conditions the prior by re-weighting retrieved frames. Neither requires labels or paired data. Using a training-free encoder selection criterion, Word Error Rate on VoiceBank-DEMAND falls to 10.1% (unprocessed: 11.7%), speaker similarity recovers from 0.490 to 0.879, and the recipe transfers in part to dereverberation on WSJ0-REVERB: content improves, rendering quality does not.
1. INTRODUCTION
Unpaired enhancement uses separate degraded and clean speech collections, but their marginal distributions do not specify which clean signal corresponds to each degraded input. The introduction identifies content drift in unpaired drifting and proposes input-conditioned methods to preserve the input while learning from clean speech.
1. INTRODUCTION
- Unpaired training uses degraded and clean collections without correspondence, so their marginal distributions do not identify which clean signal belongs to each degraded input.Existing approaches add reconstruction requirements or transport assumptions to constrain correspondence.Source: §1 · p. 1
- Drifting iteratively moves generated samples toward clean examples in feature space until the generated distribution matches the clean distribution.Unlike methods requiring a corruption-mechanism assumption, drifting does not impose one.Source: §1 · p. 1
- In unpaired speech enhancement, DriftSE can modify linguistic content while maintaining non-intrusive quality.The paper attributes losses of linguistic content and speaker identity to an objective decoupled from the input and potentially misaligned feature-space gradients.Source: §1 · p. 1
- Input-conditioned drifting re-tethers the target to the degraded input using anchor and key encoders, without requiring paired data.The paper also proposes a Bayesian diagnosis, a self-paired GAN-based corrector, and a training-free encoder-selection criterion.Source: §1 · p. 1
2. METHOD
Standard drifting moves generated speech toward a clean-speech prior without tying each output to its degraded input, allowing linguistic content and speaker identity to drift. Input-conditioned drifting adds an anchor and a key to restore that connection without paired data.
2.1. The content-drift problem in unpaired drifting
- Unpaired drifting attracts generated frames to a generic clean bank and repels other generated frames, but its prior-only objective cannot tie outputs to their own inputs.At equilibrium, attributes the encoder does not resolve can average across the bank, producing clean speech that loses the input’s words and voice.Source: §2.1 · pp. 1–2
2.2. Input-conditioned drifting: anchors and keys
- Input-conditioned drifting reinjects the degraded input through either the missing likelihood or a conditional prior, without paired data.The two routes correspond to changing the target set or its retrieval weighting.Source: §2.2 · p. 2, §2.1 · p. 2
- The anchor encoder supplies an input-frame target, adding a force that pulls generated features toward the input’s own words and voice.This Gaussian-likelihood interpretation is valid only when corruption barely moves the anchor representation.Source: §2.2 · p. 2
- The key encoder reweights clean-bank rows using an input embedding, yielding a conditional prior assembled from input-like clean speech.Unlike the anchor, the key lets residual noise affect which clean rows are retrieved.Source: §2.2 · p. 2
- Bank, anchor, and key encoders require different properties, so the paper screens them separately using a margin between an utterance and its own clean version versus other utterances.The bank should separate clean from degraded speech; the anchor should be corruption-robust; and the key should retrieve the same rows despite degradation.Source: §2.2 · p. 2
3. EXPERIMENTS AND RESULTS
The experiments track input-blind drift, test anchoring and keying, compare unsupervised systems, and examine transfer to dereverberation and a self-paired corrector.
3.1. Setup
- The clean bank is disjoint from both evaluation corpora, so the loss never sees the clean signal of the utterance being processed.The study uses VoiceBank–DEMAND for denoising and WSJ0-REVERB for dereverberation, and rescores baselines within its own pipeline because non-intrusive metrics differ across implementations.Source: §3.1 · p. 3
3.2. Content drift over training
- WER rises 13.4±1.2 points and speaker similarity falls 0.09±0.02 per 100k steps, while SCOREQ saturates within the first 5k steps.The unanchored system’s best WER is 17–20% in that early window, still worse than the unprocessed input.Source: §3.2 · p. 3, Figure 2 · p. 3
3.3. Effect of anchoring and keying
- Anchors slow content drift more than sixfold, and adding the key takes WER below the unprocessed input without changing the pairing level.The content anchor carries the WER reduction, while the identity anchor carries speaker-similarity recovery.Source: §3.3 · p. 3
3.4. Effect of key strength
- 10.51% WER is reached by the confidence gate at the background quality of β=1.The gate interpolates between β=1 and β=4 using recognizer confidence; that confidence separates input-correct from input-wrong words with AUC 0.90–0.92. A fixed-β sweep gives U-shaped WER, with its minimum at β=4.Source: §3.4 · p. 3
3.5. Comparison with other unsupervised systems
- SelfSE attains the best WER and quality in the comparison, while only DiffUSEEN and the keyed system among resynthesis systems improve on input WER.Replacing the pretrained bank encoder with whitened complex-STFT frames performs worse than the unprocessed input in WER, speaker similarity, and DNSMOS.Source: §3.5 · p. 3, Table 2 · p. 4
3.6. Dereverberation
- On dereverberation, the full system improves WER, ESTOI, and SRMR over input, but not SCOREQ.The content-anchor margin falls from 0.205 under noise to 0.102 under reverberation, while the identity margin rises from 0.585 to 0.633; stronger keying trades dereverberation strength for fidelity.Source: §3.6 · p. 4, Table 3 · p. 4
3.7. Self-paired corrector
- The self-paired corrector raises SCOREQ while preserving WER on the anchored generator, but costs 1.0 WER point on the keyed one.Its construction fails on dereverberation because that generator leaves clean input almost unchanged, yielding pairs without an artifact to learn from.Source: §3.7 · p. 4
4. DISCUSSION AND CONCLUSIONS
Input-conditioned drifting reduces WER below the input while improving quality, and resynthesizes speech in one step. Its benefits are bounded: dereverberation gains are confined to content, and rendering quality remains an open gap.
4. DISCUSSION AND CONCLUSIONS
- 1.7M parameters enable one-step resynthesis, versus 5.2M and 30 steps for the nearest generative competitor.Speaker identity stays close to the input (ECAPA 0.879 against 0.888), and the anchors and key act only during training.Source: §4 · p. 4
- Masking systems keep the lead on WER and intrusive quality (SelfSE 9.3%); dereverberation improves content but not rendering quality.The corrector raises quality on the anchored denoiser but costs 1.0 WER point on the keyed one.Source: §4 · p. 4
- A room-acoustics bank encoder reproduces collapse (42.6% WER); unlike mode collapse, content drift can match the clean marginal while each output's input link decays.The analysis also reports that conditioning on the input stops the drift and interprets MOS-GAN's behavior as content drift rather than training instability.Source: §4 · p. 4
- Input-conditioned drifting reduces WER below the input while improving quality, but the dereverberation quality gap remains open.Source: §4 · p. 4