Source-linked AI summary
Test-time adaptation for speech enhancement with an autoregressive speech prior
Sofiene Kammoun, Simon Leglaive, Xavier Alameda-Pineda, Timo Gerkmann
TL;DR
Speech enhancement models may generalize poorly to mismatched or unseen acoustic conditions, where labeled target data are unavailable. The paper adapts a pretrained model at test time using an autoregressive clean-speech prior and KL divergence, yielding consistent speech-quality improvements across noisy-speech benchmarks, especially under noise mismatch.
Problem
Supervised speech enhancement can generalize poorly when test acoustics differ from training, while deployment often lacks labeled target signals or training data.
Method
A pretrained speech enhancement model is adapted per utterance by minimizing KL divergence between enhanced-speech distributions and an autoregressive prior over clean neural-audio-codec latent representations.
Results
The method consistently improves speech quality across multiple real and synthetic noisy-speech benchmarks, especially under mismatched noise conditions.
Takeaways & Limitations
The clean-speech prior discriminates between noisy and clean speech and provides an unsupervised adaptation signal for unseen acoustic conditions.
Takeaways & Limitations
Without reinitialization, many adaptation steps, or a small learning rate, the model can collapse toward the prior and ignore the noisy input.
Abstract
from arXiv · showhide
Test-time adaptation (TTA) offers a promising direction for improving speech enhancement models under mismatched acoustic conditions, without requiring access to labeled target data. In this work, we propose a single-utterance TTA method that regularizes a pretrained speech enhancement model using an autoregressive prior trained on clean speech latent representations extracted from a neural audio codec. Adaptation is performed by minimizing the Kullback-Leibler divergence between the enhanced speech distribution and the clean speech prior. Experiments across multiple noisy speech datasets show consistent improvements in speech quality, particularly under training-testing noise mismatch conditions. Code and audio examples are available online.
1. INTRODUCTION
Speech enhancement models can fail under acoustic conditions that differ from training, while practical deployment often lacks labeled target data. The proposed single-utterance TTA method uses a clean-speech autoregressive prior to adapt a pretrained model toward outputs more consistent with clean speech.
- Motivation: Supervised speech enhancement can degrade under strong training-testing acoustic mismatch or unseen distortions.Examples include reverberation, bandwidth reduction, clipping, and codec artifacts.
- Existing adaptation strategies: Fine-tuning and unsupervised domain adaptation require labeled source data, whereas TTA updates the pretrained model during inference using unsupervised losses.TTA does not modify the original supervised training procedure.
- Existing adaptation strategies: Classification-oriented entropy minimization and feature alignment do not readily transfer to speech enhancement because it is typically formulated as regression.
- Proposed approach: The proposed method adapts a pretrained speech enhancement model with an autoregressive prior trained on clean neural-audio-codec speech representations by minimizing KL divergence.The prior measures how likely enhanced representations are to resemble clean speech.
- Proposed approach: The clean-speech prior guides adaptation toward higher prior log-density and enhanced outputs closer to clean reference speech.
2. METHOD
The method combines supervised speech enhancement in a neural audio codec latent space with single-utterance test-time adaptation guided by an autoregressive clean-speech prior. Adaptation minimizes KL divergence between the enhancement model and prior, using a short noisy segment while preserving input dependence through reset, small-step updates.
- Supervised speech enhancement: The supervised enhancement model represents clean and noisy waveforms in a pretrained neural audio codec’s latent space and models clean speech conditionally on noisy speech.The model is trained by minimizing negative log-likelihood on paired noisy-clean data, which reduces to MSE under the Gaussian formulation.
- Autoregressive prior: The clean-speech prior is an autoregressive Gaussian model whose mean and time-varying variances are predicted from preceding latent representations.A shared orthogonal matrix provides a global eigenbasis, enabling a full covariance model while remaining computationally efficient.
- Autoregressive prior: The prior is trained on a clean speech corpus by minimizing average negative log-likelihood with teacher forcing.Its full covariance structure was empirically found to yield higher log-density values than a diagonal covariance model on clean speech.
- Unsupervised test-time adaptation: At test time, the model adapts its inference parameters on one noisy utterance by minimizing KL divergence between the pretrained enhancement distribution and frozen clean-speech prior.The intractable expectation is approximated using predicted preceding latent means, with gradients stopped through those means.
- Unsupervised test-time adaptation: Adaptation uses a randomly selected one-second noisy segment, then applies the adapted model to produce the latent representation of the complete enhanced signal.The enhanced latent sequence is decoded by the neural audio codec, and the procedure is framed as calibration to new acoustic noise characteristics.
- Unsupervised test-time adaptation: The parameters are reset before each new noisy signal, and adaptation uses few gradient steps with a small learning rate to avoid collapse toward the prior.Without reinitialization, continued optimization could make the inference model independent of the noisy input.
3. EXPERIMENTS
Experiments evaluate the proposed TTA method across mismatched noisy-speech datasets using non-intrusive quality metrics and WER. Results show that adaptation benefits lower-quality outputs and unseen noise conditions, but excessive adaptation can degrade performance.
- Experimental setup: The evaluation spans DNS Challenge V5, TIMIT-DEMAND, EARS-WHAM, and Libri1Mix under varying degrees of acoustic mismatch.DNS Challenge V5 contains 600 real noisy recordings without parallel clean references; metrics include DNSMOS SIG, BAK, OVRL, and WER when references are available.
- Prior model validation: The clean speech prior assigns higher log-density to clean than noisy speech, supporting its use as a weak supervision signal during TTA.Prior validation uses EARS and Libri1Mix distributions and qualitative comparisons of noisy, enhanced, and clean speech.
- TTA results: On DNS Challenge V5, speech quality improves during early TTA steps but eventually degrades as the model may collapse toward the prior.The comparison is made against the pretrained model at k = 0.
- TTA results: Lower initial prior-log-density quartiles benefit more from adaptation and typically require more steps to reach optimal performance.Higher-log-density quartiles show limited or negative gains; on Libri1Mix, the WER increase for Q4 is at most about 0.015.
- TTA results: A logistic regression model predicts the optimal TTA step from the enhanced speech’s initial prior log-density.The model is trained on DNS Challenge results and applied to other datasets; the optimal step is defined as k* = arg max_k OVRL(k).
- TTA results: Table 1 reports improvements across all datasets for both the oracle-selected k = k* and the predicted k = k̃, with smaller gains for k̃.Gains are especially important for DNS Challenge and TIMIT-DEMAND, particularly in BAK, where noise types mismatch supervised pretraining.
4. CONCLUSION
The paper presents a single-utterance TTA framework that uses an autoregressive clean-speech prior in neural-audio-codec latent space. KL-divergence adaptation enables unsupervised adjustment to unseen acoustic conditions without changing the original training pipeline or accessing source data.
- Conclusion: The proposed framework adapts a pretrained speech enhancement model by optimizing KL divergence against an autoregressive clean speech prior in neural-audio-codec latent space.The prior discriminates between noisy and clean speech and serves as the adaptation signal.
- Conclusion: The method enables unsupervised adaptation to unseen acoustic conditions without modifying the original training pipeline or requiring source data.