Source-linked AI summary
Noisy Target Training Meets Drifting for Weakly-Supervised Speech Enhancement
Yoshiki Masuyama, Dominik Klement, Christoph Boeddeker, Julius Richter, Kohei Saijo, Gordon Wichern, Takahiro Edo, Jonathan Le Roux
TL;DR
Synthetic clean–noisy pairs used in supervised speech enhancement may mismatch real recordings, while noisy-target training can leave residual noise. The paper adds unpaired Sinkhorn drifting to NyTT to align enhanced speech with the clean-speech distribution. The combination improves perceptual metrics over mean-teacher NyTT on VoiceBank-DEMAND and is demonstrated on real CHiME-3 recordings.
Problem
Synthetic clean–noisy pairs used for supervised speech enhancement can mismatch real acoustic conditions, and NyTT can retain residual noise without explicitly matching clean-speech distributions.
Source: §1 · p. 1, §2.1 · p. 2Method
The method combines NyTT with a Sinkhorn drifting loss that aligns enhanced-speech and unpaired clean-speech distributions in an SQA model’s latent space.
Source: §1 · p. 1, §3.3 · p. 3Results
The combination improves several perceptual metrics over mean-teacher NyTT on VoiceBank-DEMAND and demonstrates effectiveness in domain adaptation on real CHiME-3 recordings.
Source: §5 · p. 4Takeaways & Limitations
Unpaired clean speech can provide distribution-level supervision alongside NyTT without requiring clean counterparts for the noisy training data.
Source: §1 · p. 1NyTT can retain residual noise, and mean-teacher variants do not explicitly encourage enhanced speech to follow the clean-speech distribution.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
In speech enhancement, standard supervised training requires pairs of noisy and clean speech. Such pairs are typically created by artificially degrading clean speech, leading to a domain mismatch between synthetic training data and real recordings. Noisy target training (NyTT) mitigates this limitation by training an enhancement model to predict a noisy recording from a noisier version generated by adding extra noise. However, since the training target is noisy, NyTT might learn to leave residual noise in the enhanced output. In this paper, we address this issue in NyTT by incorporating a Sinkhorn drifting loss that matches the distribution of the enhanced speech to that of unpaired clean speech. The Sinkhorn drifting loss attracts enhanced speech toward high-density regions of the clean-speech distribution while repelling it from the current enhanced-speech distribution in latent space. This loss uses only unpaired clean speech rather than the clean counterpart of each degraded recording, preserving the weakly-supervised nature of NyTT. Experimental results on VoiceBank-DEMAND and CHiME-3 real recordings demonstrate the efficacy of the proposed combination.
1. INTRODUCTION
Synthetic clean–noisy training pairs can mismatch real acoustic conditions, while noisy-target training can leave residual noise and does not explicitly align enhanced and clean-speech distributions. The paper combines NyTT with unpaired Sinkhorn drifting to address these limitations while retaining weak supervision.
1. INTRODUCTION
- Synthetic clean–noisy pairs can mismatch real-world acoustic conditions because clean speech is artificially degraded for supervised training.The paper identifies this mismatch as a potential cause of performance degradation on in-the-wild recordings.Source: §1 · p. 1
- NyTT trains on real noisy recordings but can retain residual noise and does not explicitly align enhanced-speech and clean-speech distributions.Mean-teacher variants mitigate residual noise but still do not explicitly encourage distribution matching.Source: §1 · p. 1
- Prior unpaired drifting improved perceptual quality on VoiceBank-DEMAND but degraded SI-SDR and WER even relative to noisy input.The paper uses this contrast to motivate combining distribution learning with NyTT’s signal-level training.Source: §1 · p. 1
- The proposed method adds a Sinkhorn drifting loss to NyTT, matching enhanced speech to unpaired clean speech without requiring clean counterparts for each noisy recording.The authors expect NyTT’s waveform-level supervision and distribution-level supervision to complement each other.Source: §1 · p. 1
2. RELATED WORK
Clean-target training relies on synthetic pairs that may mismatch real recordings, motivating weakly supervised alternatives. NyTT uses noisy targets but can retain noise, while unpaired clean-speech methods seek distribution alignment with distinct trade-offs.
2.1. Noisy target training (NyTT) and its variants
- Synthetic clean–noisy training pairs can mismatch real-world acoustic conditions and lead to performance degradation on real recordings.Clean speech is unavailable in real noisy environments, so training pairs are artificially generated by adding noise to recorded clean speech.Source: §2.1 · p. 2
- NyTT uses in-domain noisy speech as its target and adds extra noise to the model input, but its noisy target can leave residual noise in enhanced output.The observation and added noise are difficult to separate when they come from the same distribution.Source: §2.1 · p. 2
- Mean-teacher NyTT updates teacher parameters by exponential moving average and uses teacher-enhanced observations as targets, but does not explicitly match clean-speech distributions.The teacher’s target suppresses observation noise and provides a clearer pseudo-target.Source: §2.1 · p. 2
2.2. Speech enhancement training with unpaired clean speech
- Unpaired-clean methods align enhanced and clean-speech distributions, but adversarial training can constrain denoising and require multiple discriminators.The cited approaches rely on sensitive adversarial training; one method adds noise prediction to partially mitigate the data-fidelity issue.Source: §2.2 · p. 2
3. METHODS
The methods frame enhancement as distribution matching in latent space and use Sinkhorn-induced drift within NyTT. The final objective combines sample-level NyTT and mean-teacher supervision with unpaired clean-speech distribution matching, using UrgentMOS embeddings.
3.1. Speech enhancement based on drifting models
- Drifting models train enhanced-speech features to move toward clean-speech features, with an attraction term and a self-correction term for the current enhanced distribution.The field vanishes when the enhanced and clean distributions match; clean samples may be drawn independently of noisy inputs.Source: §3.1 · p. 2
3.2. Drifting field induced by Sinkhorn divergence
- Sinkhorn drifting derives its field from Sinkhorn-divergence gradient flows, using globally balanced transport weights while retaining attraction–repulsion structure.Under mild assumptions, zero drift implies an exact distribution match; Sinkhorn drifting also improves training stability.Source: §3.2 · pp. 2–3
3.3. NyTT with Sinkhorn drifting loss
- The final objective combines NyTT reconstruction, mean-teacher supervision, and unpaired Sinkhorn drift to retain sample-level faithfulness while matching enhanced and clean speech distributions.Clean speech is sampled independently of the noisy input, so paired clean targets are not required.Source: §3.3 · p. 3
- The method uses UrgentMOS utterance-level embeddings because matching unpaired phonetic features may be unreasonable, while maximizing quality scores directly ignores clean-speech diversity.The authors leave detailed comparison with naïve score maximization for future work.Source: §3.3 · p. 3
4. EXPERIMENTS
The experiments evaluate the proposed NyTT and unpaired Sinkhorn drifting combination on VoiceBank-DEMAND and real CHiME-3 recordings. The results report improved perceptual metrics, with CHiME-3 word error rate remaining higher than for unprocessed noisy speech.
4.1. Experiments on VoiceBank-DEMAND
- The VoiceBank-DEMAND experiments use 28 training speakers, unpaired clean utterances from EARS, and a DPRNN enhancement model with about 1.5 million parameters.The unpaired clean speech is selected to mismatch VoiceBank-DEMAND in speaker and linguistic content.Source: §4.1 · p. 3
- For VoiceBank-DEMAND CTT, paired and unpaired Sinkhorn drifting improve PESQ, DNSMOS, and UTMOS over standard negative-SNR training, with unpaired drifting preserving reasonable SI-SDR.Table 1 reports mean scores with 95% confidence intervals; its orange rows denote CTT.Source: §4.1 · p. 4, Table 1 · p. 4
- On VoiceBank-DEMAND, NyTT+ with unpaired drifting raises SI-SDR from 6.60 dB to 18.58 dB versus original DriftSE while retaining a similar DNSMOS score.The authors attribute improved faithfulness to NyTT+ complementing unpaired drifting, which alone is insufficient to ensure faithfulness to an observed signal.Source: §4.1 · p. 4, Table 1 · p. 4
- On VoiceBank-DEMAND, original drifting with SQA latents and Sinkhorn drifting with DistillHuBERT latents slightly improve UTMOS over NyTT+ but lag substantially behind the proposed combination.The ablation results suggest that both Sinkhorn drifting and the SQA latent representation contribute to the improvement.Source: §4.1 · p. 4
4.2. Experiments on CHiME-3 real recordings
- On CHiME-3, adding unpaired Sinkhorn drifting to NyTT+ improves DNSMOS and UTMOS over NyTT+, but WER remains higher than for unprocessed noisy speech.The DNSMOS and UTMOS improvements over NyTT+ are statistically significant under paired t-tests (p < 10^-10).Source: §4.2 · p. 4, Table 2 · p. 4
5. CONCLUSION
The paper combines NyTT with unpaired Sinkhorn drifting, aligning enhanced-speech and clean-speech distributions in an SQA model's latent space. Experiments report improved perceptual metrics over NyTT with mean-teacher training on VoiceBank-DEMAND and effectiveness in CHiME-3 domain adaptation.
5. CONCLUSION
- The proposed method combines NyTT with unpaired Sinkhorn drifting to align enhanced-speech and clean-speech distributions in an SQA model's latent space.The conclusion reports improved perceptual metrics over NyTT with the mean-teacher framework on VoiceBank-DEMAND and effectiveness in adaptation on real CHiME-3 recordings.Source: §5 · p. 4