Source-linked AI summary
SNOAR: Self-training with Noise filtering, Observation adding, and Remixing for Speech Denoising Without Clean References
Samuele Cornell, Matthew Maciejewski
TL;DR
Noisy-recording denoising methods avoid the sim-to-real gap but still fall short of supervised performance, while RemixIT can propagate teacher errors such as suppressed speech. SNOAR removes residual noise from teacher speech estimates with DNF, then adds part of the noisy observation to restore suppressed speech and attenuate artifacts. On real CHiME-3 recordings, SNOAR lowers three teachers’ WER by 1.3 to 3.7 points and raises UTMOS by 0.5 to 0.8; RemixIT degrades every teacher’s perceptual scores.
Problem
Noisy-recording denoising methods avoid the sim-to-real gap but still fall short of supervised performance, while RemixIT can propagate teacher errors such as suppressed speech.
Source: §1 · p. 1Method
SNOAR removes residual noise from teacher speech estimates with DNF, then adds part of the noisy observation to restore suppressed speech and attenuate artifacts.
Source: §1 · p. 1Results
On real CHiME-3 recordings, SNOAR lowers three teachers’ WER by 1.3 to 3.7 points and raises UTMOS by 0.5 to 0.8; RemixIT degrades every teacher’s perceptual scores.
Source: §1 · pp. 1–2Takeaways & Limitations
SNOAR raises UTMOS by 0.5 to 0.8, with the largest gain for the teacher trained without any clean speech, while RemixIT degrades every teacher’s perceptual scores.
Source: §1 · p. 2For observation adding with γ = 0.2, λ must be below 0.2, corresponding to at least about 14 dB teacher noise reduction; otherwise the loop risks collapse to λ = 1.
Source: §2.2 · p. 2
Abstract
from arXiv · showhide
Unsupervised speech denoising methods such as MixIT and noisy-target training can be trained directly on noisy recordings, avoiding the sim-to-real gap of synthetic training data, but remain well below supervised performance. Teacher--student self-training, as in RemixIT, narrows this gap by remixing a teacher's estimates into new mixtures that supervise a student; however, since the student is trained only toward the teacher's estimates, teacher errors, in particular over-suppressed speech, are never recovered and propagate as the teacher is updated from the student. We propose SNOAR, a self-training framework that corrects the teacher's estimates before use: the noise leaked into its speech estimate is removed by differential noise filtering (DNF), and part of the noisy observation is added back in to restore suppressed speech and attenuate artifacts. We show that the two corrections depend on each other, as without DNF the loop collapses to the identity mapping. Compared to RemixIT, SNOAR is more stable along training and outperforms it in signal-based, perceptual, and word error rate metrics, reaching 97\% of the SI-SDR improvement of a fully-supervised topline on WHAM!; on real CHiME-3 recordings, it improves every metric of three different teachers, while RemixIT degrades their perceptual quality.
1. INTRODUCTION
Supervised denoising is limited by the sim-to-real gap, while noisy-recording methods still trail supervised performance. SNOAR corrects teacher estimates before self-training, and outperforms RemixIT on synthetic WHAM! and real CHiME-3 recordings.
1. INTRODUCTION
- Supervised denoising relies on synthetic mixtures because real-world recordings lack clean paired references, restricting training to domains with studio-quality clean speech.The mismatch between synthetic mixtures and real recordings is the sim-to-real gap, which degrades deployment performance.Source: §1 · p. 1
- Noisy-recording approaches such as NyTT and MixIT need only in-domain noisy speech and noise-only clips, but remain below supervised performance.The authors argue these approaches are more scalable because noise-only clips can be obtained from the same recordings.Source: §1 · p. 1
- RemixIT trains students on remixed teacher estimates, so suppressed or distorted speech is reproduced and can pass into successive teacher generations.The authors describe this error reinforcement as the regression analogue of confirmation bias in classification pseudo-labeling.Source: §1 · p. 1
- SNOAR corrects teacher estimates before using them as self-supervision: DNF removes leaked noise, then observation adding restores some suppressed speech and attenuates artifacts.The two corrections depend on each other; without DNF, observation adding drives the loop toward the identity mapping.Source: §1 · p. 1
- On WHAM!, SNOAR improves a MixIT teacher from 8.3 to 13.3 dB SI-SDR, within 0.43 dB of the supervised topline, and leads unsupervised systems in WER and perceptual scores.On CHiME-3, it lowers three teachers’ WER by 1.3 to 3.7 points and raises their UTMOS by 0.5 to 0.8, while RemixIT degrades every teacher’s perceptual scores.Source: §1 · pp. 1–2
2. METHOD
SNOAR builds on noisy-recording denoising and RemixIT by correcting teacher estimates before using them as student targets. It combines differential noise filtering (DNF), observation adding (OA), and cross-recording remixing, with the two corrections shaping the noise dynamics across training.
2.1. Speech denoising without clean references
- NyTT and MixIT train without clean references by adding a second noise recording and using the mixture and added noise as targets, but cannot separate same-distribution noises.Differential noise filtering uses the model’s noise estimate to remove the residual noise from its speech estimate.Source: §2.1 · p. 2
2.2. SNOAR: self-training with noise filtering, observation adding and remixing
- The method retains RemixIT’s remixing and loss while correcting its speech target, and can use supervised, unsupervised, or hybrid teachers.At inference, two-output models use DNF, while three-output MixIT-derived models use the speech output alone.Source: §2.2 · pp. 2–3
- OA reduces suppressed speech and model artifacts by adding part of the original observation, but returns a fraction of its noise to the self-supervision.In vanilla RemixIT, the teacher’s retained-speech and artifact components pass to the student unchanged.Source: §2.2 · p. 2
- With DNF, the residual-noise update depends on OA rather than the previous λ; for γ = 0.2, its lower equilibrium is approximately 0.042, corresponding to approximately 27.6 dB noise reduction.The passage says this is far more noise reduction than any of the authors’ teachers achieves.Source: §2.2 · p. 2
- SNOAR filters the teacher’s speech estimate against its noise estimate with DNF, then combines the filtered estimate with the noisy observation through OA before remixing it with another recording’s noise estimate.The student is trained to reconstruct the corrected target; the teacher may track the student through an EMA.Source: §2.2 · pp. 2–3, Figure 1 · p. 3
3. EXPERIMENTAL SETUP
Experiments use WHAM! mixtures and real CHiME-3 recordings, with TF-GridNet systems trained and evaluated using reference-free checkpoint selection. The setup reports multiple signal, perceptual, and recognition metrics.
Data
- WHAM! training uses the official 16 kHz anechoic single-speaker mixtures, with separate noise pools for the recorded and added noise; testing uses the untruncated max version.The authors note that WHAM! results are not directly comparable with.Source: §3 · p. 3
- CHiME-3 adaptation uses real channel-5 recordings, with development-set checkpoint selection and testing on et05 real.Added noise comes from the speech-free official background pool recorded in separate sessions.Source: §3 · p. 3
Model and training
- All systems use TF-GridNet and negative SI-SDR loss; self-training uses an EMA teacher, and checkpoints are selected with UTMOS rather than early stopping.WHAM! self-training runs for 150k steps; CHiME-3 runs for 7.5k steps. The supervised WHAM! topline alone uses oracle SI-SDR for selection and hyperparameter tuning.Source: §3 · p. 3
Evaluation
- Outputs are normalized to −26 dBov before evaluation, which includes SI-SDR, STOI, PESQ, DNSMOS, UTMOS, and OWSM v4 WER.Table 1 reports WHAM! test results by teacher initialization; its SI-SDR confidence-interval half-width is at most 0.08 dB for every system.Source: §3 · p. 3, Table 1 · p. 3
4. EXPERIMENTAL RESULTS
On WHAM!, SNOAR outperforms RemixIT and approaches the supervised topline; on real CHiME-3 recordings, it improves all three teachers while RemixIT degrades their reference-free scores.
4.1. Results on WHAM!
- 13.25 dB SI-SDR for SNOAR versus 12.10 dB for RemixIT, within 0.43 dB of the supervised topline; SNOAR also attains 10.9% WER and 3.45 UTMOS.The topline scores 9.5% WER and 3.60 UTMOS; the MixIT teacher starts at 2.26 UTMOS.Source: §4.1 · p. 3
- Both students reduce residual noise from λ = 0.218 to about 0.03, while SNOAR retains more speech and introduces fewer artifacts than RemixIT.SNOAR reaches α = 0.949 and r = 0.048, compared with α = 0.939 and r = 0.062 for RemixIT.Source: §4.1 · p. 4, Table 2 · p. 4
- RemixIT peaks after tens of thousands of steps and then degrades on all three metrics, whereas SNOAR generally increases and plateaus.RemixIT’s reported Table 1 results require selecting its checkpoint.Source: §4.1 · p. 4
- Adding observation raises α from 0.944 to 0.949 and lowers r from 0.059 to 0.048; without DNF, the student reproduces its input with λ = 0.697.DNF alone ends near the selected RemixIT checkpoint; γ = 0.5 diverges, while annealing γ from 0.2 to 0.05 yields the same α, λ, and r as fixed γ.Source: §4.1 · p. 4, Table 2 · p. 4
4.2. Results on CHiME-3 real noisy speech
- The CHiME-3 comparison includes one MixIT teacher trained only on real data and two jointly trained teachers using out-of-domain synthetic mixtures.The WHAM! WSJ-based teacher matches CHiME-3’s read-WSJ speech domain and is strongest of the three on every metric.Source: §4.2 · p. 4, Table 3 · p. 4
- SNOAR improves every metric for all three CHiME-3 teachers, reducing WER by 3.7, 1.7, and 1.3 points and raising UTMOS by 0.48, 0.72, and 0.76.RemixIT falls below each teacher on every reference-free metric and lowers WER only for the two weaker teachers.Source: §4.2 · p. 4, Table 3 · p. 4
5. CONCLUSION
SNOAR corrects teacher estimates before using them as self-supervision, combining DNF with observation adding (OA). The corrections interact: on WHAM! it approaches the supervised topline, and on CHiME-3 it improves three teachers where RemixIT falls below them.
5. CONCLUSION
- SNOAR corrects teacher estimates with DNF and OA; without DNF, the self-training loop can collapse to identity, while with DNF it converges from any teacher to residual noise about 27 dB below input noise.On WHAM!, SNOAR comes within 0.43 dB of the supervised topline from a MixIT teacher initially 5 dB below it; on CHiME-3 it improves WER and reference-free metrics for three teachers while RemixIT falls below each on all of them.Source: §5 · p. 4