Source-linked AI summary
Room Impulse Response Embeddings for Speech Enhancement in Noisy and Reverberant Environments
Adrian Meise, Reinhold Haeb-Umbach
TL;DR
Noisy-reverberant speech complicates learning RIR representations, while prior spatial self-supervised work has less often addressed speech enhancement. The paper trains embeddings in three stages, ending with a teacher-student procedure that matches reverberant teacher embeddings from noisy-reverberant input. The embeddings retain acoustic-room information, and conditioning enhancement models yields consistent gains across reverberant and noisy-reverberant speech.
Problem
Spatial self-supervised representations are less studied for speech enhancement, and noise can shift and disperse RIR embeddings in noisy recordings.
Source: §1 · p. 1Method
A three-stage approach trains RIR embeddings on reverberant speech, continues contrastive learning on noisy-reverberant speech, then distills reverberant-input teacher embeddings to a student using noisy-reverberant input.
Source: §1 · p. 1, §2.3 · p. 2Results
Conditioning discriminative deep-filtering enhancement models on the embeddings yields consistent gains across considered metrics for reverberant and noisy-reverberant speech.
Source: §1 · p. 1, §5.2 · p. 4Takeaways & Limitations
T60 and DRR remain linearly accessible from the student embeddings for reverberant and noisy-reverberant input.
Source: §6 · p. 4Denoising artifacts can affect RIR-encoder training, making the resulting embeddings dependent on the denoising system.
Source: §2 · p. 2
Abstract
from arXiv · showhide
We propose a self-supervised approach for learning room impulse response (RIR) representations from single-channel noisy-reverberant speech. It consists of first training on reverberant data, then on noisy-reverberant data, and finally with a teacher-student approach, where the student learns to replicate the teacher's embeddings when given a noisy version of the reverberant input. We assess their representational capabilities by estimating acoustic room parameters from them. Conditioning a discriminative speech enhancement model on the derived embeddings yields consistent gains across all evaluated metrics, including downstream word error rate, for both reverberant and noisy-reverberant speech.
1. INTRODUCTION
Prior work shows that room and propagation information can support dereverberation, but spatial self-supervised representations remain less studied for speech enhancement. This work learns noise-robust RIR embeddings from noisy-reverberant speech and evaluates them for acoustic-parameter estimation and speech enhancement.
1. INTRODUCTION
- Room impulse responses, transfer functions, and sound-propagation parameters can provide side information that helps models account for reverberant acoustics.The paper situates self-supervised learning as an alternative to supervised estimation or assuming this information is known.Source: §1 · p. 1
- Spatial self-supervised representations have been explored for acoustic-parameter estimation, room classification, RIR estimation, and RIR generation, but their usefulness for speech enhancement is less investigated.Prior work also used RIR embeddings to improve ASR and diffusion-based dereverberation; that enhancement analysis considered only a generative diffusion approach.Source: §1 · p. 1
- Noise can accompany multipath propagation in distant-microphone recordings, and prior work reports noise-related shifts and dispersion in RIR embeddings.The robustness of self-supervised features to noise remains an open challenge in speech and audio processing.Source: §1 · p. 1
- Conditioning a discriminative deep-filtering enhancement model on the embeddings yields consistent improvements for reverberant and noisy-reverberant speech.The paper evaluates representation quality through acoustic-parameter estimation before testing enhancement.Source: §1 · p. 1
2. SSL RIR REPRESENTATION LEARNING
The approach learns RIR representations from reverberant speech, adapts them to noisy-reverberant input, then trains a student to reproduce the reverberant-only teacher's embeddings. This avoids relying on a separate denoising system whose artifacts could affect the learned embeddings.
2. SSL RIR REPRESENTATION LEARNING
- Denoising before RIR encoding can introduce artifacts that affect encoder training and make embeddings dependent on the denoising system.The approach instead trains directly on noisy-reverberant input.Source: §2 · pp. 1–2
2.1. Contrastive learning of RIR representations
- Contrastive training pairs different utterances convolved with the same RIR as positives and the same utterance convolved with different RIRs as negatives.The objective encourages embeddings to ignore source speech and represent RIR characteristics, using weighted InfoNCE and cosine similarity for negative pairs.Source: §2.1 · p. 2
2.2. Noise-robust RIR representation learning
- The second stage continues contrastive training on noisy-reverberant speech, varying noise across same-RIR positives and usually holding it constant across different-RIR negatives.This builds on the reverberant-trained model and encourages the encoder to distinguish RIRs rather than noise samples.Source: §2.2 · p. 2
2.3. Teacher-student training
- A frozen stage-1 teacher supplies reverberant-input embeddings for the stage-2 student to match from noisy-reverberant input.The student is trained with embedding MSE; validation retains contrastive loss as the selection criterion. Figure 1 depicts this teacher-student setup.Source: §2.3 · p. 2, Figure 1 · p. 2
3. EXPERIMENTAL SETUP
Experiments use EARS speech and a Conformer-based RIR encoder, training in stages with reverberant and noisy-reverberant data. Evaluation includes unseen rooms and noise types, acoustic-parameter estimation, and speech enhancement.
3. EXPERIMENTAL SETUP
- Training uses 100 hours of EARS clean speech, EARS-Reverb, and WHAM! noise at −2.5 to 17.5 dB SNR, with additional noise sources in teacher-student training.The encoder is a 10-layer Conformer with 256-dimensional ℓ2-normalized embeddings, initialized from a pretrained checkpoint and fine-tuned on EARS-Reverb.Source: §3 · p. 2
- Generalization tests use measured BUT-ReverbDB rooms and DEMAND noise at 0 to 20 dB SNR, with rooms and noise types not seen during training.The test sets are generated from the clean EARS test split.Source: §3 · pp. 2–3
- A 514-parameter linear estimator predicts T60 and DRR from embeddings, compared with a 270,210-parameter 1D-CNN operating on waveforms.Performance is reported using MAE and Spearman correlation; Table 1 covers raw-waveform and stage-wise embedding estimation, with parameter counts and best results indicated.Source: §3 · p. 3, Table 1 · p. 3
4. EVALUATION
The student retains acoustic-room information under noisy-reverberant input and yields the strongest reported embedding structure. Stage-1 embeddings do not retain T60 and DRR information on noisy-reverberant input, while conditioning enhancement models improves parameter estimates and embedding organization across stages.
4. EVALUATION
- All embedding-based models estimate T60 and DRR better than the CNN estimator on reverberant input, and stages 2 and 3 retain parameter information for that input.The authors note that more advanced estimators could improve this task.Source: §4 · p. 3
- More than 1 dB improvement in DRR estimation over stage 2 is achieved by the student on noisy-reverberant data, with correlation similar to reverberant-input results.Stage 1 fails to represent T60 and DRR on noisy-reverberant data, while stage 2 contains relevant parameter information.Source: §4 · p. 3
- Student embeddings form distinguishable, separated room clusters resembling the teacher's reverberant-input space, whereas stage 2 shows scattered and interspersed samples.Stage 1 forms room clusters on reverberant input but loses that grouping when directly applied to noisy-reverberant input.Source: §4 · p. 3
5. APPLICATION TO SPEECH ENHANCEMENT
The section evaluates RIR-embedding conditioning for deep-filtering speech enhancement on reverberant and noisy-reverberant data. Conditioning produces consistent improvements across reported metrics, with results also assessed using weak and strong ASR models.
5.1. Deep filtering
- DeepFilterNet predicts filters for degraded input but has limited dereverberation; the two-step approach adds late-reverberation filtering, motivating RIR conditioning of filter estimates.Source: §5.1 · p. 3
- FiLM layers after each decoder block use embedding-derived feature scaling and shifting to influence filter-coefficient estimates.Source: §5.1 · p. 4
- Evaluation pairs stage-1 embeddings with reverberant data and stage-3 student embeddings with noisy-reverberant data, measuring speech quality, intelligibility, and weak/strong ASR WER.Source: §5.1 · p. 4
5.2. Enhancement results
- Conditioning improves all reported metrics on both datasets; for reverberant data, strong-ASR WER improves over input, while noisy-reverberant FiLM gains are larger for strong than weak ASR.Table 2 reports reverberant and noisy-reverberant performance, with WER for weak and strong ASR models.Source: Table 2 · p. 4, §5.2 · p. 4
6. CONCLUSIONS
The work develops noise-robust RIR embeddings through three training stages. The embeddings retain room-parameter information and yield consistent gains when used to condition deep-filtering enhancement models.
6. CONCLUSIONS
- A three-stage curriculum trains RIR embeddings on reverberant speech, then noisy-reverberant speech, and finally uses teacher-student training to reproduce teacher embeddings from noisy inputs.Source: §6 · p. 4
- T60 and DRR are linearly accessible from the embeddings and are maintained by the student for reverberant and noisy-reverberant input.Source: §6 · p. 4
- Conditioning deep-filtering enhancement models on RIR embeddings yields consistent gains across considered metrics, including weak- and strong-ASR word error rates.Source: §6 · p. 4
- Future work will investigate other discriminative and multi-stage models, and extend the representations to multi-channel, frame-wise use in dynamic scenarios.Source: §6 · p. 4