Source-linked AI summary
Multi-task self-supervised learning for Robust Speech Recognition
Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Joao Monteiro, Jan Trmal, Yoshua Bengio
TL;DR
Meaningful speech representations remain difficult to learn from unlabelled audio because speech varies across hierarchical time scales, speakers, languages, and acoustic conditions. PASE+ combines online distortion, a revised convolutional-QRNN encoder, and refined workers, outperforming prior PASE and standard acoustic features while transferring to mismatched environments.
Problem
Self-supervised learning still faces difficulty extracting useful speech representations from unlabelled audio amid hierarchical structure and substantial variability.
Method
PASE+ combines online speech distortion, a convolutional encoder with QRNN, and workers solving self-supervised tasks using clean-signal targets.
Results
PASE+ significantly outperforms prior PASE and standard acoustic features across TIMIT, DIRHA, and CHiME-5 evaluations.
Takeaways & Limitations
PASE+ learns transferable representations that remain effective in highly mismatched and realistic noisy acoustic environments.
Takeaways & Limitations
The study identifies future evaluation on other downstream tasks, including speaker, emotion, language, and sequence-to-sequence recognition.
Abstract
from arXiv · showhide
Despite the growing interest in unsupervised learning, extracting meaningful knowledge from unlabelled audio remains an open challenge. To take a step in this direction, we recently proposed a problem-agnostic speech encoder (PASE), that combines a convolutional encoder followed by multiple neural networks, called workers, tasked to solve self-supervised problems (i.e., ones that do not require manual annotations as ground truth). PASE was shown to capture relevant speech information, including speaker voice-print and phonemes. This paper proposes PASE+, an improved version of PASE for robust speech recognition in noisy and reverberant environments. To this end, we employ an online speech distortion module, that contaminates the input signals with a variety of random disturbances. We then propose a revised encoder that better learns short- and long-term speech dynamics with an efficient combination of recurrent and convolutional networks. Finally, we refine the set of workers used in self-supervision to encourage better cooperation. Results on TIMIT, DIRHA and CHiME-5 show that PASE+ significantly outperforms both the previous version of PASE as well as common acoustic features. Interestingly, PASE+ learns transferable representations suitable for highly mismatched acoustic conditions.
1. INTRODUCTION
Self-supervised speech learning must recover information across multiple time scales despite substantial speaker, linguistic, and acoustic variability. PASE+ addresses this challenge with distortion-aware training, a revised encoder, and cooperative self-supervised workers.
- Speech contains hierarchical information from samples to semantic content across different time scales, complicating unsupervised representation learning.
- PASE+ improves PASE with online speech distortions, a convolutional-QRNN encoder, and workers estimating speech transformations from clean-signal targets.
- Clean-signal targets paired with distorted inputs encourage implicit denoising and distortion-invariant representations without manual annotations.
- PASE+ is designed for robust speech recognition in noisy and reverberant environments and significantly boosts performance on challenging tasks.
2. SELF-SUPERVISED LEARNING WITH PASE+
PASE+ combines online contamination, convolutional and recurrent encoding, skip connections, and diverse workers to learn robust speech representations from raw signals.
- Architecture: PASE+ supports frozen feature extraction or supervised integration and uses twelve workers sharing encoded features to solve self-supervised tasks.
- Online speech contamination: Online contamination applies multiple probabilistic distortions during self-supervised training, including reverberation, noise, masking, clipping, and overlapped speech.
- PASE+ encoder: Skip connections sum projected and downsampled intermediate convolutional features, combining abstraction levels and improving gradient flow.
- PASE+ encoder: QRNN layers efficiently model long-term dependencies using convolutional multiplicative gates and minimalist recurrent pooling.
- Self-supervised workers: Workers estimate speech transformations and related features, including FBANKs, Gammatone features, derivatives, neighboring frames, and longer analysis windows.
- Binary tasks: Binary workers capture higher-level information by training classifiers on anchor, positive, and negative encoded representations.
- Binary tasks: LIM samples positives within an anchor sentence and negatives from another sentence, enabling discrimination based partly on speaker identity.
- Binary tasks: GIM averages representations from two-second chunks and contrasts sentences to learn sequence-level information complementary to LIM.
3. CORPORA AND ASR SETUP
PASE+ is pretrained on LibriSpeech and evaluated with out-of-domain speech-recognition datasets spanning simulated and real noisy, reverberant, overlapping environments.
- Self-supervised pretraining uses 50 hours of LibriSpeech, while target recognition experiments use out-of-domain datasets.
- Evaluation covers clean and contaminated TIMIT, simulated domestic-environment DIRHA, and real dinner-party recordings from CHiME-5.
- The experiments use hybrid HMM-DNN recognizers, with toolkit and acoustic-model configurations varying across datasets.
4. RESULTS
PASE+ improves robust speech recognition through online distortion, a revised encoder, expanded self-supervised workers, and higher-dimensional representations. It outperforms the previous PASE version and standard acoustic features across clean, noisy, reverberant, and mismatched conditions.
- Model ablation: Online distortions improve performance in both noisy and clean TIMIT, acting as regularization when the supervised classifier uses limited data.The contamination module generates training variants with reverberation, additive noise, masking, clipping, and overlapped speech.
- Model ablation: QRNN substantially improves noisy and reverberated conditions by enabling efficient longer-context modeling, while 256-dimensional embeddings further improve representation capacity.Skip connections and an expanded worker set also contribute improvements in the ablation sequence.
- PASE+ achieves relative improvements of 9.5% in clean TIMIT and 17.7% in noisy conditions over the previous PASE version.
- Standard speech features: On DIRHA, frozen PASE+ yields a 13.5% relative WER improvement over the best hand-crafted feature performance.PASE+ also outperforms the supervised end-to-end PASE baseline, and fine-tuning adds a 3.1% relative improvement over frozen features.
- Standard speech features: On CHiME-5, frozen PASE+ improves over MFCC by 3.0% relatively and combines complementarily with MFCC and MFCC+ivectors for 1.2% and 3.7% gains.The results support transfer to highly mismatched and realistic acoustic environments.
5. CONCLUSION
The paper presents PASE+ as a multi-task self-supervised approach for robust speech recognition. Across different tasks, it outperforms standard acoustic features and transfers to unseen noisy environments, while future work extends beyond pure self-supervision and speech recognition.
- PASE+ combines online speech distortion, a convolutional encoder with QRNN, and workers solving multiple self-supervised problems.
- PASE+ significantly outperforms standard acoustic features with frozen weights and gains further improvements when optimized with the target acoustic-model objective.
- Training on artificially distorted LibriSpeech enables good performance in unseen, realistic, and challenging noisy acoustic scenarios.
- Future work will explore semisupervised workers and downstream tasks including speaker, emotion, language, and sequence-to-sequence speech recognition.