Source-linked AI summary
Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks
Santiago Pascual, Mirco Ravanelli, Joan Serrà, Antonio Bonafonte, Yoshua Bengio
TL;DR
Speech representation learning lacks an easy way to capture the hierarchical structure of long, variable-length speech without costly annotation. The paper introduces PASE, which jointly trains one encoder through multiple self-supervised workers; experiments show robust, transferable features that outperform traditional hand-crafted features across speech tasks and acoustic conditions.
Problem
Speech is difficult to represent without supervision because it is high-dimensional, long, variable-length, and hierarchically structured, while annotated corpora are costly.
Method
PASE jointly trains a single speech encoder with multiple lightweight workers solving complementary regression and binary discrimination tasks.
Results
PASE learns robust representations from raw speech that outperform traditional hand-crafted features across speaker identification, emotion classification, and automatic speech recognition, including noisy and reverberant speech.
Takeaways & Limitations
The embeddings retain information about speaker identity, phonemes, and emotional cues and can transfer across datasets, tasks, and acoustic conditions.
Abstract
from arXiv · showhide
Learning good representations without supervision is still an open issue in machine learning, and is particularly challenging for speech signals, which are often characterized by long sequences with a complex hierarchical structure. Some recent works, however, have shown that it is possible to derive useful speech representations by employing a self-supervised encoder-discriminator approach. This paper proposes an improved self-supervised method, where a single neural encoder is followed by multiple workers that jointly solve different self-supervised tasks. The needed consensus across different tasks naturally imposes meaningful constraints to the encoder, contributing to discover general representations and to minimize the risk of learning superficial ones. Experiments show that the proposed approach can learn transferable, robust, and problem-agnostic features that carry on relevant information from the speech signal, such as speaker identity, phonemes, and even higher-level features such as emotional cues. In addition, a number of design choices make the encoder easily exportable, facilitating its direct usage or adaptation to different problems.
1. Introduction
Speech representation learning is difficult because annotation is costly and single-task supervision can limit exportability. PASE addresses this by jointly learning multiple self-supervised views to produce robust, transferable features.
- Motivation: Annotated corpora are costly to collect, while supervised representations can become biased toward the task used for training.These limitations motivate learning from unlabeled speech and seeking representations that generalize across applications.
- Motivation: Speech is challenging for self-supervision because it forms high-dimensional, long, variable-length sequences with hierarchical structure.Relevant structure includes phonemes, syllables, and words.
- PASE proposal: PASE jointly tackles multiple self-supervised tasks with cooperating neural networks to impose consensus constraints on one speech representation.Different tasks provide distinct views or soft constraints, reducing reliance on superficial signal features.
- Results: Experiments report that PASE representations outperform traditional hand-crafted features across speaker identification, emotion classification, and automatic speech recognition.The representations are learned directly from raw speech waveforms.
- Results: PASE features also work on speech corrupted by substantial noise and reverberation, despite being learned from clean data.The encoder is designed to be efficient, fully parallelizable, and usable as a transferable feature extractor.
2. Problem-agnostic Speech Encoder
PASE uses a fully convolutional encoder and multiple lightweight workers that jointly learn complementary speech information. Regression workers model signal features, while discrimination workers learn higher-level relations, and the encoder is trained through averaged worker losses.
- Architecture: PASE combines a fully convolutional speech encoder with seven MLP workers that cooperatively solve self-supervised tasks.The architecture encodes raw speech before distributing the representation to the workers.
- Encoder: The encoder uses SincNet filters followed by seven convolutional blocks, decimating waveform samples into speech frames.The resulting representation has a 10 ms stride at 16 kHz and about a 150 ms receptive field.
- Regression workers: Regression workers reconstruct or predict waveform, spectrum, MFCC, and prosodic features at increasing levels of abstraction.Waveform reconstruction uses an upsampling decoder and L1 minimization, while other regression workers use target features and MSE.
- Discrimination workers: Three binary discrimination workers learn higher-level abstractions by distinguishing sampled positive and negative examples from encoded speech.The tasks include local and global information maximization and sequence predicting coding.
- Training and use: Encoder and workers are jointly optimized with the average worker loss, so encoder updates compromise among all self-supervised objectives.The learned encoder can later be frozen for classification or fine-tuned jointly with a classifier.
3. Corpora and Tasks
PASE is pretrained on clean LibriSpeech speech and evaluated through supervised speaker identification, emotion classification, and automatic speech recognition tasks. The evaluation includes limited-data speaker identification to assess learned representations across applications.
- Pretraining corpus: Self-supervised PASE training uses approximately 15 seconds of speech for each of 2,484 LibriSpeech speakers.Training sentences were randomly selected from the specified LibriSpeech portion.
- Downstream tasks: The learned representations are evaluated on speaker identification, speech emotion classification, and automatic speech recognition.These tasks test information related to speaker characteristics, emotion, and speech content.
- Downstream tasks: Speaker identification uses VCTK with 109 speakers and only 11 seconds of training speech per speaker.The subset is intended to make the task more challenging and realistic.
4. Results
The experiments show that combining all self-supervised workers produces the strongest overall representations, with worker contributions varying by task. PASE features outperform standard features across speech classification tasks and transfer effectively to noisy, reverberant speech.
- 4.1. Worker Ablation: All workers contribute: removing any worker never improves performance, while the complete system achieves the best results.Some workers help broadly, whereas others are more application-dependent.
- 4.1. Worker Ablation: The MFCC worker is especially crucial because it preserves prior knowledge about important speech-frequency bands.
- 4.1. Worker Ablation: Prosody improves emotion recognition by +131% in relative error, while SPC improves speaker and emotion recognition by +16% and +13% in relative error, respectively.Prosody features encode intonation, expressiveness, and voicing; SPC captures longer contextual information.
- 4.2. Comparison with Standard Features: PASE-FineTuned consistently achieves the best performance across the considered tasks and classifiers, exceeding MFCCs and FBANKs.PASE-Frozen also usually performs better than the traditional features.
- 4.2. Comparison with Standard Features: 97.7% emotion-recognition accuracy exceeds the reported 80% human-level performance, while phoneme accuracy reaches 85.3% on TIMIT.The phoneme result corresponds to a 14.7% error rate.
- 4.3. Transferability: On DIRHA, PASE outperforms competing systems under substantial noise and reverberation, including with its frozen encoder.PASE-FineTuned also surpasses the previous SincNet results.
5. Conclusion
The paper proposes a multi-task self-supervised approach and an exportable encoder that converts waveforms into latent embeddings. These embeddings retain information useful for speaker identity, phonemes, emotion, and transfer across datasets, tasks, and acoustic conditions.
- PASE combines multiple self-supervised tasks to learn speech representations and provides an exportable encoder for waveform-to-embedding conversion.
- The learned embeddings carry information about speaker identity, phonemes, and emotional cues.
- The embeddings transfer across different datasets, tasks, and acoustic conditions.The framework is also described as extensible to semi-supervised learning and additional self-supervised tasks.