Source-linked AI summary
Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
Andy T. Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, Hung-yi Lee
TL;DR
Speech representation learning seeks transforms that expose high-level speech information to downstream tasks, while prior approaches commonly rely on unidirectional context or speech quantization. Mockingjay pre-trains bidirectional Transformer encoders on unlabeled speech with masked acoustic reconstruction and supports downstream fine-tuning. It improves phoneme, speaker, and sentiment tasks and outperforms Mel-features even when using only 0.1% of the labeled data.
Problem
Speech contains rich phonetic, linguistic, speaker, and sentiment information that surface features may not make readily accessible to downstream tasks, while prior methods commonly use unidirectional prediction or quantization.
Method
Mockingjay pre-trains multi-layer bidirectional Transformer encoders on unlabeled speech using Masked Acoustic Modeling, reconstructing masked frames from left and right context.
Results
Mockingjay representations improve phoneme classification, speaker recognition, and sentiment classification, outperforming other representations and Mel-features across the reported downstream tasks.
Takeaways & Limitations
The learned representations transfer across tasks and datasets, remain effective with downstream fine-tuning, and support supervised training in low-resource settings.
Abstract
from arXiv · showhide
We present Mockingjay as a new speech representation learning approach, where bidirectional Transformer encoders are pre-trained on a large amount of unlabeled speech. Previous speech representation methods learn through conditioning on past frames and predicting information about future frames. Whereas Mockingjay is designed to predict the current frame through jointly conditioning on both past and future contexts. The Mockingjay representation improves performance for a wide range of downstream tasks, including phoneme classification, speaker recognition, and sentiment classification on spoken content, while outperforming other approaches. Mockingjay is empirically powerful and can be fine-tuned with downstream models, with only 2 epochs we further improve performance dramatically. In a low resource setting with only 0.1% of labeled data, we outperform the result of Mel-features that uses all 100% labeled data.
1. INTRODUCTION
Mockingjay learns speech representations without labels using bidirectional Transformer encoders and masked acoustic reconstruction. It addresses limitations of unidirectional prediction and supports strong downstream performance, including low-resource settings.
- 1. INTRODUCTION: Mockingjay uses unlabeled speech, multi-layer Transformers, and multi-head self-attention to encode past and future contexts jointly.Masked frames are reconstructed and predicted during unsupervised pre-training.
- 1. INTRODUCTION: Unlike prior unidirectional methods, Mockingjay predicts masked current frames from bidirectional context rather than predicting future frames from past sequences.This design alleviates the unidirectionality constraint described for previous approaches.
- 1. INTRODUCTION: 35.2% absolute improvement over Mel-features was reported for phoneme classification accuracy, alongside 28.0% for speaker recognition and 6.4% for sentiment discrimination.The sentiment dataset was unseen during pre-training.
- 1. INTRODUCTION: With 0.36 hours (0.1%) of transcribed speech, Mockingjay outperformed Mel-features trained with 360 hours (100%) of labels.This result supports the reported low-resource benefit of the learned representations.
2. MOCKINGJAY
Mockingjay pre-trains a bidirectional Transformer by masking speech frames and reconstructing them from context. It exposes encoder hidden states as representations for downstream models, with masking strategies designed to encourage global inference.
- 2. MOCKINGJAY: The model uses multi-layer Transformer encoders with multi-head self-attention, residual connections, layer normalization, and feed-forward sub-layers.All encoder layers and sub-layers produce outputs with dimension Hdim.
- 2. MOCKINGJAY: Masked Acoustic Modeling randomly selects 15% of input frames and predicts them from left and right context using an L1 reconstruction loss.A two-layer feed-forward prediction head is used during training and removed afterward.
- 2. MOCKINGJAY: Selected frames are zero-masked 80% of the time, replaced by random frames 10% of the time, and left untouched 10% of the time.The sub-random process is applied utterance-wise to reduce mismatch between training and inference.
- 2. MOCKINGJAY: Consecutive and dynamically sampled masking discourage reliance on local acoustic smoothness and expose varied masking patterns during training.The model is intended to infer global structure rather than only local information.
- 2. MOCKINGJAY: Mockingjay representations are Transformer hidden states, taken mainly from the last layer or combined across layers with a learnable weighted sum.The latter approach is analogous to the reported ELMO-style integration.
3. IMPLEMENTATION
The implementation defines BASE and LARGE Mockingjay configurations using spectrogram reconstruction targets and shared Transformer dimensions. The study also evaluates last-layer, fine-tuned, and all-layer weighted representations.
- 3. IMPLEMENTATION: BASE and LARGE use Mel-features as input and reconstruct either Mel-scale or linear-scale spectrograms.Mel-scale spectrograms are described as more concise acoustic features than linear-scale spectrograms.
- 3. IMPLEMENTATION: Both configurations use Hdim=768, Fdim=3072, and Anum=12, differing in layer number, downsampling factor, and consecutive masking number.The detailed configuration differences are reported in Table 1.
- 3. IMPLEMENTATION: Mockingjay is pre-trained on the LibriSpeech train-clean-360 subset with Adam optimization, warmup and decay, dropout, and batch size 6 on one 1080Ti GPU.Fine-tuning uses a learning rate of 4e-3 and two epochs.
4. EXPERIMENT
Mockingjay representations are evaluated against Mel-features and APC across phoneme classification, speaker recognition, and sentiment classification, including low-resource settings. Across these tasks, Mockingjay generally outperforms the comparison representations, with short fine-tuning providing further gains.
- Mockingjay representations are evaluated on phoneme classification, speaker recognition, and sentiment classification using identical downstream model architectures and hyperparameters.
- The experiments compare BASE, LARGE, BASE-FT2, BASE-FT500, and LARGE-WS Mockingjay settings, alongside APC and 160-dimensional log Mel-features.BASE-FT2 fine-tunes BASE for 2 epochs, while LARGE-WS combines hidden states from all LARGE encoder layers through a learnable weighted sum.
- 4.1. Comparing with other representations: With all 360 hours of labels, BASE and LARGE improve phone classification accuracy over Mel-features by 11.8% and 15.2%, respectively.
- 4.2. Phoneme Classification: BASE-FT2 achieves the highest phone classification performance, improving absolutely over APC and Mel-features by 10.2% and 35.2%, respectively.The gap between BASE-FT2 and BASE-FT500 is only 3.9%, and LARGE-WS improves over LARGE.
- 4.1. Comparing with other representations: With only 0.36 hours (0.1%) of labels, BASE-FT2 reaches 57.9% accuracy, exceeding Mel-features at 49.1% accuracy trained with all 360 hours (100%).APC performs well with full resources but fails to generalize for limited labeled data.
- BASE and LARGE outperform APC and Mel-features for speaker recognition, while BASE-FT2 achieves the highest accuracy; LARGE-WS also improves over LARGE.For sentiment classification on MOSEI, LARGE-WS achieves the highest score without fine-tuning.
5. CONCLUSION
Mockingjay representations encode phonetic, speaker, and sentiment information and improve performance across downstream tasks. They also show promise in low-resource settings and transfer across tasks and datasets.
- Mockingjay representations contain phonetic, speaker, and sentiment information and improve performance across a wide range of downstream tasks.
- The learned representations are robust in low-resource settings and can transfer across different tasks and datasets.