Source-linked AI summary
Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
Shaoshi Ling, Yuzong Liu, Julian Salazar, Katrin Kirchhoff
TL;DR
The paper addresses the need to train end-to-end ASR with less labeled audio by learning acoustic representations from unlabeled speech. It reconstructs filterbank slices from past and future context to create DeCoAR features, then trains a CTC-based ASR system. DeCoAR improves benchmark performance and reaches performance similar to 960 hours of labeled LibriSpeech using 100 hours after pretraining.
Problem
End-to-end ASR requires vast amounts of transcribed audio, while abundant unlabeled audio is less costly to obtain.
Method
DeCoAR reconstructs temporal slices of filterbank features from bidirectional context on unlabeled audio, then supplies the representations to a CTC-based end-to-end ASR model.
Results
DeCoAR yields a 42% relative improvement on WSJ over log-mel filterbank features and achieves similar LibriSpeech performance with 100 hours of labeled data as training on 960 hours.
Takeaways & Limitations
Unlabeled-data pretraining can substantially reduce the labeled audio required for the evaluated end-to-end ASR systems.
Takeaways & Limitations
The demonstrated implementation uses LSTMs and CTC, although the authors state that the approach should work with other layers and downstream ASR models.
Abstract
from arXiv · showhide
We propose a novel approach to semi-supervised automatic speech recognition (ASR). We first exploit a large amount of unlabeled audio data via representation learning, where we reconstruct a temporal slice of filterbank features from past and future context frames. The resulting deep contextualized acoustic representations (DeCoAR) are then used to train a CTC-based end-to-end ASR system using a smaller amount of labeled audio data. In our experiments, we show that systems trained on DeCoAR consistently outperform ones trained on conventional filterbank features, giving 42% and 19% relative improvement over the baseline on WSJ eval92 and LibriSpeech test-clean, respectively. Our approach can drastically reduce the amount of labeled data required; unsupervised training on LibriSpeech then supervision with 100 hours of labeled data achieves performance on par with training on all 960 hours directly. Pre-trained models and code will be released online.
1. INTRODUCTION
End-to-end ASR requires substantial labeled audio, although unlabeled audio is cheaper to obtain. The paper proposes learning context-aware acoustic representations from unlabeled data before training ASR with limited labeled data.
- 1. INTRODUCTION: End-to-end ASR models require large amounts of transcribed audio, whereas obtaining abundant unlabeled audio is substantially less costly.
- 1. INTRODUCTION: Self-training uses labeled data to seed hypotheses on unlabeled audio, but it is sensitive to hypothesis quality and confidence calibration.
- 1. INTRODUCTION: The proposed framework learns context-aware acoustic representations from abundant unlabeled audio, then applies them to ASR with limited labeled data.Representation learning reconstructs filterbank features from previous and future context using forward and backward LSTMs; the representations are then used with CTC-based ASR.
- 1. INTRODUCTION: The paper is organized around related work, DeCoAR learning, experimental setup, WSJ and LibriSpeech results, and conclusions.
2. RELATED WORK
Prior work explored semi-supervised ASR and unsupervised speech representations, including APC, CPC, and wav2vec. This paper combines bidirectional contextual processing with frame reconstruction to produce DeCoAR representations for a general semi-supervised framework.
- 2. RELATED WORK: Earlier work applied semi-supervised learning mainly to hybrid ASR, while comparatively few approaches addressed end-to-end ASR.
- 2. RELATED WORK: Prior acoustic representation methods included APC, WaveNet auto-encoders with CPC, and wav2vec for unsupervised speech-processing tasks.
- 2. RELATED WORK: DeCoAR uses bidirectional context to autoregressively reconstruct unseen acoustic frames, producing representations that depend on the entire input sentence.
- 2. RELATED WORK: The framework combines ELMo-style bidirectionality with APC-style reconstruction and requires no architecture change.
3. DEEP CONTEXTUALIZED ACOUSTIC REPRESENTATIONS
DeCoAR combines bidirectional contextual encoding with acoustic-frame reconstruction to learn sentence-level representations from unlabeled speech. These representations are then transferred to CTC-based ASR, with the framework intended to support broader model choices.
- 3.1. Representation learning from unlabeled data: DeCoAR combines ELMo-style bidirectional context with APC-style reconstruction to predict acoustic-feature slices from past and future frames.The input consists of 40-dimensional log filterbank features, and forward and backward LSTMs encode both temporal directions.
- 3.1. Representation learning from unlabeled data: The model applies forward and backward LSTMs to the full input sequence, producing representations that encode previous and future context for each frame.The forward and backward states are concatenated before reconstruction.
- 3.1. Representation learning from unlabeled data: The reconstruction loss sums errors over all possible temporal slices, and the slices can be predicted simultaneously at each position.A position-dependent feed-forward network with 512 hidden dimensions produces the predictions.
- 3.2. Using DeCoAR for ASR: After unsupervised pre-training, DeCoAR parameters are frozen and connected to additional BLSTM layers trained with CTC on labeled speech.The reconstruction layer is removed, and concatenated DeCoAR vectors are used by the ASR-specific network.
- 3.1. Representation learning from unlabeled data: Unlike unidirectional speech representations, DeCoAR uses bidirectional context and functions as a representation of the entire input sentence.The authors describe the framework as applicable to other layer choices and downstream ASR models.
4. EXPERIMENTAL SETUP
The experiments evaluate supervised and semi-supervised CTC-based BLSTM ASR systems on WSJ and LibriSpeech. Semi-supervised systems use DeCoAR or wav2vec features, with varying amounts of labeled data and fixed decoding procedures.
- 4.1. Experimental setup: WSJ uses 30%, 50%, and 100% labeled-data conditions corresponding to 25, 40, and 81 hours, while LibriSpeech ranges from 100 to 960 hours.WSJ uses dev93 for validation and eval92 for evaluation; LibriSpeech uses dev-clean for validation.
- 4.1. Experimental setup: Experiments compare a fully supervised filterbank system with semi-supervised wav2vec and DeCoAR systems, all based on deep BLSTMs with CTC loss.The setup includes three experimental system types.
- 4.2. ASR systems: The supervised baseline uses normalized log-mel filterbank features extracted with a 25ms window and 10ms frame rate, processed by a six-layer BLSTM.Each BLSTM direction has 512 cells.
- 4.2. ASR systems: DeCoAR pre-training uses a four-layer BLSTM with 1024 cells per sub-layer and slice size 18, followed by a projection layer and two-layer BLSTM CTC ASR network.The representation-learning models are trained on eight GPUs with batches grouped by sequence length.
- 4.3. Evaluation: Evaluation reports word error rate using WFST-based decoding with task-specific language models and fixed acoustic-model and blank-prior scales.The decoder uses an unpruned trigram language model for WSJ and a 4-gram language model for LibriSpeech.
5. RESULTS
DeCoAR outperforms filterbank and wav2vec features in semi-supervised WSJ and LibriSpeech experiments, while substantially reducing the labeled data needed. The analyses indicate that bidirectional context supports the gains, though larger reconstruction slices can degrade performance.
- 5.1. Semi-supervised WSJ results: 42% relative improvement over filterbank features and 20% over wav2vec features was achieved on semi-supervised WSJ.Using 30% labeled data produced performance comparable to 100% labeled filterbank training, while 50% produced better eval92 performance.
- 5.2. Semi-supervised LibriSpeech results: 100 hours of labeled LibriSpeech data achieved performance very similar to 960 hours of labeled filterbank training with DeCoAR pre-training.On test-other, 360 hours reached performance on par with the filterbank baseline.
- 5.3.1. Context window size: Larger masked prediction slices can degrade performance while increasing training time.This effect was observed when varying the context window size during pre-training.
- 5.4. Unidirectional versus bidirectional context: Bidirectional context appears to be the largest contributor to DeCoAR’s improved performance, as the unidirectional model performed comparably to wav2vec.The comparison used unidirectional versus bidirectional context during pre-training.
- 5.4.1. DeCoAR as denoiser: DeCoAR reconstructs the current frame almost perfectly, while predictions deeper inside the masked slice become simpler and can develop artifacts.For a slice size of 18, background energy is removed and artifacts appear by i = 8.
- 5.4.1. DeCoAR as denoiser: Reconstructing a missing middle frame suggests that DeCoAR representations capture longer-term phonetic structure during unsupervised pre-training.The representations support downstream ASR with only two additional layers.
6. CONCLUSION
The paper introduces DeCoAR, a semi-supervised ASR approach that learns contextual acoustic representations from unlabeled audio before CTC-based recognition. It reports substantial WSJ improvement and LibriSpeech performance comparable to full-data training with only 100 labeled hours.
- 6. CONCLUSION: DeCoAR learns deep contextualized acoustic representations from unlabeled audio and uses them to train a CTC-based end-to-end ASR model.The approach uses a deep bidirectional LSTM reconstruction objective before supervised ASR training.
- 6. CONCLUSION: 42% relative improvement over log-mel filterbank features was achieved on WSJ.The comparison is against a baseline trained on log-mel filterbank features.
- 6. CONCLUSION: On LibriSpeech, pre-training followed by 100 hours of labeled data achieved performance similar to training on 960 labeled hours.The framework can also be applied to other end-to-end ASR models.