Source-linked AI summary
Generative Pre-Training for Speech with Autoregressive Predictive Coding
Yu-An Chung, James Glass
TL;DR
Learning transferable speech representations from unlabeled data is difficult because downstream tasks may require different speech information. The paper pre-trains APC on unlabeled speech and evaluates it for recognition, translation, and speaker identification, finding consistent advantages over competing features and objectives while reducing labeled-data and parameter requirements.
Problem
Learning meaningful, general speech representations from unannotated speech that transfer across tasks with different information requirements remains challenging.
Method
The paper uses autoregressive predictive coding as a generative pre-training objective and evaluates representations transferred to speech recognition, speech translation, and speaker identification.
Results
APC representations consistently outperform log Mel spectrograms and other evaluated objectives across the three speech tasks, while APC-based Transformers outperform the RNN alternative.
Takeaways & Limitations
APC is effective for transfer learning across diverse speech tasks and for reducing downstream labeled-data size and model parameters.
Takeaways & Limitations
The experiments found frozen APC weights performed better than updating them, leaving more sophisticated fine-tuning as future work.
Abstract
from arXiv · showhide
Learning meaningful and general representations from unannotated speech that are applicable to a wide range of tasks remains challenging. In this paper we propose to use autoregressive predictive coding (APC), a recently proposed self-supervised objective, as a generative pre-training approach for learning meaningful, non-specific, and transferable speech representations. We pre-train APC on large-scale unlabeled data and conduct transfer learning experiments on three speech applications that require different information about speech characteristics to perform well: speech recognition, speech translation, and speaker identification. Extensive experiments show that APC not only outperforms surface features (e.g., log Mel spectrograms) and other popular representation learning methods on all three tasks, but is also effective at reducing downstream labeled data size and model parameters. We also investigate the use of Transformers for modeling APC and find it superior to RNNs.
1. INTRODUCTION
The paper seeks general, meaningful speech representations that transfer across tasks requiring different speech information, using APC as a generative pre-training approach. APC is motivated by preserving signal information that downstream models can select rather than discarding potentially useful variability.
- Motivation: Unsupervised speech representations can exploit scalable unlabeled data and are intended to support diverse downstream properties and tasks.The motivating properties include phonetic content, speaker characteristics, and emotional cues.
- Motivation: The proposed goal is a generative pre-training approach producing general, meaningful representations transferable to unknown tasks with different information requirements.Phonetic content may matter more for recognition, whereas speaker information may matter more for speaker-related applications.
- Motivation: Existing objectives may discard useful variability, whereas APC is presented as preserving original-signal information for downstream access.The paper defines accessibility in terms of how linearly separable the representations are.
- Evaluation scope: The paper evaluates APC transfer learning on speech recognition, speech translation, and speaker identification.These experiments are described as the paper’s empirical evaluation of APC across different speech applications.
2. AUTOREGRESSIVE PREDICTIVE CODING
APC is a self-supervised objective that predicts future speech-frame information from sequential acoustic features. The paper implements its encoder with either an autoregressive RNN or a Transformer, producing representations used to predict the target sequence.
- Objective: APC encodes a sequence of acoustic feature vectors and learns to predict information about a future frame.The objective is inspired by neural language modeling and aims to encode what a reasonable spectrogram should look like.
- Objective: Because APC derives its training target from the input, it is self-supervised and can use large quantities of unlabeled speech.This makes APC suitable for pre-training without manually annotated targets.
- Encoder architectures: The RNN encoder uses standard L-layer unidirectional GRUs to process the input sequence autoregressively.Its trainable parameters include the output projection W and the GRU layers.
- Encoder architectures: The Transformer encoder uses stacked decoder blocks with multi-headed self-attention, position-wise feedforward layers, and sinusoidal positional encodings.The sinusoidal encodings provide positional information without introducing additional parameters.
- Encoder architectures: The Transformer output is projected back to the dimensionality of the input acoustic features to produce the prediction sequence.The paper denotes the Transformer-specific mappings as W_in and W_out and ties them in practice as a regularization.
3. TRANSFER LEARNING WITH APC
Transfer learning with APC trains an encoder on unlabeled speech, transforms downstream acoustic features into higher-level representations, and then trains downstream models with the encoder frozen or adapted. The approach supports sequence or single-value labels and compares both adaptation strategies.
- Pre-training: APC is pre-trained on 360 hours of LibriSpeech speech using normalized 80-dimensional log Mel spectrograms and varying prediction horizons n.The training subset contains audio from 921 speakers.
- Representation transfer: For a labeled downstream dataset, the trained encoder transforms each surface feature sequence into a higher-level representation while retaining its associated label.The labels may be sequences or single values, depending on the task.
- Representation transfer: The extracted representation is the output of the encoder’s last RNN or Transformer layer.The paper notes that combining internal representations across layers could be a better alternative.
- Transfer strategies: Downstream training can either keep the APC encoder frozen or update it so the representations adapt to the target task.The experiments examine both transfer-learning approaches.
4. EXPERIMENTS
The experiments evaluate APC transfer learning across speech recognition, speech translation, and speaker identification, including comparisons with other representations and tests of data and model efficiency. APC generally outperforms competing features across these tasks while reducing labeled-data and downstream-model requirements.
- Experimental setup: The study evaluates transfer learning on automatic speech recognition, speaker identification, and automatic speech translation.
- Experimental setup: APC uses both RNN and Transformer backbones, denoted R-APC and T-APC, alongside comparisons with CPC and PASE.
- Speech recognition: In ASR, frozen APC transfer learning reduces WER by more than 25%, from 18.3 to 13.7, while scratch training performs worse than the baseline.
- Speech recognition: T-APC reaches 16.4 versus 18.3 for log Mel when using half the labeled ASR training data, and T-APC with two encoder layers matches log Mel with four layers at 18.6 versus 18.3.
- Speech translation: In speech translation, APC outperforms log Mel, CPC, and PASE on both development and test sets, with T-APC scoring 14.3 versus 13.8 for S-Transformer and 14.6 for the cascaded system.
- Speaker identification: In speaker identification, APC almost always outperforms other features across training-shot conditions, with T-APC nearly twice as good as log Mel in one-shot learning at 17.6 versus 8.7.
5. CONCLUSIONS
The paper concludes that APC is an effective generative pre-training objective for transferring speech representations across tasks. It also identifies frozen APC weights, improved fine-tuning, positional encoding, more unlabeled data, and additional applications as future directions.
- APC representations consistently and mostly significantly outperform log Mel spectrograms and representations learned by CPC and PASE on ASR, speech translation, and speaker identification.
- APC representations are the most effective among the compared methods at reducing downstream labeled data size and model parameters on ASR and speaker identification.
- The Transformer-based APC model is more effective than the RNN-based model used in prior work.
- Frozen APC weights perform better than updating them during downstream training, although the paper considers adaptation toward the target task more ideal.
- Future work includes more sophisticated fine-tuning, improved positional-information injection, training on more unlabeled data, and applying APC to speech synthesis.