PaperlayerView exact revision on arXiv

Source-linked AI summary

TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech

Andy T. Liu, Shang-Wen Li, Hung-yi Lee

arXiv:2007.06028v3eess.AScs.CLcs.LG

TL;DR

Speech applications rely heavily on human annotations, motivating methods that learn from unlabeled data. TERA pre-trains Transformer Encoders by reconstructing acoustic features from stochastic alterations across time, frequency, and magnitude, achieving strong results across phone classification, keyword spotting, speaker recognition, and speech recognition.

  • Problem

    Speech applications rely heavily on large amounts of human annotations, motivating self-supervised learning from unlabeled speech.

  • Method

    TERA pre-trains Transformer Encoders to reconstruct acoustic features from altered inputs using stochastic alterations across time, frequency, and magnitude.

  • Results

    TERA demonstrates strong results across phone classification, keyword spotting, speaker recognition, and speech recognition, including a 4.51% improvement over Mockingjay on keyword spotting.

  • Takeaways & Limitations

    TERA transfers to downstream datasets not used in pre-training and supports adaptation to downstream speech-language-processing tasks with limited labeled data.

Abstract

from arXiv · show

We introduce a self-supervised speech pre-training method called TERA, which stands for Transformer Encoder Representations from Alteration. Recent approaches often learn by using a single auxiliary task like contrastive prediction, autoregressive prediction, or masked reconstruction. Unlike previous methods, we use alteration along three orthogonal axes to pre-train Transformer Encoders on a large amount of unlabeled speech. The model learns through the reconstruction of acoustic frames from their altered counterpart, where we use a stochastic policy to alter along various dimensions: time, frequency, and magnitude. TERA can be used for speech representations extraction or fine-tuning with downstream models. We evaluate TERA on several downstream tasks, including phoneme classification, keyword spotting, speaker recognition, and speech recognition. We present a large-scale comparison of various self-supervised models. TERA achieves strong performance in the comparison by improving upon surface features and outperforming previous models. In our experiments, we study the effect of applying different alteration techniques, pre-training on more data, and pre-training on various features. We analyze different model sizes and find that smaller models are strong representation learners than larger models, while larger models are more effective for downstream fine-tuning than smaller models. Furthermore, we show the proposed method is transferable to downstream datasets not used in pre-training.

I. INTRODUCTION

TERA addresses the reliance of speech applications on annotated data by pre-training Transformer Encoders from altered unlabeled speech. Its auxiliary task combines time, frequency, and magnitude alterations with reconstruction to learn contextualized representations for downstream use.

  • Self-supervised learning formulates an auxiliary task that trains models on unlabeled data while targeting useful representations.
  • TERA representations can support downstream speech and language processing through feature extraction or fine-tuning.Speech representations are intended to capture high-level semantic information from acoustic features.
  • TERA applies stochastic alteration along time, frequency, and magnitude axes, then reconstructs altered acoustic frames.The three alterations can be applied separately or together during Transformer Encoder pre-training.
  • The study evaluates TERA on phoneme classification, keyword spotting, speaker recognition, and automatic speech recognition.It also compares alteration methods separately and in combination.
  • TERA differs from prior work by combining three-axis alteration with reconstruction, evaluating ASR beyond classification, and examining transfer to downstream tasks.

II. RELATED WORK

Speech self-supervised pre-training includes contrastive predictive coding and reconstruction-based approaches. Prior methods use temporal prediction or reconstruction, while TERA broadens alteration across multiple acoustic axes.

  • Speech pre-training methods are divided into contrastive predictive coding and reconstruction branches.
  • CPC predicts near-future acoustic frames while contrasting them with frames from other sequences or distant times.Wav2vec uses CPC loss to pre-train speech representations for speech recognition.
  • VQ-wav2vec combines wav2vec with BERT-like training in a two-stage speech representation pipeline.
  • TERA’s comparison framework includes altered-input reconstruction alongside established contrastive approaches.
  • TERA is compared with contrastive methods including CPC, wav2vec, vq-wav2vec, BERT + vq-wav2vec, wav2vec 2.0, Bidir-CPC, and Modified CPC.

1) Autoregressive reconstruction:

Reconstruction-based speech pre-training includes autoregressive prediction, bidirectional contextual reconstruction, and BERT-style masking. These approaches vary in targets, masking policies, and whether they remain fully self-supervised.

  • APC autoregressively encodes past acoustic context and predicts future frames, while VQ-APC adds vector quantization as a bottleneck.
  • DeCoAR combines bidirectional contextual modeling with APC-style reconstruction to learn contextualized acoustic representations.
  • BERT-style speech methods pre-train encoders by masking input frames, sharing Transformer parameters, or reconstructing shuffled frame orders.
  • Some reconstruction approaches use phoneme labels with CTC loss, unlike methods that are fully self-supervised.
  • TERA is compared with APC, VQ-APC, DeCoAR, Mockingjay, Audio ALBERT, SpecAugment, and NPC.

1) Time Alteration:

TERA’s time alteration reconstructs contiguous corrupted spans from surrounding context, while frequency and magnitude alterations modify other acoustic dimensions. The design uses stochastic policies and speech-informed alteration widths.

  • Time Alteration: Time alteration corrupts contiguous frame segments so the model reconstructs them from neighboring past and future context.
  • Time Alteration: The time policy masks selected frames to zero 80% of the time, replaces them with random segments 10% of the time, and leaves them unchanged 10% of the time.Leaving frames unchanged exposes the model to real inputs during training and reduces train-test inconsistency.
  • Time Alteration: Time alteration uses 7-frame blocks corresponding to 85ms, with 15% of total frames selected for alteration.The width lies within the reported average phoneme-duration range, and overlapping blocks can create longer altered spans.
  • Frequency Alteration: Frequency alteration masks one randomly selected consecutive block of frequency bins across all time steps.The policy can mask no frequencies, allowing the model to receive complete frequency information during training.
  • Frequency Alteration: Frequency alteration is reported to improve speaker representations without compromising phoneme classification or ASR performance.
  • Magnitude Alteration: Magnitude alteration augments input-feature magnitudes by adding sampled Gaussian noise with a specified probability.

B. Pre-training TERA

TERA pre-trains Transformer Encoders by reconstructing original acoustic features from inputs altered along time, frequency, and magnitude axes. A stochastic policy samples these alterations during training, and an L1 reconstruction loss updates the encoder and prediction network.

  • TERA minimizes reconstruction error between original acoustic features and outputs predicted from altered frames.The altered input is processed by Transformer Encoders and a prediction network before the L1 loss is computed.
  • Three alteration types modify time steps, frequency bins, or feature magnitudes, and can be applied separately or in combination.Time alteration corrupts blocks of frames, frequency alteration masks frequency-bin blocks, and magnitude alteration changes feature magnitudes.
  • A stochastic policy samples alteration patterns dynamically for each input sequence during pre-training.The policy samples random patterns based on the alterations selected for the training configuration.
  • The Transformer Encoder uses 768-dimensional hidden states, 12 self-attention heads, 0.1 dropout, and 3072-dimensional intermediate layers.The prediction network is a two-layer feed-forward network with hidden size 768.
  • TERA primarily uses 80-dimensional log Mel spectrograms, while also exploring MFCC, FBANK, and fMLLR inputs.Features use 25 ms windows, 10 ms stride, and per-utterance CMVN; pre-training runs use 200k or 1M total steps for 100 or 960 hours of data.

C. Incorporating with Downstream Tasks

TERA representations are incorporated into downstream models either as frozen extracted features or through joint fine-tuning. Layer-weighting experiments favor the deepest layer for representation extraction.

  • TERA supports downstream use through feature extraction with frozen encoder parameters or fine-tuning with downstream models.In fine-tuning, the pre-trained TERA and randomly initialized downstream model are updated together.
  • A learnable weighted sum of all TERA layers is used to identify the best layer for representation extraction.The weighted sum most consistently favors the deepest layer for TERA, APC, and vq-wav2vec in phoneme and speaker classification.

IV. EXPERIMENTAL SETUP

The experiments evaluate TERA across four downstream speech tasks using established datasets and classifier settings. The setup emphasizes comparison with prior work through standardized splits, labels, and evaluation measures.

  • TERA is evaluated on phoneme classification, speaker recognition, keyword spotting, and speech recognition.The experiments use publicly known downstream settings and fixed random seeds.
  • Pre-training uses 100-, 360-, and 500-hour LibriSpeech subsets totaling 960 hours, while LibriSpeech also supplies several downstream tasks.The train-clean-100 subset is used for phoneme classification and speaker recognition.
  • Phoneme Classification: LibriSpeech phoneme classification uses 41 phoneme classes and linear, linear-concat, and one-hidden-layer classifiers.The linear-concat setting concatenates eight windows, while the one-hidden setting probes information not linearly accessible.
  • Phoneme Classification: TIMIT phoneme accuracy maps 48 manual phonemes to 39 classes and uses the same three classifier settings as LibriSpeech.Accuracy is reported on the test set for both LibriSpeech and TIMIT.
  • Keyword Spotting: Keyword spotting is a balanced 12-class task evaluated with a two-hidden-layer feed-forward classifier and temporal mean pooling.The setup tests domain shift from LibriSpeech pre-training to Speech Commands keyword detection.

D. Speaker Classification Setup

Speaker classification measures both frame-wise and utterance-wise accuracy, while ASR uses hybrid DNN/HMM models on TERA representations or fine-tuned TERA. Downstream training also applies SpecAugment regularization.

  • Speaker Classification: Speaker recognition uses LibriSpeech train-clean-100, containing 251 speakers, with frame-wise and utterance-wise linear classification.Both tasks report test-set speaker classification accuracy, and frame classification is described as more difficult.
  • Speaker Classification: Utterance-wise speaker classification is more common, while strong frame-wise performance implies strong utterance-wise performance.The paper reports both tasks for completeness because the implication does not hold in reverse.
  • ASR: ASR uses hybrid DNN/HMM models with either an MLP or a five-layer liGRU followed by two fully connected layers.First-pass decoding uses a 4-gram language model and beam search, with WER computed using NIST SCTK.
  • ASR: For ASR, TERA is either frozen as a representation extractor or updated jointly with the downstream DNN during fine-tuning.ASR pre-training uses 40-dimensional fMLLR features because the PyTorch-Kaldi toolkit works best with fMLLR inputs.
  • Training Regularization: SpecAugment is additive to TERA pre-training, delaying overfitting and improving final downstream accuracy numbers.The policy zeroes selected time and frequency blocks and omits time warping.

A. The Effect of Different Alterations

TERA combines time, frequency, and magnitude alterations during self-supervised pre-training, with each alteration contributing differently across downstream tasks. Time alteration is indispensable for strong phonetic and detection performance, while frequency and magnitude alterations further benefit selected tasks.

  • A. The Effect of Different Alterations: TERA evaluates all seven combinations of time, frequency, and magnitude alterations after pre-training on 100 hours of LibriSpeech.The combinations are time, freq, mag, time+freq, time+mag, freq+mag, and time+freq+mag.
  • 1) The Effect of Time Alteration: Time alteration improves phoneme classification and keyword spotting, with the time-only model achieving the best keyword-spotting performance.Models with time alteration also outperform models without it, while adding other alterations can compromise keyword spotting.
  • 1) The Effect of Time Alteration: Time alteration is considered indispensable because it establishes bidirectional audio context and supports better downstream results.The authors speculate that without time alteration, the Transformer fails to encode proper context and yields sub-optimal performance.
  • 2) The Effect of Frequency Alteration: Frequency alteration boosts phoneme classification and provides strong speaker classification, but it does not help keyword spotting.The frequency-only model is sufficient for strong speaker recognition, while time+freq improves phoneme classification over time alone.
  • 3) The Effect of Magnitude Alteration: Magnitude alteration further improves phoneme and speaker recognition, while its keyword-spotting gains do not exceed the time-only model.The time+freq+mag model improves over time+freq for phoneme classification on seen and unseen data.

1) Phoneme Classification Results:

TERA performs strongly in phoneme classification, including under domain change from LibriSpeech to TIMIT, while model size affects feature extraction and fine-tuning differently. Its broader benchmark results also show strong keyword-spotting and speaker-recognition performance.

  • 1) Phoneme Classification Results:: TERA outperforms other methods on LibriSpeech phoneme classification and all methods except CPC with the linear concat classifier on TIMIT.TERA is not affected by the domain change, and magnitude alteration improves TIMIT phoneme classification accuracy.
  • 1) Phoneme Classification Results:: TERA-large underperforms TERA-base for feature extraction, whereas TERA-large outperforms TERA-base for ASR fine-tuning.The comparison suggests model size has different effects for representation extraction and downstream fine-tuning.
  • 2) Keyword Spotting Results:: TERA improves keyword spotting by 4.51% over Mockingjay after adding frequency and magnitude alteration.The best keyword-spotting representation is vq-wav2vec, while CPC achieves comparable performance.
  • 3) Speaker Recognition Results:: TERA achieves the highest speaker-classification accuracy in both frame-wise and utterance-wise settings.The comparison discusses how different pre-training objectives affect speaker information preserved in each frame.

1) Fine-tuning TERA:

Fine-tuning TERA improves downstream performance across tasks, while more unlabeled data benefits TERA even when noisy data harms time-only masked reconstruction methods.

  • Fine-tuning TERA: Fine-tuning TERA produces considerable performance increases across all evaluated downstream tasks.The pre-trained time+freq+mag TERA-base model was initialized from 960 hours of LibriSpeech.
  • Fine-tuning TERA: Limited-label TIMIT fine-tuning still improves performance without observed overfitting.
  • Fine-tuning TERA: Without pre-training, the same TERA architecture with SpecAugment either underperforms or overfits, showing that architecture alone provides no benefit.
  • Pre-training on more data: TERA-base improves from 100 to 960 hours of pre-training across downstream tasks, whereas NPC and Mockingjay worsen when noisy train-other-500 data is added.
  • Pre-training on more data: TERA benefits from noisy additional data while time-only masked reconstruction methods generally do not.NPC has one exception: keyword spotting improves with more data.
  • Acoustic features: For phoneme classification, log Mel outperforms FBANK and MFCC, while FBANK outperforms MFCC.The passage also reports an opposite feature ranking for keyword spotting.

4) Effect of Different Network Depth:

TERA’s depth and alteration policy affect representation learning: smaller models perform better for extraction, and TERA’s masking policy outperforms SpecAugment under matched pre-training conditions.

  • Effect of Different Network Depth: Performance decays as TERA depth increases for representation extraction, making the 3-layer base model sufficient for the proposed pre-training task.
  • Effect of Different Network Depth: During ASR fine-tuning, larger TERA models outperform smaller models, reversing the extraction pattern.
  • Masking policy: TERA’s time alteration uses shorter masks than SpecAugment, whose variable masks make reconstruction pre-training loss volatile and unstable.SpecAugment masks up to 100 time steps, compared with TERA’s length of 7.
  • Masking policy: TERA determines the number of time masks from a 15% maximum alteration percentage, so mask count varies with input length.SpecAugment instead uses two consecutive time-mask blocks.
  • Masking policy: TERA’s selected time blocks may be zeroed, replaced with random blocks, or left unchanged, reducing reconstruction and train-test inconsistencies.
  • Comparing TERA with SpecAugment: Under matched 960-hour pre-training and 3-layer architectures, TERA largely outperforms SpecAugment on phoneme classification, keyword spotting, and speaker recognition.The comparison supports TERA for self-supervised representation learning, while SpecAugment is described as suitable for ASR regularization.
  • Speech representations for ASR: Frozen-representation ASR reports WER on LibriSpeech test-clean, with TERA model size having little influence on performance.TERA representations provide constant improvement over MFCC, FBANK, and fMLLR in the same ASR framework.

E. Speech Pre-training for ASR Comparison

For ASR, TERA supports both frozen representation extraction and fine-tuning, with fine-tuning favoring deeper models and improving over extraction; transfer to TIMIT remains supported.

  • Speech Pre-training for ASR Comparison: Fine-tuning TERA with liGRU generally improves with depth, and TERA-large achieves the best WER among the TERA liGRU models.
  • Speech Pre-training for ASR Comparison: Large models are effective for ASR fine-tuning despite performing poorly for feature extraction.
  • Speech Pre-training for ASR Comparison: The proposed ASR WER is reported as within the expected range, but comparisons are constrained by differing language-model setups.
  • Speech Pre-training for ASR Comparison: MLP fine-tuning shows a similar depth trend: deeper TERA models help, with large achieving the best MLP WER.MLP outperforms liGRU on the medium model and is similar for the remaining model sizes.
  • Cross-dataset transfer: When transferred from LibriSpeech pre-training to TIMIT ASR, larger extracted models provide no significant gain, consistent with smaller models being better feature extractors.
  • Conclusion: The conclusion reports strong results across phone classification, keyword spotting, speaker recognition, and speech recognition, plus transfer to downstream datasets.
Loading 2007.06028v3…