PaperlayerView exact revision on arXiv

Source-linked AI summary

BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition

Yu Zhang, Daniel S. Park, Wei Han, James Qin, Anmol Gulati, Joel Shor, Aren Jansen, Yuanzhong Xu, Yanping Huang, Shibo Wang, Zongwei Zhou, Bo Li, Min Ma, William Chan, Jiahui Yu, Yongqiang Wang, Liangliang Cao, Khe Chai Sim, Bhuvana Ramabhadran, Tara N. Sainath, Françoise Beaufays, Zhifeng Chen, Quoc V. Le, Chung-Cheng Chiu, Ruoming Pang, Yonghui Wu

arXiv:2109.13226v3eess.AScs.CLcs.LGcs.SD

TL;DR

Prior semi-supervised ASR evidence has focused mainly on audiobook-domain settings and smaller supervised tasks. This paper scales Conformer pre-training and self-training to models up to 8B parameters and unlabeled datasets approaching one million hours, then evaluates broad ASR and non-ASR workloads. It matches Voice Search state-of-the-art using 3% of 34k hours, improves the full task, and reports strong benchmark results across domains, while acknowledging that experimental options were not exhaustively explored.

  • Problem

    Prior semi-supervised ASR settings centered on audiobook data, with limited domain generalization and smaller supervised tasks than practical industrial workloads.

  • Method

    The paper pre-trains and self-trains Conformer models up to 8B parameters on large unlabeled audio datasets, then fine-tunes them across ASR and non-ASR tasks.

  • Results

    Across diverse downstream tasks, the models improve or match state-of-the-art performance; on 34k-hour Voice Search, 3% of labeled data matches reported state-of-the-art.

  • Takeaways & Limitations

    Large-scale pre-training, self-training, and model scaling substantially improve labeled-data efficiency while transferring to varied speech domains and non-ASR tasks.

Abstract

from arXiv · show

We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio. We find that the combination of pre-training, self-training and scaling up model size greatly increases data efficiency, even for extremely large tasks with tens of thousands of hours of labeled data. In particular, on an ASR task with 34k hours of labeled data, by fine-tuning an 8 billion parameter pre-trained Conformer model we can match state-of-the-art (SoTA) performance with only 3% of the training data and significantly improve SoTA with the full training set. We also report on the universal benefits gained from using big pre-trained and self-trained models for a large set of downstream tasks that cover a wide range of speech domains and span multiple orders of magnitudes of dataset sizes, including obtaining SoTA performance on many public benchmarks. In addition, we utilize the learned representation of pre-trained networks to achieve SoTA results on non-ASR tasks.

I. INTRODUCTION

The paper examines whether large-scale semi-supervised learning can overcome the limited generalization and scale of prior audiobook-focused ASR settings. It studies large pre-trained and self-trained models across downstream tasks, finding improved labeled-data efficiency and strong results across ASR and non-ASR benchmarks.

  • Motivation: Prior semi-supervised ASR work largely centers on Libri-Light and LibriSpeech, limiting domain generalization, relative unlabeled-data scale, and task size.Libri-Light contains 60k hours, is tailored to the supervised task, and is not much larger than industrial-scale labeled datasets.
  • Approach: The study scales models from 600M to 8B parameters and uses unlabeled datasets ranging from hundreds of thousands to approximately one million hours.It constructs P-models from pre-training and PS-models from pre-training plus self-training.
  • Approach: Downstream experiments cover labeled audio datasets from tens of hours to tens of thousands of hours across diverse domains and languages.The paper considers training P-models, downstream self-training with P-models, and fine-tuning PS-models.
  • Key findings: 3% of the 34k-hour Voice Search training data yields results comparable to reported state-of-the-art performance.The result uses subsets without additional labeled data.
  • Key findings: Pre-trained models match or exceed state-of-the-art across varied ASR tasks and achieve state-of-the-art on multiple public audio-classification benchmarks.The figure summarizes public and non-public ASR results and public audio-classification results using task-specific metrics.
  • Key findings: The gains from larger models, pre-training, and self-training diminish as downstream labeled datasets grow, but remain meaningful for industrial-scale tasks.Figure 2 compares preparation methods and model sizes on Voice Search subsets.

B. Outline

The report uses Conformer encoders with wav2vec 2.0 pre-training and noisy-student-style self-training on large YouTube datasets, then evaluates them across ASR and non-ASR tasks. It reports strong benchmark performance, studies scaling and data efficiency, and notes that the experimental options are not exhaustively explored because of limited budgets.

  • Methods: Conformer encoders with 600M, 1B, and 8B parameters are trained using wav2vec 2.0 pre-training and RNN-T or CTC fine-tuning.The models are scaled to 8B parameters using GShard with the GSPMD backend.
  • Methods: YouTube data provides a 1M-hour unlabeled pre-training set and a 500k-hour filtered unlabeled self-training set for constructing P- and PS-models.P-models are pre-trained, whereas PS-models are pre-trained and self-trained with large unlabeled datasets.
  • ASR Tasks: Fine-tuning P- and PS-models matches Voice Search benchmarks using 3% of the full data and improves the full task through pre-training.The paper also reports state-of-the-art or near-state-of-the-art performance on YouTube and public datasets.
  • Experiments with Voice Search: The Voice Search experiments vary labeled-task size, model size, pre-training, upstream self-training, and downstream self-training.The study uses the 34k-hour dataset and 100h and 1000h subsets, and also explores cross-lingual benefits.
  • Non-ASR Tasks: Pre-trained representations support state-of-the-art performance on multiple non-ASR tasks using linear models, with intermediate layers performing best for some benchmarks.For AudioSet, intermediate Conformer layers pre-trained on the native unlabeled dataset yield state-of-the-art results.
  • Discussions and Future Directions: The experiments do not exhaustively explore all options because the tasks have varied existing setups and limited experimentation budgets.The report describes important experiment-specific elements in the corresponding sections and appendix.

II. METHODS

The methods combine Conformer encoders with wav2vec 2.0 pre-training and noisy student self-training, then adapt them to downstream ASR models. Experiments vary model scale, decoder type, preparation, and pseudo-labeling choices.

  • Model Architecture: Conformer encoders combine attention, feed-forward, and convolutional modules after convolutional subsampling to produce final features.These features feed either an RNN transducer with a 2-layer LSTM decoder or a CTC model.
  • Model Scale: The study evaluates 600M-, 1B-, and 8B-parameter Conformer models, denoted XL, XXL, and G.The models use relative attention and convolutional kernels of size 5.
  • Pre-training: wav2vec 2.0 pre-training masks encoded features and uses contrastive loss to align context vectors with target context vectors.The target context vectors are obtained by applying a linear layer to the initial encoded features.
  • Self-training: Noisy student training uses teacher-generated pseudo-labels to train an augmented student model on unlabeled data.The procedure may filter pseudo-labels by confidence, mix them with labeled data, and fine-tune a new model with augmentation.

D. Gshard/GSPMD: Making 8B-parameter Models Trainable and Efficient

The study builds large-scale YouTube datasets and uses distributed training infrastructure to prepare models for downstream tasks. Its data ranges from weakly labeled and pseudo-labeled audio to approximately a million hours of unlabeled audio.

  • Gshard/GSPMD: GShard with the GSPMD backend trains the 8B model using pipeline parallelism suited to its many-layer architecture.The approach reduces interdevice communication costs and compilation or startup time by dividing the model into pipeline stages.
  • YouTube Data: YT-L combines 350k hours of weakly labeled transcript-audio segments with 1000 hours of labeled audio.The weakly labeled portion selects high-confidence transcript-audio islands using force alignment and a pre-existing acoustic model.
  • YouTube Data: YT-T contains 500k hours of segmented pseudo-labeled audio generated by the YT teacher model.User-uploaded transcripts are discarded, and the teacher trained on YT-L supplies pseudo-labels.
  • YouTube Data: YT-U contains approximately 1M hours of segmented unlabeled audio collected from speech-heavy YouTube videos.A 100M-parameter bidirectional-LSTM teacher identifies speech boundaries during segmentation.
  • Data Diversity: YouTube-based datasets span a wide range of sub-domains, supporting downstream tasks across varied domains and languages.The downstream task collection ranges from tens of hours to tens of thousands of hours of audio.

B. Pre-trained and Self-trained Models

The paper defines prepared Conformer models by size, decoder, and preparation status, then fine-tunes pre-trained and self-trained variants on downstream ASR tasks. Self-training creates upstream models intended for fine-tuning on smaller tasks.

  • Model Notation: Models are named Conformer{Size}-{Decoder}-{Preparation}, combining model size, decoder type, and preparation method.Sizes are XL, XXL, or G; decoders are CTC or RNNT.
  • Preparation: Preparation is Null, P, or PS, representing no preparation, pre-training, or pre-training followed by self-training.Pre-training uses wav2vec 2.0 on YT-U and prepares only the encoder.
  • Preparation: PS-models are pre-trained on YT-U and self-trained on pseudo-labeled YT-T data.Although PS-models do not directly see YT-L, YT-T inherits information through pseudo-labels from a teacher trained on YT-L.
  • Downstream Evaluation: The study fine-tunes pre-trained and self-trained models on downstream ASR tasks and compares them with existing benchmarks.The evaluation covers tasks spanning a wide range of dataset sizes and domains.

A. YouTube

The YouTube experiments compare pre-trained and self-trained Conformers across downstream speech tasks. Large, diverse YouTube pre-training benefits broader domains than Libri-Light, while pseudo-labeling and task-specific conditions affect outcomes.

  • Data and setup: YT-L supplies partially labeled and weakly labeled supervision, while YT-T provides unlabeled audio for self-training.The student is trained on YT-T pseudo-labeled by a YouTube LSTM teacher.
  • Results: Machine-generated transcripts alone outperform the default labeled dataset for the YouTube task.
  • Results: XXL models do not significantly improve beyond XL models on YouTube, possibly because YT-L and YT-T labels are highly noisy.
  • Cross-domain comparison: Pre-training on YT-U benefits a wider variety of SpeechStew domains than Libri-Light, consistent with YT-U being larger and more diverse.YT-U contains 1 million hours versus 60k for Libri-Light.
  • Self-training: SpeechStew noisy-student training filters 50% of pseudo-labeled YT-T data and significantly improves AMI-IHM, Callhome, and TED-LIUM, with small degradation elsewhere.
  • Self-training: On SpeechStew, the PS-model performs worse than the P-model across the board, hypothesized partly to reflect upstream–downstream text-normalization differences.
  • Cross-domain comparison: YT-U pre-trained models lag behind SoTA on LibriSpeech and Switchboard/Fisher, where task-specific language models and domain-matched Libri-Light pre-training are used.

D. CHiME-6

The CHiME-6 experiments evaluate pre-trained and self-trained Conformers on 40 hours of distant-microphone conversational speech. Fine-tuning through SpeechStew and self-training yields substantial relative WER improvements.

  • Data and setup: CHiME-6 contains 40 hours of distant-microphone conversational speech in everyday home environments, enhanced with BeamformIt and guided source separation.
  • Evaluation: The study reports P- and PS-model results against previous SoTA and Libri-Light-pre-trained baselines.
  • Results: The directly fine-tuned ConformerXXL-RNNT-P shows strong CHiME-6 performance, unlike the Libri-Light-pre-trained Conformer XXL, which fails to train directly.
  • Results: 11% relative WER improvement over SoTA is achieved by training on SpeechStew before fine-tuning on CHiME-6.
  • Results: 20% relative WER improvement is achieved by fine-tuning the PS-model on CHiME-6, producing the strongest reported performance.

V. EXPERIMENTS WITH VOICE SEARCH

Voice Search experiments vary labeled-data scale, model size, pre-training, self-training, and language to assess data efficiency. Gains from pre-training and scaling persist at large task sizes but become relatively smaller.

  • Experimental design: The study constructs 100h, 1000h, and 34kh Voice Search tasks from a 34k-hour labeled English (US) dataset.
  • Experimental design: Cross-lingual experiments use Hungarian, Chinese, and Hindi Voice Search tasks with English or native-language YouTube pre-training.
  • Pre-training: Pre-training improves performance across 100h, 1000h, and 34kh tasks, although relative improvement decreases as labeled data grows.
  • Model scaling: An 8B-parameter Conformer G is included in model-size experiments spanning XL, XXL, and G models.
  • Model scaling: For 600M-to-1B models trained from scratch, pre-training is no longer necessary for benefiting from model size on full Voice Search; the 8B model fails to converge.

D. Cross-lingual Benefits

Cross-lingual Voice Search experiments compare no pre-training, English pre-training, and native-language pre-training across labeled-data scales. Benefits are strongest with smaller datasets and vary by language at full scale.

  • Experimental design: Each language uses XL RNN-T baselines, English YouTube pre-training, and native-language YouTube pre-training on 100h, 1000h, and full Voice Search data.
  • Findings: English and native-language pre-training are more effective with smaller labeled datasets.
  • Findings: English pre-training hurts performance on the full dataset, while native pre-training benefits persist for Hungarian at full scale.
  • Self-training: Upstream self-training gives further gains over pre-training alone on 100h and 1000h Voice Search, but not on the full dataset.
  • Self-training: Downstream noisy-student training uses pseudo-labels from a teacher on 20% of YT-U, filtering 50% by confidence-per-word before mixing with labeled data.
  • Limitations: Downstream noisy-student training adds no gains on the full Voice Search task.
  • Non-ASR representations: The Conformer XL Non-RA representation outperforms its relative-attention counterpart on audio classification tasks.

A. Non-Semantic Speech (NOSS) Benchmark

The NOSS benchmark evaluates speech representations across non-semantic tasks, where intermediate Conformer embeddings achieve strong results with simple linear models. Performance generally exceeds prior representations, while the best layer lies inside the network rather than at its end.

  • NOSS covers speech classification tasks including emotion, speaker, and language identification while excluding word-meaning tasks.
  • 4/7 public NOSS tasks achieve new state of the art using only task-specific linear layers.These results outperform prior systems using complex task-specific architectures on Voxforge, CREMA-D, SAVEE, and Masked Speech.
  • 20% relative improvement is achieved on CREMA-D over prior state of the art using visual and acoustic features.DementiaBank remains below state of the art because the strongest comparison uses additional textual information.
  • All four Conformer models outperform most prior non-semantic speech representations on most tasks, except YAMNet on ESC-50.YAMNet’s supervised classes form a superset of ESC-50’s labeled classes.
  • Embeddings from most Conformer layers outperform previous results, indicating useful representations beyond language-focused tasks.
  • The best Conformer layer is in the middle of the network rather than at the penultimate position.Average accuracy is measured across NOSS tasks as a function of model layer, with the best layer marked for each model.

B. AudioSet

AudioSet tests whether unsupervised Conformer representations transfer to non-speech audio event classification. Intermediate layers outperform final outputs, and in-domain pretraining produces the strongest reported spectrogram-only result.

  • AudioSet contains nearly 2 million clips labeled across an ontology of 527 audio-event classes.
  • Intermediate Conformer layers perform significantly better than the final output layer for AudioSet classification.The final layer lags prior approaches, suggesting the pretraining objective is not well matched to audio event classification while still producing useful internal features.
  • 0.308 mAP is obtained by a speech-only YT-U model at layer 10 on AudioSet.The model had not been exposed to AudioSet’s diverse ontology.
  • 0.340 mAP is achieved by an AudioSet-pre-trained ConformerXL, exceeding prior work using spectrograms alone for training and evaluation.
  • Relative attention slightly degrades AudioSet performance, consistent with the NOSS results but contrary to the ASR results.The authors hypothesize that whole-clip prediction reduces the value of relative positional information.
  • Increasing model size, pretraining, and self-training improves performance mainly through labeled-data efficiency, with smaller effects on very large labeled datasets.

APPENDIX A EXPERIMENT DETAILS

The appendix summarizes pretraining and downstream training configurations for large Conformer models, including hardware, learning-rate schedules, batch-size scaling, and noisy-student training.

  • Pretraining uses Google Cloud TPU V3 chips.
  • XL and XXL P-model training scales batch size by 4× for each larger Voice Search dataset.A fixed transformer learning-rate schedule is used across the 100h, 1000h, and 34kh tasks.
  • XL PS-model training scales batch size by 4× and learning rate by 3× for progressively larger tasks.Encoder and decoder learning-rate schedules are set identically for PS-models.
  • Conformer XL and XXL models trained from scratch use batch size 1024 and 33k warm-up steps.Noisy-student training keeps supervised-training parameters and requires 2× to 4× longer training time.
  • SpeechStew training uses 100k supervised steps with batch size 2048 and 200k noisy-student steps with batch size 1024.
  • CHiME-6 1B-parameter PS-model hyperparameters match the prior Libri-Light and SpeechStew training configuration.
Loading 2109.13226v3…