PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Speech Recognition via Large-Scale Weak Supervision

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever

arXiv:2212.04356v1eess.AScs.CLcs.LGcs.SD

TL;DR

Speech recognition systems often require task-specific fine-tuning and may generalize poorly beyond their training distributions. Whisper scales weakly supervised multilingual, multitask training to 680,000 hours and transfers well to existing datasets zero-shot, without fine-tuning, while improving robustness.

  • Problem

    Existing speech systems require complex fine-tuning, while dataset-specific patterns may not generalize reliably across distributions.

  • Method

    Whisper trains a single audio-conditional model on 680,000 hours of diverse multilingual, multitask weakly supervised audio-text data for zero-shot transfer.

  • Results

    Whisper transfers well to existing datasets zero-shot, improves speech-recognition robustness, and benefits from joint multilingual and multitask training at sufficient scale.

  • Takeaways & Limitations

    Scaling weakly supervised pretraining can substantially improve robust speech recognition without self-supervision, self-training, or dataset-specific fine-tuning.

  • Takeaways & Limitations

    Long-form transcription still suffers from repetition loops, omitted segment boundaries, and hallucinated transcripts despite improved decoding strategies.

Abstract

from arXiv · show

We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.

1. Introduction

The paper identifies dataset-specific fine-tuning and limited high-quality supervised data as obstacles to robust, broadly deployable speech recognition. It introduces Whisper, trained on 680,000 hours of weakly supervised audio, with multilingual and multitask training, and demonstrates zero-shot transfer without dataset-specific fine-tuning.

  • Motivation: Unsupervised audio encoders learn strong speech representations but lack an equivalently effective decoder, making task performance dependent on complex fine-tuning.Fine-tuning also risks brittle patterns that fail to generalize across datasets and distributions.
  • Motivation: The goal is speech recognition that works reliably “out of the box” across broad environments without supervised decoder fine-tuning for each deployment.The introduction frames dataset-specific fine-tuning as a weakness limiting usefulness and robustness.
  • Contributions: 680,000 hours of labeled audio data scale weakly supervised speech recognition by the next order of magnitude, producing Whisper models that transfer well zero-shot.This removes the need for dataset-specific fine-tuning to achieve high-quality results.
  • Contributions: 117,000 hours cover 96 other languages, while 125,000 hours contain X→en translation data, enabling multilingual and multitask pre-training.For sufficiently large models, joint multilingual and multitask training has no drawback and can provide benefits.
  • Contributions: The work shows that simple weakly supervised scaling can support robust speech recognition without self-supervision or self-training, and releases models and inference code.The released resources are intended as a foundation for further research on robust speech recognition.

2. Approach

Whisper uses a sequence-to-sequence Transformer to jointly perform multiple speech-processing tasks, with task specifications encoded as decoder tokens. Its web-scale training data is minimally normalized, filtered for quality and language consistency, and segmented into short audio-transcript pairs.

  • Data preparation: Raw transcript text is used without significant standardization, eliminating the need for a separate inverse text normalization step when producing naturalistic transcriptions.The approach relies on sequence-to-sequence models to learn the mapping between utterances and their transcribed form.
  • Data preparation: Internet audio-transcript pairs provide diverse coverage across environments, recording setups, speakers, and languages, while filtering removes low-quality or machine-generated transcripts.Additional inspection targeted high-error, large data sources and identified partially transcribed, poorly aligned, and undetected machine-generated captions.
  • Data preparation: An audio language detector excludes pairs whose spoken and transcript languages mismatch, except that English transcripts are retained as X→en examples.The detector checks spoken language against transcript language according to CLD2.
  • Data preparation: Audio is divided into 30-second segments paired with the transcript text occurring within each segment, including some nonspeech segments for voice activity detection training.Nonspeech segments are included with sub-sampled probability.
  • Multitask modeling: A single encoder-decoder Transformer replaces separate pipeline components by jointly modeling transcription, translation, language identification, voice activity detection, and alignment.Tasks and conditioning information are represented as sequences of input tokens to the decoder, enabling one-to-many mappings from audio.
  • Multitask modeling: Task specification tokens condition the decoder on the requested task, language, and other information, while transcript history supplies longer-range context for resolving ambiguous audio.The decoder is trained as an audio-conditional language model and receives transcript text context with some probability.

3. Experiments

Experiments show that zero-shot Whisper models are substantially more robust across diverse speech distributions than supervised LibriSpeech systems, while also performing strongly on multilingual speech and translation. Results additionally expose limitations in language identification, dataset-specific evaluation, and metric standardization.

  • Evaluation methodology: Standardized text normalization produces WER drops of up to 50 percent on several datasets by reducing penalties for non-semantic transcription differences.The authors release the normalizer code but caution that developing it may overfit to Whisper’s transcription style and affect direct comparisons.
  • Robustness and human comparison: 2.5 WER is the best zero-shot Whisper model’s LibriSpeech test-clean result, yet its robustness on other datasets surpasses all benchmarked LibriSpeech models by large amounts.The smallest model has 39 million parameters and 6.7 WER on LibriSpeech test-clean, while remaining roughly competitive with the best supervised model on other datasets.
  • Robustness and human comparison: Whisper’s best zero-shot models roughly match human accuracy and robustness, motivating zero-shot and out-of-distribution evaluations when comparing systems with humans.The paper argues that standard test-set comparisons can conflate capabilities because humans and machines receive different amounts and types of training supervision.
  • Multilingual speech recognition: Whisper outperforms XLS-R, mSLAM, and Maestro zero-shot on Multilingual LibriSpeech, but significantly underperforms prior work on VoxPopuli and only beats VP-10K+FT.The Multilingual LibriSpeech result uses a simple text standardizer, preventing direct comparison or claims of state-of-the-art performance.
  • Speech translation: 29.1 BLEU is Whisper’s zero-shot state-of-the-art result on CoVoST2 X→en without CoVoST2 training data.The paper attributes this result to 68,000 hours of noisy X→en translation data in pre-training, compared with 861 hours of training data referenced for prior work.
  • Language identification: Whisper’s zero-shot Fleurs language-identification performance is not competitive with prior supervised work and underperforms supervised state of the art by 13.6%.The dataset contains 102 languages, including 20 absent from Whisper’s training data, upperbounding accuracy at 80.4%; performance on the 82 overlapping languages reaches 80.3%.
  • Long-form transcription: Whisper performs better than compared commercial and open-source ASR models on most long-form datasets spanning varied recording lengths and conditions.The comparison includes commercial services and NVIDIA’s STT Conformer-CTC Large model, with commercial systems queried using default English settings as of September 1st, 2022.

4. Analysis and Ablations

Whisper’s zero-shot performance generally improves with model size, dataset size, and scale, though gains diminish for English recognition and at the largest dataset sizes. Multitask and multilingual training can cause negative transfer in small models but yields positive transfer at larger scales, while normalization and long-form decoding heuristics improve robustness.

  • Model size: Performance continues to increase with model size across multilingual speech recognition, speech translation, and language identification, while English recognition shows diminishing returns.The diminishing returns for English recognition may reflect saturation near human-level performance.
  • Dataset size: 680,000 hours of labeled audio makes the Whisper dataset one of the largest ever created in supervised speech recognition.The dataset-size study trains medium-sized models on 0.5%, 1%, 2%, 4%, and 8% subsets and compares them with a model trained on the full dataset.
  • Dataset size: All increases in dataset size improve performance on all tasks, but improvement rates vary substantially across tasks and dataset sizes.English recognition improves rapidly from 3,000 to 13,000 hours, then slows between 13,000 and 54,000 hours; the full dataset adds only a further 1 point drop in WER.
  • Dataset size: Multilingual speech-recognition WER follows a power-law trend through 54,000 hours, then improves only a further 7 points when scaling to the full dataset.X→en translation is practically zero at 7,000 hours or less, then improves roughly log-linearly through 54,000 hours before diminishing returns at full scale.
  • Transfer: Small jointly trained models exhibit negative transfer, but multilingual and multitask models scale better and eventually outperform English-only models.In the largest experiments, joint models slightly outperform English-only models even without adjusting for compute spent per task.
  • Long-form transcription: Adding long-form transcription interventions incrementally reduces overall WER, although improvements are not evenly distributed across datasets.The interventions include beam search with 5 beams, combined no-speech and average log-probability thresholds, and constraining the initial timestamp to 0.0–1.0 second.

5. Related Work

Prior speech-recognition research has emphasized scaling compute, model size, and datasets, while related work has also developed multitask learning and studied robustness under distribution shift. These lines of research provide context for large-scale, multilingual, multitask speech systems.

  • Scaling Speech Recognition: Speech-recognition research has consistently documented benefits from scaling compute, models, and datasets.Early deep-learning work reported improvements from model depth and size and used GPUs to make larger-model training tractable.
  • Scaling Speech Recognition: Recent speech-recognition studies explored billion-parameter models and training datasets containing up to 1,000,000 hours of data.The cited examples are Zhang et al. (2020) for billion-parameter models and Zhang et al. (2021) for up to 1,000,000 hours of training data.
  • Multitask Learning: Multitask learning has a long history spanning speech recognition, multilingual modeling, NLP, and sequence-to-sequence systems.The passage cites early multilingual speech models, foundational NLP multitask learning, and sequence-to-sequence work using multiple encoders and decoders.
  • Robustness: Robustness research examines model transfer and generalization under distribution shift and other perturbations across machine-learning fields.Prior work highlighted poor generalization between datasets despite high performance on IID test sets.

6. Limitations and Future Work

The paper identifies limitations in long-form decoding, lower-resource language coverage, and the exclusive focus on zero-shot transfer. Future work includes improved decoding, more balanced training data, fine-tuning, language-model analysis, and auxiliary objectives.

  • Improved decoding strategies: Larger models reduce perception-related errors, but long-form transcription still suffers from stubborn seq2seq, language-model, and text-audio-alignment failures, including repeat loops.These remaining errors are described as non-human/perceptual rather than simple confusions of similar-sounding words.
  • Increase Training Data For Lower-Resource Languages: Whisper performs poorly on many languages, and language performance is strongly predicted by language-specific training-data volume.The pre-training dataset is English-heavy because the collection pipeline primarily sourced English-centric parts of the internet.
  • Studying fine-tuning: The study evaluates only zero-shot transfer, although fine-tuning could improve results in domains with high-quality supervised speech data.Fine-tuning would also enable direct comparisons with supervised systems.
  • Studying the impact of Language Models on Robustness: Whisper’s robustness may partly come from its audio-conditional language-model decoder, but the contribution of encoder and decoder training remains unclear.Future studies could ablate the decoder, train a decoder-less CTC model, or combine existing encoders such as wav2vec 2.0 with a language model.
  • Adding Auxiliary Training Objectives: Although Whisper achieves good performance without unsupervised pre-training or self-teaching, incorporating auxiliary training objectives could further improve results.This distinguishes Whisper from most recent state-of-the-art speech recognition systems.

7. Conclusion

Whisper shows that scaling weakly supervised pretraining can substantially improve speech recognition robustness. Its results come from large, diverse supervised training and zero-shot transfer, without self-supervision or self-training.

  • Conclusion: Whisper argues that scaling weakly supervised pretraining has been underappreciated in speech recognition research.This conclusion challenges the emphasis placed on other large-scale training techniques.
  • Conclusion: The results are achieved without self-supervision or self-training techniques.These techniques had been mainstays of recent large-scale speech recognition work.
  • Conclusion: Training on a large and diverse supervised dataset with zero-shot transfer significantly improves speech recognition robustness.The approach focuses on simple supervised training rather than fine-tuning-based transfer.

A. Evaluation Datasets. · A.1. Short-form English-only datasets · A.2. Long-form English-only datasets

The evaluation covers short-form and long-form English-only speech datasets, including established ASR corpora, conversational and accented speech, noisy meetings, broadcasts, podcasts, interviews, and earnings recordings. Short-form datasets use corpus-specific splits or preprocessing, while long-form datasets preserve full recordings with aligned transcripts or labels.

  • A.1. Short-form English-only datasets: Short-form evaluation uses LibriSpeech test-clean and test-other splits, TED-LIUM 3’s test split, and the English subset of Common Voice 5.1.TED-LIUM 3 uses segmented manual transcripts, while Common Voice 5.1 was downloaded from its official website.
  • A.1. Short-form English-only datasets: Short-form evaluation additionally includes the Artie bias corpus, a Common Voice subset, plus CallHome and Switchboard from LDC2002S09 and LDC2002T43.These corpora extend evaluation beyond standard read-speech datasets to bias and conversational speech.
  • A.1. Short-form English-only datasets: WSJ uses LDC93S6B and LDC94S13B with s5-recipe preprocessing, while CORAAL uses 231 interviews and FairSpeech preprocessing.The CORAAL interviews come from Kendall and Farrington (2021).
  • A.1. Short-form English-only datasets: CHiME-6 is created from CHiME-5 using stage 0 of the s5 track1 recipe to fix synchronization, then evaluated with binaural recordings and corresponding transcripts.The recordings are identified as * P??.wav.
  • A.2. Long-form English-only datasets: Long-form evaluation uses 11 full-length TED-LIUM 3 test talks, slicing labeled audio spans and assigning concatenated text labels.Each source file is sliced from the first labeled segment’s beginning to the last labeled segment’s end.
  • A.2. Long-form English-only datasets: Meanwhile contains 64 segments from The Late Show with Stephen Colbert, labeled using manually corrected closed captions with released video timestamps.The YouTube video IDs and start and end timestamps are available in the code release.
  • A.2. Long-form English-only datasets: Rev16 uses 16 error-free podcast files, Kincaid46 uses 46 audio files and reference transcripts, and Earnings-21, Earnings-22, and CORAAL use repository or full-interview recordings.The human transcription benchmark uses 25 Kincaid46 examples; Rev16 excludes files with audio-label mismatches.

A.3. Multilingual datasets

The multilingual evaluation datasets comprised test splits, ASR data, speech-translation pairs, and downloaded corpora collected through official or publicly available implementations.

  • Multilingual datasets: Multilingual LibriSpeech contributed the test split for each language in the MLS corpus.The paper used the test splits from every language.
  • Multilingual datasets: Fleurs provided audio files and transcripts, with numerical utterance IDs matched to English transcripts for translation evaluation.Data collection used the HuggingFace datasets implementation.
  • Multilingual datasets: VoxPopuli supplied ASR data in 16 languages, including English, collected with the official repository script.The collection used the get asr data.py script.
  • Multilingual datasets: Common Voice 9 was downloaded from its official website.The source was the Common Voice Corpus 9.
  • Multilingual datasets: CoVOST 2 supplied X into English data collected through the official repository.The passage specifies the X into English direction.

B. Compared Models … D.4. Long-form Transcription

The paper compares Whisper with numerous LibriSpeech-trained ASR models, standardizes English and non-English transcripts for evaluation, and reports raw performance across transcription, translation, and long-form tasks. The standardization is explicitly a best-effort procedure rather than a claim of linguistic correctness.

  • C. Text Standardization: English transcript normalization removes bracketed and parenthetical phrases, expands contractions, and applies punctuation, symbol, numeric, spelling, and whitespace transformations.The procedure aims to penalize genuine word mistranscriptions rather than formatting or punctuation differences.
  • C. Text Standardization: Non-English normalization removes bracketed and parenthetical phrases, replaces markers and punctuation with spaces, lowercases text, and collapses successive whitespace.Different language-specific transformations would be needed, but the paper uses this basic procedure because it lacks linguistic knowledge for all languages.
  • C. Text Standardization: For Chinese, Japanese, Thai, Lao, and Burmese, spaces are inserted between letters, effectively measuring character error rate instead of word error rate.The transformation targets languages that do not use spaces to separate words.
  • C. Text Standardization: The standardization is acknowledged as imperfect and potentially unexpected, without claiming that its output is more correct; code is released for future improvement.Its stated purpose is to distinguish innocuous wording differences from genuine mistranscriptions.
  • D.1. English Transcription: Raw English transcription results are organized as WER (%) tables for greedy decoding and for beam search with temperature fallback.These results are presented in Tables 8 and 9.
  • D.2. Multilingual Transcription: Raw multilingual transcription results report WER (%) on Multilingual LibriSpeech, CommonVoice9, VoxPopuli, and Fleurs, while speech translation reports BLEU on Fleurs and CoVoST2.These results are presented in Tables 10–15.
  • D.4. Long-form Transcription: Long-form English transcription is reported separately using a WER (%) table.This result is presented in Table 16.

E. Training Dataset Statistics · F. Hyperparameters

This section presents training dataset statistics and the hyperparameter configurations used for Whisper, including changes for Whisper Large V2 and model learning rates.

  • E. Training Dataset Statistics: The training dataset statistics are presented in Figure 11.The supplied passage identifies the figure but provides no underlying values.
  • E. Training Dataset Statistics: The dataset portion of the section is organized around a dedicated training-statistics figure.Figure 11 is explicitly labeled as covering training dataset statistics.
  • F. Hyperparameters: Table 17 lists Whisper training hyperparameters.The supplied passage provides the table title but no parameter values.
  • F. Hyperparameters: The hyperparameter portion comprises general settings, Large V2-specific changes, and learning-rate information.These topics correspond to Tables 17, 18, and 19, respectively.
  • F. Hyperparameters: Table 18 lists hyperparameters changed for Whisper Large V2.The supplied passage identifies the affected model variant but does not specify the changes.
  • F. Hyperparameters: Table 19 presents Whisper model learning rates.The supplied passage identifies learning rates as the table’s subject but provides no values.
Loading 2212.04356v1…