Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling

1 INTRODUCTION

• Whisper is a 1.5-billion-parameter model trained on 680,000 hours of weakly supervised speech data, but large pre-trained ASR models are challenging to deploy in low-latency or resource-constrained settings.The introduction notes that ASR systems have surpassed human-level accuracy on many academic benchmarks.Source: §1 · p. 1
• Prior work applied knowledge distillation to BERT and audio classification, and pseudo-labelling to Seq2Seq summarisation, but comparable results had not been achieved for speech recognition.The cited examples include KD for BERT (Devlin et al., 2019; Sanh et al., 2019; Jiao et al., 2020; Sun et al., 2020), pseudo-labelling for summarisation (Kim & Rush, 2016; Shleifer & Rush, 2020), and KD for audio classification (Peng et al., 2021; Chang et al., 2021).Source: §1 · p. 1
• The authors distill Whisper for Seq2Seq ASR using pseudo-labels from an open-source dataset spanning 10 domains, with a WER-based filtering heuristic to select high-quality labels.Pseudo-labelling also standardizes transcription formatting and supplies sequence-level distillation signal; the filtering heuristic is reported to support downstream performance.Source: §1 · p. 1
• The best Distil-Whisper model performs within 1% WER of the original Whisper checkpoint, is 5.8 times faster with 51% fewer parameters, and beats Whisper by 0.1% WER on long-form audio.The long-form gain is attributed to a lower propensity to hallucinate.Source: §1 · p. 2
• Sharing Whisper’s encoder weights lets Distil-Whisper serve as an assistant for speculative decoding, improving inference speed 2 times with an 8% increase to parameter count while preserving Whisper’s outputs.The method is presented as a drop-in replacement for existing Whisper-based speech recognition pipelines.Source: §1 · p. 2
• The authors release Distil-Whisper’s training code, inference code, and models to support further research on speech recognition distillation.They describe large-scale speech-data pseudo-labelling as an under-explored, promising technique for knowledge distillation.Source: §1 · p. 2

2 RELATED WORK

• NLP distillation has achieved compression and speed gains with retained or improved benchmark performance, although distilled models can fare worse than pretrained counterparts on out-of-distribution data.DistilBERT preserved 97% of GLUE language-understanding capabilities with 40% less model size and 60% greater speed (Sanh et al., 2019); DistilBART achieved 37% compression and 48% greater speed while outperforming its original model on XSUM and CNN/Daily Mail (Shleifer & Rush, 2020; Narayan et al., 2018; See et al., 2017). Du et al. (2023) found that distilled models can perform significantly worse than pretrained counterparts on out-of-distribution tests.Source: §2 · p. 2
• Prior ASR distillation compressed encoder-only models or an early Seq2Seq model, but worsened WER; this work targets Seq2Seq compression while preserving WER on out-of-distribution data.Peng et al. (2021) reported 79% compression and 59% greater speed alongside a 6.9% WER increase; Chang et al. (2021) reported a 7.0% WER increase, and Pang et al. (2018) found 2.2% worse WER for their best distilled model.Source: §2 · p. 2
• Earlier Whisper distillation reduced parameters with marginal performance decrement, but used small in-distribution data and did not optimize latency or assess out-of-distribution robustness (Shao et al., 2023).This work seeks joint compression and latency improvements while evaluating out-of-distribution WER and robustness to noisy audio.Source: §2 · p. 2

• Whisper’s 680,000 hours of web-scraped noisy speech pre-training yields competitive results with fully supervised systems in zero-shot recognition without fine-tuning (Radford et al., 2022).Source: §3 · p. 3
• Whisper encodes speech features into hidden states, typically halving sequence length through encoder-stem convolution, then autoregressively predicts each token from prior tokens and those states.Source: §3 · p. 3
• Whisper is trained on audio-text pairs with standard cross-entropy loss to predict the target class labels.Source: §3 · p. 3
• The five Whisper variants share a Seq2Seq architecture but differ in dimensionality; this paper reports results for large-v2, which followed regularised, longer training of large-v1 (Radford et al., 2022).For every size, encoder and decoder widths, attention heads, and transformer-block layer counts match between them.Source: §3 · p. 3

4.1 KNOWLEDGE DISTILLATION

• KD matches the teacher’s full next-token distribution with token-level KL while transferring sequence-level knowledge through teacher-generated pseudo-labels (Hinton et al., 2015; Kim & Rush, 2016).KL provides information over all classes and has lower gradient variance than CE.Source: §4.1 · pp. 3–4
• Students are initialized by copying maximally spaced teacher layers and trained with CE; the paper uses this strategy for all distillation methods (Shleifer & Rush, 2020; Li et al., 2022).Source: §4.1 · p. 4
• The final training objective combines KL and pseudo-label terms with weights αKL = 0.8 and αPL = 1.0 (Shleifer & Rush, 2020).Source: §4.1 · p. 4

4.2 PSEUDO-LABEL SELECTION: WER THRESHOLD

• They normalize references and Whisper pseudo-labels, discard samples whose WER exceeds a validation-tuned threshold λ, and report improved transcription quality and downstream performance.Source: §4.2 · p. 4

5 CHUNKED LONG-FORM TRANSCRIPTION

• Whisper’s fixed 30-second receptive field prevents it from processing longer inputs at once, while real-world applications such as meetings require transcribing audio lasting minutes or hours.Source: §5 · p. 5
• Chunked long-form transcription yields up to 9 times improvements in inference speed over sequential transcription and requires only greedy decoding for reliable transcription.The algorithm was used to evaluate both Whisper and Distil-Whisper.Source: §5 · p. 5
• Adjacent audio chunks overlap, and their inferred text is joined at the strides by finding the longest common sequence between overlaps.Chunks can be transcribed in any order and batched in parallel, provided adjacent chunks are joined correctly at their boundaries.Source: §5 · p. 5

6 SPECULATIVE DECODING

• Speculative decoding speeds inference by verifying assistant-generated candidate tokens in one main-model pass, retaining only matching tokens and ensuring the exact output of the main model (Leviathan et al., 2023).The i-th candidate is kept only if all preceding candidates match the main model’s validation tokens.Source: §6 · p. 5
• Following DistilSpec (Zhou et al., 2023), this work uses a knowledge-distilled student, Distil-Whisper, as the assistant model for Whisper.DistilSpec proposes a distilled assistant to better align its distribution with the main model.Source: §6 · p. 5

7.1 DATA

• The training corpus combines nine public speech-recognition datasets, totaling 21,170 hours, over 18,260 speakers, and 10 domains.Datasets were pre-processed and streamed through Hugging Face Datasets; pseudo-labels came from Whisper large-v2 using greedy decoding, chosen for faster inference after little downstream difference from beam search was observed.Source: §7.1 · p. 5

7.2 TRAINING

• The student copies and freezes the teacher’s encoder and uses a two-layer decoder initialized from its first and last decoder layers.Distil-medium.en and distil-large-v2 are distilled from the corresponding medium.en and large-v2 teachers; training minimizes a weighted sum of KL divergence and pseudo-label loss.Source: §7.2 · p. 6, Figure 1 · p. 6
• Training uses batch size 256 for 80,000 optimization steps across eight epochs, without augmentation or regularization.The setup relies on dataset diversity to support generalization and robustness.Source: §7.2 · p. 6

7.3 SHORT-FORM EVALUATION

• Short-form evaluation covers 15 datasets: 11 in-distribution test splits and four zero-shot out-of-distribution test sets.Effective robustness is measured relative to GigaSpeech, an in-distribution reference shared with the pre-trained and distilled Whisper training data.Source: §7.3 · p. 7
• Noise robustness is assessed by measuring WER on LibriSpeech test-clean as white or pub noise is added at specified signal-to-noise ratios.The comparison includes Distil-Whisper, Whisper, and eight other LibriSpeech-trained models.Source: §7.3 · p. 7

7.4 LONG-FORM EVALUATION

• Long-form evaluation uses four out-of-distribution datasets and measures repeated 5-gram duplicates, insertion, substitution, and deletion error rates.These metrics quantify repetition and hallucination as well as substitution and deletion behavior in long-form transcriptions.Source: §7.4 · p. 8

8.1 SHORT-FORM EVALUATION

• 10.1% average WER, distil-large-v2 is one percentage point above large-v2 while 5.8 times faster and has fewer than half its parameters across four OOD short-form sets.Distil-medium.en averages 2.0% higher WER than large-v2, with 6.8x faster inference and 75% model compression.Source: §8.1 · p. 8
• 0.8% absolute overall WER increase (10.7% relative), distil-large-v2 remains within 2% relative WER on the reference distribution and has comparable effective robustness to large-v2.It improves over the pre-trained baseline by 12.8% relative on SPGISpeech but performs worse on the other three OOD datasets.Source: §8.1 · p. 8, Table 6 · p. 9

8.2 LONG-FORM EVALUATION

• 0.1% lower average WER, distil-large-v2 outperforms or equals large-v2 on four of five OOD long-form test sets while delivering 5.8 times faster batched inference.On Meanwhile, which has a single speaker and a high frequency of uncommon words, Distil-Whisper performs worse. With chunked decoding, distil-large-v2 is 57.5 times faster than sequential large-v2 while within 1.2% WER.Source: §8.2 · p. 9, Table 7 · p. 9

8.3 ROBUSTNESS TO ADDITIVE NOISE

• As additive noise intensifies, Distil-Whisper WER degrades less severely than LibriSpeech-trained models and approaches Whisper, particularly below 10 dB pub-noise SNR.The student and teacher share the same copied, frozen encoder, which the paper identifies as the reason for their similar noise robustness, especially under natural shifts such as pub noise.Source: §8.3 · p. 9, Figure 2 · p. 10

8.4 ROBUSTNESS TO HALLUCINATIONS

• 1.3 times fewer repetition errors and 1.2% lower absolute IER than Whisper give distil-large-v2 the lowest WER of the reported models, despite 1.4% higher SER and DER within 0.3%.Wav2Vec 2.0 has fewer repetition errors than both Whisper models, although its average WER is worse.Source: §8.4 · p. 10, Table 8 · p. 11

8.5 SPECULATIVE DECODING

• 2.0 times faster, speculative decoding with distil-large-v2 provides that speed-up over large-v2 alone while mathematically ensuring the same outputs.For medium.en, distil-medium.en as assistant provides a 2.4 times speed-up, versus 2.0 times with tiny.en; sharing the encoder means distil-large-v2 adds only 8% to parameter count when assisting large-v2.Source: §8.5 · pp. 10–11, Table 9 · p. 11

9.1 WER THRESHOLD

• A 10% WER threshold provides a good trade-off between admitting mistranscribed or hallucinated labels and discarding more data, retaining only the easiest samples.The authors note that longer training might make higher thresholds more favorable; filtering may compensate for the lower accuracy of greedy-decoded labels versus beam-search labels.Source: §9.1 · p. 11, Table 10 · p. 12

9.2 DATASET SCALING

• Above 13,933 hours, or 2% of Whisper pre-training data, additional pseudo-labelled data yields no further improvement to OOD WER.OOD performance improves rapidly from 435 to 3,483 hours and then slows through 13,933 hours; ID validation performance improves with every dataset-size increase.Source: Table 11 · p. 12, §9.2 · p. 12

9.3 MODEL SIZE

• 5.8 times faster than the teacher, the 2-layer decoder model has a maximum 2.1% increase in average OOD WER.With a 16-layer decoder, the model stays within 0.1% OOD WER of the 32-layer teacher and improves latency 1.9 times.Source: §9.3 · p. 12, Table 12 · p. 13
• Reducing the encoder from 32 to 16 layers in a model with a 2-layer decoder cuts parameters by an additional 19% but degrades OOD WER by 3.1% absolute.The result suggests that a deep encoder is important for maintaining strong WER in distilled Seq2Seq ASR models.Source: §9.3 · p. 12, Table 12 · p. 13

• 49% smaller and 5.8 times faster, Distil-Whisper performs within 1% WER of Whisper on OOD short-form audio.Source: §10 · p. 13
• On OOD long-form audio, Distil-Whisper outperforms Whisper because it produces fewer hallucinations and repetitions.Source: §10 · p. 13
• Large-scale pseudo-labelling is effective for distilling ASR models, particularly when combined with a WER threshold filter.Source: §10 · p. 13
• With speculative decoding, Distil-Whisper and Whisper produce the same outputs with inference 2 times faster.Source: §10 · p. 13

11 ACKNOWLEGEMENTS

• The authors thank Nicolas Patry and Arthur Zucker for the chunked long-form transcription implementation, Jo˜ao Gante for speculative decoding, and Google’s TPU Research Cloud for Cloud TPU resources.Source: §11 · p. 13

A.1 TRAINING DATA

• Training combines varied English speech corpora, including People’s Speech (Galvez et al., 2021; approximately 12,000 hours), GigaSpeech (Chen et al., 2021; 2,500 hours), and Common Voice (Ardila et al., 2020; approximately 2,400 hours).The datasets span interviews, podcasts, narrated speech, crowdsourced recordings, telephone calls, talks, meetings, and parliamentary speech; the appendix also describes short- and long-form evaluation sets.Source: §A.1 · pp. 20–21

B.1 TRAINING

• Models train for 80,000 steps (eight epochs) with a slanted triangular learning-rate schedule, and checkpoints are selected by macro-average performance across 13 validation sets.When student and teacher encoders match, the student encoder is frozen and run once; evaluation uses normalized-text WER, while latency is measured on a single A100 across batch sizes 1, 4, and 16.Source: §B.1 · p. 23, §B.2 · p. 23

C EVALUATION RESULTS

• Evaluation tables cover WER across 15 short-form and five long-form test sets, long-form transcription algorithms, and error types; the supplied passages give no table cell results.Short-form latency uses batch size 1 without timestamp prediction; long-form latency uses chunked transcription at batch sizes 4 and 16.Source: §B.2 · p. 23, Table 16 · p. 24, Table 17 · p. 24

D.1 EARLY EXIT

• Table 20 reports WER on 100 LibriSpeech test-clean examples as the early-exit threshold varies, with latency measured relative to the medium.en model using all 24 decoder layers.Source: Table 20 · p. 27

D.2 DISTILLATION OBJECTIVE

• Adding KL divergence lowers OOD WER by 0.3% absolute versus pseudo-label training alone, while adding MSE has negligible WER effect.MSE adds hidden-state memory overhead and reduces maximum batch size, so the final objective uses pseudo-label and KL terms only.Source: §D.2 · pp. 27–28, Table 21 · p. 28

D.3 BATCHED SPECULATIVE DECODING

• 2.0× and 2.4× speed-ups result from batch-1 speculative decoding with distil-large-v2 assisting large-v2 and distil-medium.en assisting medium.en, respectively.The distil-medium.en assistant outperforms tiny.en’s 2.0× speed-up; at batch size 16, speculative decoding is slower than using the main model alone. Accepted outputs match the original model.Source: §D.3 · p. 28

D.4 STRATEGIES FOR RELIABLE LONG-FORM TRANSCRIPTION

• 4.1% WER is Distil-Whisper’s best long-form TED-LIUM validation result with 15-second chunks, while Whisper performs best with 30-second chunks.Long-form WER is 2.4% lower for Whisper and 3.0% lower for Distil-Whisper than each model’s short-form result; chunk length may depend on training audio length.Source: §D.4 · pp. 28–29, Table 23 · p. 29

D.5 FLASH ATTENTION 2

• Flash Attention 2 refines an IO-aware exact attention algorithm by optimizing GPU work partitioning for computational efficiency.Flash Attention uses tiling to reduce data movement between GPU high-bandwidth memory and on-chip memory.Source: §D.5 · p. 29
• 3.3 times faster than large-v2 across batch sizes, distil-large-v2 performs within 1% WER and has 0.7% lower average OOD WER than small.en at batch size 16.It is faster than base.en at batch sizes 1 and 4, and distil-medium.en is faster than tiny.en at batches 1 and 4 and base.en at batch 16, with 3.2% better OOD performance.Source: §D.5 · p. 29, Table 24 · p. 30
• 31% faster with FA2 at batch size 16, distil-large-v2 gains more than large-v2, which is 22% faster; distil-medium.en matches tiny.en with 7.8% lower average OOD WER.FA2 benefits the encoder more than the decoder; Distil-Whisper has 32 encoder layers and only 2 decoder layers, versus 32 of each in Whisper. A static key/value cache could further speed decoder inference, but is left for future work.Source: §D.5 · p. 29
• On a 16GB T4 GPU, distil-large-v2 is slower than small.en at batch sizes 4 and 16; distil-medium.en is faster than small.en at 4 but slower at 16.At batch size 1, distil-large-v2 is comparable to base.en and distil-medium.en is faster than tiny.en; at higher batches, distil-medium.en is slower than base.en.Source: §D.5 · p. 30, Table 25 · p. 30
• Distil-Whisper’s copied Whisper encoders require more memory than small.en, so T4 throughput saturation at higher batches reduces its RTF benefits.The same diminishing-benefit trend occurs for medium.en and large-v2 Whisper checkpoints.Source: §D.5 · p. 30
• T4 may suffice for Whisper and Distil-Whisper at batch size 1, but performance stagnates beyond that, making higher-memory A100 GPUs preferable.The benchmark reports RTF over 256 examples from each of four OOD test sets, using 40GB A100 and 16GB T4 GPUs.Source: §D.5 · pp. 29–30