PaperlayerView exact revision on arXiv

Source-linked AI summary

Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery

Rasmus Aagaard, Nicki Skafte Detlefsen

arXiv:2609.27980v1cs.CLcs.LG

TL;DR

Encoder pruning in Whisper remains comparatively overlooked, leaving open whether multiple layers can be removed without major transcription degradation. This paper ranks encoder layers by leave-one-layer-out WER change, removes six, and applies label-free distillation. Mean WER across four languages is 20.1% after distillation, versus 21.9% after zero-shot pruning and 18.2% at baseline.

  • Problem

    Encoder pruning remains comparatively overlooked in Whisper, despite potential to improve deployment options for resource-constrained devices; whether multiple layers can be removed without major transcription degradation remains uncertain.

    Source: §1 · p. 1

  • Method

    The approach ranks encoder layers by leave-one-layer-out WER change, removes the six least important, and distills the pruned encoder to match the full encoder’s hidden states using MSE loss.

    Source: §2 · p. 2

  • Results

    20.1% mean WER after label-free distillation across all languages, versus 21.9% after zero-shot pruning and 18.2% at baseline.

    Source: §5 · p. 4

  • Takeaways & Limitations

    The pruned encoder is 1.22-1.24× faster and requires no inference-framework changes, with 1.75× speedups at larger batch sizes on consumer hardware.

    Source: §5 · p. 4

    Distillation used English-only audio, leaving uncertain whether multilingual data would improve recovery in the languages most affected by pruning.

    Source: §4.4 · p. 4

Abstract

from arXiv · show

Pruning large pre-trained transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as pruning the decoder led to significant end-to-end transcription speedups. For instance, the {\tt whisper-large-v3-turbo} variant reduced the decoder from 32 to 4 layers, while Distill-Whisper similarly reduced the decoder to only 2 layers. Although some attention has been put towards reducing the size of the encoder, no approach has seen wide adoption. This could be due to the need for custom inference implementations to take advantage of the compressed model. We present an approach that ranks encoder layers by the leave-one-layer-out change in Word Error Rate (WER). The six layers that cause the least change are removed, corresponding to $18.5\%$ of the encoder stack. The pruned model requires no custom inference code as it is simply a more shallow encoder with fewer layers. We further distill using unlabeled monolingual speech data to recover performance degradation caused by the zero-shot layer pruning. Mean WER across four languages increases to $20.1\%$ after distillation, compared to $21.9\%$ zero-shot, going from a baseline of $18.2\%$. We release all of our code (https://github.com/rasgaard/whisper-encoder-layer-prune) and the pruned model (https://huggingface.co/rasgaard/whisper-large-v3-turbo-encoder-pruned).

1 Introduction

The introduction motivates pruning Whisper’s often-overlooked encoder to exploit layer redundancies, and presents six-layer removal and knowledge distillation as a compression strategy with minor performance degradation.

Introduction

  • Because removing individual encoder layers causes only small WER changes, the authors hypothesize that dropping several together can avoid major transcription degradation.Source: §1 · p. 1
  • Removing six encoder layers causes only minor performance degradation, and knowledge distillation further closes the gap by training the pruned encoder to match the original encoder’s hidden states with MSE loss.The original encoder serves as teacher and the pruned encoder as student.Source: §1 · p. 1
  • The proposed approach reduces whisper-large-v3-turbo’s encoder stack by 18.5% and works with existing inference libraries without architectural changes.Source: §1 · p. 1

2. We show that multilingual performance lost from zero-

The section argues that label-free knowledge distillation can largely recover multilingual performance lost to zero-shot layer pruning with modest training and data, and that data-driven layer selection is essential compared with random selection.

  • Label-free knowledge distillation can largely recover multilingual performance lost from zero-shot pruning with a modest training and data budget.Source: §2 · p. 2
  • Data-driven layer selection is essential relative to random selection; removing critical encoder layers 0 or 31 causes catastrophic degradation.Source: §2 · p. 2, Figure 2 · p. 2

2 Methods

The method ranks Whisper encoder layers by leave-one-layer-out WER change, removes six low-impact layers, and uses unlabeled speech to distill the pruned encoder toward the full encoder’s hidden states.

Layer importance ranking

  • The authors rank all 32 encoder layers by leave-one-layer-out ∆WER on FLEURS, averaging results across Danish, English, German, and French.They use the full encoder and 32 encoders each missing one layer.Source: §2 · p. 2
  • The first half of the encoder is least sensitive to layer removal, while layers 0 and 1 are clear outliers because they transform the signal significantly.The authors observe that residual connections carry most of the signal and hidden states remain highly similar across layers.Source: §2 · p. 2

Pruning and distillation

  • They remove layers [5, 6, 7, 9, 11, 12] and distill the pruned encoder to match the full encoder’s hidden states using MSE loss on unlabeled audio.The removed-layer set is defined as R = {L5, L6, L7, L9, L11, L12}, yielding encoder Enc/R.Source: §2 · p. 2

Distillation training setup

  • Distillation freezes the full encoder and decoder and trains the pruned encoder with AdamW, batch size 8, for 2,000 steps on English-only People’s Speech validation data.The run took about half an hour on an Nvidia A100 GPU, and mean WER increase was evaluated every 500 steps to track convergence.Source: §2 · p. 2

3 Results

Zero-shot encoder pruning increased WER, while distillation partly recovered the loss. Removing six encoder layers also reduced model size and improved inference speed.

  • A 3.8% WER increase after zero-shot pruning fell to a 1.9% increase after distillation.Source: §3 · p. 2
  • Removing six encoder layers eliminated 118M parameters and cut bfloat16 model size from 1543 MB to 1318 MB, saving 225 MB without decoder or inference-code changes.Source: §3 · p. 2
  • The encoder accounted for 56% of transcription time at batch size 8 versus 36% at batch size 1.Source: §3 · p. 2
  • Random six-layer selections produced ∆WER increases from +5 to +471, compared with +3.8 for the ∆WER-guided selection.The random selections were drawn from layers 1–15.Source: Figure 3 · p. 3
  • 1.22–1.24× encoder speedups yielded end-to-end speedups of 1.75× at batch size 8 and 1.08× at batch size 1 on an Apple M4 Pro.Benchmarks used Transformers with Apple’s MPS backend and a 60-second out-of-distribution English clip; results were averaged over five runs.Source: §3 · pp. 2–3, Table 3 · p. 3

4 Discussion

The discussion finds that layer selection strongly affects pruning outcomes, with a sharp degradation boundary beyond six removed layers. It also reports multilingual recovery from English-only distillation while noting limits in language and model scope.

4.1 Sensitivity to layer selection

  • Across 50 randomly sampled k=6 combinations from layers 1–15, selection strongly affected performance, and ranking by ΔWER importance retained performance effectively.Source: §4.1 · p. 3

4.2 Removing > 6 layers

  • At k=7, zero-shot degradation jumps sharply; even the best seventh layer, L8, causes 2.6× the degradation of the optimal k=6 selection.L8 creates a contiguous five-layer gap (L5–L9), which may be less damaging than fragmenting the missing processing, but the gap still exceeds what the encoder can handle.Source: Figure 4 · p. 3, Figure 5 · p. 3, §4.2 · p. 3

4.3 Multilingual recovery through monolingual distillation

  • English-only label-free distillation recovers performance across all evaluated languages, with Danish recovering the most despite being expected to take the largest pruning hit.The authors suggest that MSE loss on encoder hidden states recovers acoustic representations rather than language-specific features.Source: §4.3 · p. 3

4.4 Limitations

  • Findings cover a small language sample and Whisper large-v3-turbo’s encoder; broader low-resource or model-size tests may differ, and multilingual distillation remains untested for Danish and French.Danish and French had the worst zero-shot pruning degradation but the greatest recovery from English-only distillation.Source: §4.4 · p. 4

5 Conclusion

The conclusion demonstrates that leave-one-layer-out WER can identify six removable Whisper encoder layers, and that label-free distillation recovers some of the resulting performance loss. The pruned encoder is faster and requires no inference-framework changes.

5 Conclusion

  • Removing the six least-important layers, selected by leave-one-layer-out change in Word Error Rate, reduces the encoder stack by 18.5%.The authors conclude that layer pruning is feasible for whisper-large-v3-turbo’s encoder.Source: §5 · p. 4
  • Mean WER rises from 18.2% to 21.9% after zero-shot pruning, then falls to 20.1% after label-free distillation on English-only audio across languages.The authors also compare the layer-importance metric with a random baseline and suggest the MSE objective realigns acoustic signals rather than language-specific features.Source: §5 · p. 4
  • The pruned encoder is 1.22-1.24× faster and reaches 1.75× speedups at larger batch sizes on consumer hardware, without inference-framework changes.It produces a shallower network that can serve as a drop-in replacement.Source: §5 · p. 4
  • Future work should investigate why performance degrades sharply beyond k = 6 and whether the findings generalize to Cohere Transcribe.The authors refer to this sharp degradation as the cliff-phenomenon.Source: §5 · p. 4
Loading 2609.27980v1…