Source-linked AI summary
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, Jimmy Lin
TL;DR
Large pre-trained language models such as BERT are powerful but slow enough to challenge real-time deployment. DeeBERT adds confidence-based early exits between transformer layers to accelerate inference, achieving up to ∼40% savings with minimal quality degradation. The paper also identifies transformer-layer redundancy while leaving pre-training unchanged.
Problem
Large pre-trained language models improve NLP applications but have inference latencies that challenge real-time deployment.
Method
DeeBERT inserts classification off-ramps between BERT transformer layers and exits a sample when the current off-ramp is confident.
Results
Up to ∼40% inference acceleration was achieved on BERT and RoBERTa across six GLUE datasets with minimal downstream model-quality degradation.
Takeaways & Limitations
DeeBERT exploits redundancy in BERT models to provide quality–efficiency trade-offs and reveals differing transformer-layer behavior.
Takeaways & Limitations
The training method reduces capacity available to intermediate off-ramps, and the causes of layer redundancy and differing sample exit depths remain unknown.
Abstract
from arXiv · showhide
Large-scale pre-trained language models such as BERT have brought significant improvements to NLP applications. However, they are also notorious for being slow in inference, which makes them difficult to deploy in real-time applications. We propose a simple but effective method, DeeBERT, to accelerate BERT inference. Our approach allows samples to exit earlier without passing through the entire model. Experiments show that DeeBERT is able to save up to ~40% inference time with minimal degradation in model quality. Further analyses show different behaviors in the BERT transformer layers and also reveal their redundancy. Our work provides new ideas to efficiently apply deep transformer-based models to downstream tasks. Code is available at https://github.com/castorini/DeeBERT.
1 Introduction
DeeBERT addresses the high inference latency of large pre-trained language models by allowing BERT samples to exit at intermediate layers. Experiments report substantial inference savings with minimal downstream quality degradation.
- Large pre-trained language models improve NLP applications but remain slow and difficult to deploy in real-time or hardware-constrained settings.
- DeeBERT hypothesizes that intermediate BERT layers can provide sufficient features to classify some samples.
- DeeBERT inserts classification off-ramps between transformer layers and returns a prediction when the current off-ramp is confident.Otherwise, the sample proceeds to the next transformer layer.
- Up to ∼40% inference acceleration was observed on BERT and RoBERTa across six GLUE datasets, with minimal downstream model-quality degradation.
2 Related Work
Related work examines transformer redundancy and efficiency through pruning, quantization, distillation, and early-exit approaches. DeeBERT differs by targeting efficiency while minimizing quality degradation.
- Prior studies analyze redundancy in transformer attention heads.
- Efficiency methods include Q-BERT quantization and LayerDrop structured pruning at inference time.
- Knowledge-distillation methods compress BERT into smaller transformer-based or non-transformer-based models.
- DeeBERT is inspired by earlier early-exit work but focuses on improving efficiency with minimal quality degradation.
3 Early Exit for BERT inference
DeeBERT modifies BERT fine-tuning and inference by adding an off-ramp after each transformer layer. Entropy thresholds dynamically determine whether each sample exits early or continues, trading inference speed against accuracy.
- DeeBERT leaves pre-training unchanged and adds one off-ramp for each transformer layer during fine-tuning and inference.The original BERT classification layer serves as the last off-ramp.
- For fine-tuning, each off-ramp computes cross-entropy loss over the downstream training set using its output.
- Fine-tuning first optimizes the final off-ramp, then freezes those parameters while updating earlier off-ramps.Freezing preserves the final off-ramp’s optimal output quality.
- At inference, an off-ramp compares the entropy of its output probability distribution with threshold S to decide whether a sample exits.
- A larger S produces faster but less accurate inference, whereas a smaller S produces more accurate but slower inference.
- Single-layer off-ramps are used because predictions from different layers are highly correlated and ensembles provide no significant improvement.
4 Experiments
Experiments evaluate DeeBERT on BERT and RoBERTa across six GLUE classification datasets, measuring quality–efficiency trade-offs and layerwise behavior. Early exiting preserves quality over an initial savings range, exposes redundancy, and can substantially reduce inference runtime.
- Experimental setup: DeeBERT is evaluated on BERT and RoBERTa using six GLUE classification datasets, with runtime measured on a single NVIDIA Tesla P100.The datasets are SST-2, MRPC, QNLI, RTE, QQP, and MNLI.
- Main results: Quality generally remains mostly unchanged until a turning point in runtime savings, after which performance gradually declines for both models.The turning point typically occurs earlier for BERT than for RoBERTa.
- Main results: Early exiting can produce occasional quality spikes, possibly because smaller effective models provide regularization on some datasets.The paper notes examples in RoBERTa on SST-2 and in both models on RTE.
- Efficiency measurement: Expected saving is the fraction of transformer-layer execution avoided, and its linear relationship with measured saving supports runtime measurements as stable efficiency estimates.Expected saving depends on the number of layers and the number of samples exiting at each layer.
- Main results: RoBERTa often provides more runtime savings at roughly the same quality, but its quality drops faster than BERT’s after the turning point.Layerwise analyses associate this pattern with different quality trajectories across transformer layers.
- Layerwise analyses: Layerwise results reveal redundancy: the last several layers of BERT-large and RoBERTa-large often add little improvement, with slight declines in some cases.For base models, entropy thresholds direct samples toward earlier off-ramps when those layers offer comparable quality.
5 Conclusions and Future Work
DeeBERT exploits redundancy in BERT models to improve the quality–efficiency trade-off, accelerating inference while revealing layerwise patterns. The paper leaves open questions about balancing off-ramp capacity and explaining layer redundancy and sample difficulty.
- DeeBERT exploits redundancy in BERT models to achieve better quality–efficiency trade-offs.
- Experiments show that DeeBERT accelerates BERT and RoBERTa inference by up to ∼40%.
- The training method preserves quality at the last off-ramp but reduces model capacity available to intermediate off-ramps.
- Future work should seek better balance across off-ramps and explain transformer-layer redundancy and differences in sample exit difficulty.