PaperlayerView exact revision on arXiv

Source-linked AI summary

CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascade

Lei Li, Yankai Lin, Deli Chen, Shuhuai Ren, Peng Li, Jie Zhou, Xu Sun

arXiv:2012.14682v2cs.CL

TL;DR

Dynamic early exiting can bottleneck at high speed-up ratios because shallow representations and internal classifiers may not support accurate predictions and reliable exits. CascadeBERT cascades complete models with difficulty-aware calibration; across six GLUE classification tasks, it outperforms prior dynamic early-exiting baselines under high speed-up ratios.

  • Problem

    At high speed-up ratios, shallow PLM representations may lack semantics needed for accurate predictions, and internal classifiers may make unreliable exits.

    Source: §1 · p. 2

  • Method

    CascadeBERT progressively selects among complete PLMs from smallest to largest and uses difficulty-aware regularization to calibrate predictions for stopping.

    Source: §1 · p. 2

  • Results

    CascadeBERT achieves better task performance than previous dynamic early-exiting baselines under high speed-up ratios across six GLUE classification tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The analysis reports that difficulty-aware calibration and complete-model cascading support more calibrated and accurate predictions.

    Source: §1 · p. 2

    The shallow-feature analysis assumes that a light BERT pretrained from scratch with masked language modeling contains high-level semantic information.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Dynamic early exiting aims to accelerate the inference of pre-trained language models (PLMs) by emitting predictions in internal layers without passing through the entire model. In this paper, we empirically analyze the working mechanism of dynamic early exiting and find that it faces a performance bottleneck under high speed-up ratios. On one hand, the PLMs' representations in shallow layers lack high-level semantic information and thus are not sufficient for accurate predictions. On the other hand, the exiting decisions made by internal classifiers are unreliable, leading to wrongly emitted early predictions. We instead propose a new framework for accelerating the inference of PLMs, CascadeBERT, which dynamically selects proper-sized and complete models in a cascading manner, providing comprehensive representations for predictions. We further devise a difficulty-aware objective, encouraging the model to output the class probability that reflects the real difficulty of each instance for a more reliable cascading mechanism. Experimental results show that CascadeBERT can achieve an overall 15\% improvement under 4$\times$ speed-up compared with existing dynamic early exiting methods on six classification tasks, yielding more calibrated and accurate predictions.

1 Introduction

Dynamic early exiting can perform poorly at high speed-up ratios because shallow representations and internal classifiers may not support accurate predictions or reliable exits. CascadeBERT instead cascades complete models with difficulty-aware calibration, and outperforms prior dynamic early-exiting baselines under high speed-up ratios.

1 Introduction

  • PLM inference time hinders deployment in real-time applications, motivating approaches that accelerate inference.The introduction distinguishes model-level compression from instance-level speed-up, which adapts computation to instance complexity.Source: §1 · p. 1
  • At high speed-up ratios, shallow representations lack high-level semantics and internal classifiers can make unreliable exits, causing poor task performance.The paper’s example shows a shallow DeeBERT classifier failing on an easy MNLI instance with large word overlap.Source: §1 · p. 2
  • CascadeBERT progressively checks complete PLMs from smallest to largest and stops when a model is confident, using difficulty-aware regularization to calibrate predictions.The calibrated predictions are intended to reflect each instance’s difficulty and guide early stopping.Source: §1 · p. 2
  • CascadeBERT outperforms previous dynamic early-exiting baselines under high speed-up ratios across six GLUE classification tasks.Further analysis reports that its difficulty-aware objective calibrates predictions and that the approach generalizes.Source: §1 · p. 2

2 Investigations into Early Exiting

The investigation asks whether shallow representations support accurate predictions and whether internal classifiers make reliable exits. Its comparisons indicate that complete models perform better than shallow DeeBERT exits, while shallow DeeBERT off-ramps have lower difficulty-ranking scores.

2 Investigations into Early Exiting

  • The investigation tests whether layer representations provide enough information for accurate predictions and compares DeeBERT, fine-tuned BERT-kL, and scratch-pretrained BERT-Complete.BERT-Complete is treated as containing high-level semantic information because it is pretrained with a masked language modeling objective.Source: §2 · p. 2, §2.1 · pp. 2–3

2.1 Are Shallow Features Sufficient?

  • Complete BERT-Complete models outperform DeeBERT especially at shallow depths, while BERT-kL also outperforms DeeBERT.The authors attribute the BERT-kL advantage to task-specific adaptation during fine-tuning, which DeeBERT’s intermediate representations cannot fully exploit.Source: §2.1 · p. 3

2.2 Are Internal Classifiers Reliable?

  • Shallow DeeBERT off-ramps have lower DIS than BERT-kL and BERT-Complete, indicating less reliable exit decisions and more wrongly emitted instances.The analysis sorts instances by confidence and uses a normalized DI-Sum to calculate DIS.Source: Figure 3 · p. 3, §2.2 · pp. 3–4
  • Instance difficulty marks whether a model can predict an instance correctly, while model confidence is the maximum class probability.Difficulty is coded as 1 for instances the model cannot predict correctly and 0 for those it can handle well.Source: §2.2 · p. 3

K DISum, (3)

DIS measures how well classifier confidence ranks instance difficulty. Shallow DeeBERT off-ramps score lower than BERT-kL and BERT-Complete, while the ability to distinguish difficult from easy examples increases with layer depth.

K DISum, (3)

  • DIS measures the proportion of instance pairs correctly ranked by classifier confidence against difficulty and serves as a proxy for exit-decision quality.The normalizing factor re-scales DIS to the range from 0 to 1.Source: K DISum, (3) · p. 4
  • Shallow DeeBERT internal classifiers have lower DIS than BERT-kL and BERT-Complete, and deeper layers better distinguish difficult from easy instances.Lower DIS indicates less consistency between confidence and difficulty and is associated with unreliable exit decisions.Source: K DISum, (3) · p. 4
  • The findings motivate informing models of instance difficulty to make exiting decisions more robust.The paper links shallow-layer semantic representations to task performance and more comprehensive decisions at deeper layers.Source: K DISum, (3) · p. 4

3 Methodology

CascadeBERT replaces within-model early exits with a cascade of complete PLMs and adds difficulty-aware calibration to make confidence-based decisions more reliable. The cascade processes instances from smaller to larger models, while the regularization encourages lower confidence on difficult instances.

3 Methodology

  • CascadeBERT combines complete PLMs of increasing size with difficulty-aware calibration regularization.The cascade checks models sequentially, and the regularization informs the model of instance difficulty.Source: §3 · p. 4

3.1 Cascade Exiting

  • Each instance is processed from the smallest to largest model and emitted when its maximum class probability exceeds threshold τ.A lower τ sends more examples through the current model for faster inference; a higher τ sends more through larger models for better results.Source: §3.1 · p. 4

3.2 Difficulty-Aware Regularization

  • Difficulty-aware regularization adds a weighted, difficulty-based margin objective that adjusts difficult instances’ confidence below that of easy instances.The objective is added to the task-specific loss with weight λ, making confidence-based emitting decisions more reliable.Source: §3.2 · pp. 4–5

4 Experiments

The experiments evaluate CascadeBERT on six GLUE classification tasks against dynamic early-exiting and knowledge-distillation baselines. CascadeBERT outperforms the early-exiting baselines at tested speed-up ratios, with the largest reported margin at 4×.

4.1 Experimental Settings

  • Evaluation covers six GLUE classification tasks, using F1-score for QQP and MRPC and accuracy for the other tasks.The tasks are MNLI, MRPC, QNLI, QQP, RTE, and SST-2.Source: §4.1 · p. 5
  • The cascade selects between 2-layer and 12-layer models, and speed-up counts the layers actually executed during inference.Instances may pass through multiple models; those executed layers are included in the speed-up calculation.Source: §4.1 · p. 5

4.3 Overall Results

  • At 4× speed-up, CascadeBERT beats DeeBERT by a relative 15.5% (10.5 points on average) and outperforms all early-exiting baselines across tested ratios.Table 2 reports GLUE test results; F1-score is used for QQP and MRPC, and accuracy for the other tasks.Source: Table 2 · p. 6, §4.3 · p. 6

4.2 Baselines

  • The baselines include static early exiting, dynamic methods such as DeeBERT and PABEE, and compact models produced by knowledge distillation.The listed distillation methods include DistilBERT, BERT-PKD, BERT-of-Theseus, and BERT-PD.Source: §4.2 · p. 6

4.3 Overall Results

  • At the second layer, CascadeBERT representations show clearer class boundaries than DeeBERT representations, supporting more accurate classifier predictions.The paper presents the representation comparison as a t-SNE visualization.Source: Figure 4 · p. 7, §4.3 · p. 7
  • CascadeBERT also outperforms strong knowledge-distillation methods, including BERT-PKD and BERT-of-Theseus, at 3× speed-up.The paper attributes this comparison to the cascade's flexible options for instances with different reasoning needs.Source: Table 3 · p. 6, §4.3 · p. 7

5 Analysis

The analysis examines representation differences, difficulty-aware regularization, adding models to the cascade, and applying the framework with RoBERTa.

5.1 Visualization of Instance Representations

  • At the second layer, CascadeBERT representations have clearer class boundaries than DeeBERT's; on SST-2, DeeBERT's shallow representations are already somewhat separable.The paper notes that performance degradation across methods is negligible on SST-2, consistent with its relative ease.Source: §5.1 · p. 7

5.2 Effects of Difficulty-Aware Regularization

  • Difficulty-aware regularization raises overall performance and DIS, slightly harms task accuracy, lowers ECE, and has an exceptional DIS decrease on QQP due to its high starting DIS.The paper links the overall boost to more reliable emitting decisions and describes the ECE change as improved calibration.Source: §5.2 · pp. 7–8, Table 5 · p. 8

5.3 Impacts of More Models in Cascade

  • Adding a 6-layer model to the 2-12 cascade slightly improves performance, with marginal gains at higher speed-up ratios; the theoretical result holds under mild assumptions.The comparison is between 2-12 and 2-6-12 cascades.Source: §5.3 · p. 8, Figure 5 · p. 8

5.4 Fine-tuned Models as an Alternative

  • With fine-tuned RoBERTa-2L and vanilla KD, the cascade still clearly outperforms dynamic early-exiting baselines at approximately 3× speed-up.The table caption gives the speed-up as approximately 3× (±4%).Source: §5.4 · p. 8, Table 6 · p. 8

6 Related Work

The related-work discussion distinguishes model-level compression from instance-level inference speed-up and positions dynamic early exiting as a representative instance-level approach.

6 Related Work

  • Model-level compression includes knowledge distillation, pruning, and quantization; pruning can target attention heads or layers, while quantization uses fewer physical bits.The paper states that pruning and quantization are orthogonal to its framework and are not compared.Source: §6 · pp. 8–9
  • The paper does not compare pruning or quantization methods because those techniques are orthogonal to CascadeBERT.Source: §6 · p. 9
  • Instance-level speed-up adapts computation to instance complexity, with dynamic early exiting emitting predictions from intermediate classifiers.The paper cites applications in natural-language understanding, sequence labeling, question answering, and document ranking.Source: §6 · p. 9

7 Conclusion

The conclusion identifies a high-speed-up performance bottleneck in dynamic early exiting and presents CascadeBERT as a model-cascade alternative with difficulty-aware regularization.

  • Dynamic early exiting can bottleneck at high speed-up ratios because shallow representations are insufficient and internal classifiers make poor exiting decisions.Source: §7 · p. 9
  • CascadeBERT accelerates PLM inference by cascading complete models and applying difficulty-aware regularization.Source: §7 · p. 9
  • Experiments report substantial improvements over prior dynamic exiting methods, while further analysis finds the framework generalizable and its results more calibrated.Source: §7 · p. 9

A Comparison with FastBERT

CascadeBERT is compared with FastBERT at 3.00× and 4.00× speed-up ratios; the text reports that CascadeBERT outperforms it, especially on MNLI.

B Expected Calibration Error

Expected calibration error measures the difference between prediction confidence and accuracy across bins, with lower ECE indicating better calibration.

C Analysis for More Models in Cascade

The analysis examines whether adding a complete model improves the accuracy–speed-up trade-off, and finds that model size and exit thresholds must be chosen carefully.

Loading 2012.14682v2…