Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems

1 Introduction

• Existing efficiency techniques often require architectural changes, network manipulation, or complex training, while efficiency gains can come with accuracy compromises.The introduction identifies model cascading as an underexplored alternative.Source: §1 · pp. 1–2
• Model cascading uses models of varying capacities and passes an instance onward when its MaxProb confidence falls below a threshold.The approach processes models sequentially, allowing confident predictions to exit early and avoiding larger-model inference for those instances.Source: §1 · p. 2
• 88.93% computation improvement over the largest model (M3) on QQP in K=3, and 2.18% accuracy improvement over M3 on CB in K=3.Similar improvements are reported across settings with different numbers of models.Source: §1 · p. 2
• DTU achieves the highest average AUC of accuracy-cost curves among the compared cascading methods.The AUC summarizes accuracy across computation costs, with larger values indicating higher average accuracy.Source: §1 · p. 2

2 Related Work

• Prior NLP efficiency work includes pruning, quantization, distillation, input reduction, dynamic exiting, and adaptive model sizing.DeeBERT adds conditional exits between transformer layers; DynaBERT adjusts model width and depth.Source: §2 · p. 3
• Cascading remains underexplored in NLP and differs from ensembling by selecting one or more models instead of always using all available models.The paper also notes cascading approaches in machine learning and vision.Source: §2 · p. 3
• Unlike existing methods that typically require model changes or complex training, cascading is easy to implement and does not require such modifications.A single cascading system can meet different computation constraints, and routing can send an instance directly to a suitable model.Source: §2 · p. 3

3 Model Cascading

• Cascading aims to combine high prediction accuracy with efficient use of available models, between relying only on the smallest model and using all models.Source: §3 · p. 3

3.1 Formulation

• The formulation orders models by inference cost and represents which models handle each instance, measuring average dataset cost alongside prediction accuracy.Source: §3.1 · p. 3
• Efficacy is assessed using accuracy-cost AUC, cost at individual-model accuracy levels, and maximum accuracy relative to the largest model.Source: §3.1 · p. 4

3.2 Approaches

• MaxProb thresholds route instances sequentially, allowing confident cases to stop early while less confident cases proceed to larger models.Source: §3.2 · p. 4
• DTU uses the distance between a model’s softmax distribution and a uniform distribution, reflecting the full output distribution rather than only its highest probability.Source: §3.2 · p. 4
• Other approaches use input length as a heuristic, route instances directly to suitable models using MaxProb, or select instances randomly as a baseline.Source: §3.2 · p. 5

4.1 Experimental Details

• Experiments cover ten NLU classification datasets, BERT variants from mini to large, and multiple cascade sizes; input sequence lengths vary by dataset.Experiments use batch size 32 and Nvidia V100 GPUs.Source: §4.1 · p. 5

4.2.1 Problem Setup

• The K=2 setup uses BERT-medium as M1 and BERT-base as M2; other model combinations are analyzed in Appendix C.BERT-medium has 41.7M parameters and BERT-base has 110M.Source: §4.2.1 · p. 5

4.2.2 Results

• In K=2, the cascade matches M2 accuracy at lower computation cost across datasets; beyond that cost, QQP accuracy reaches 90.39% versus M2’s 89.99%.The efficiency benefit comes from handling many instances with smaller models and passing only a few to larger ones.Source: §4.2.2 · pp. 5–6, Figure 2 · p. 6
• At the K=2 MNLI matched-accuracy point, M1 handles 78% of instances and its accuracy on instances passed to M2 is 33.12% lower than on those it answers.M2 is 10.12% more accurate than M1 on the passed instances.Source: §4.2.2 · p. 7, Figure 3 · p. 7

4.3.1 Problem Setup

• The K=3 setup uses BERT-mini (11.3M parameters), BERT-medium (41.7M), and BERT-base (110M); medium is M2 here, unlike in K=2.Source: §4.3.1 · p. 7

4.3.2 Results and Analysis

• In K=3, the QQP cascade matches M3 accuracy using 11.07% of M3’s computation, while CB accuracy reaches 83.93%, above M3.The K=3 efficiency improvement is larger than in K=2.Source: §4.3.2 · p. 7, Figure 4 · p. 8
• In K=3, DTU achieves the highest AUC on average; in K=2, MaxProb and DTU beat both baselines and perform roughly equally on average.In K=2, the baselines are Random and Heuristic.Source: Table 2 · p. 6, §4.3.2 · p. 7, §4.2.2 · p. 7
• At M3-matched accuracy in K=3 MNLI, M1’s accuracy drops 20.04% on instances passed to M2, and M2’s drops 28.53% on instances passed to M3.Adding BERT-mini splits BERT-medium’s 78% K=2 workload between mini and medium in K=3 while maintaining accuracy.Source: §4.3.2 · pp. 7–8, Figure 5 · p. 8

• Model cascading is formulated and explored through several methods, with experiments on 10 diverse NLU datasets showing gains in efficiency and prediction accuracy.Source: §5 · p. 9
• Thresholds can be tuned on validation data to reflect a system's computation budget: lower thresholds favor low-cost models, while higher thresholds target accuracy.Source: §5 · p. 9
• Outlier/OOD detection is proposed as a future option for routing instances to larger cascade models.Source: §5 · p. 9
• Linear or LSTM models are proposed for future cascades, potentially increasing efficiency because their computation costs are significantly lower than transformers while performance is nontrivial.Source: §5 · p. 9

Limitations

• Storing multiple models is a potential downside, but mini and medium models in K=3 require only 0.44 times the base model's space, which authors say is not serious.Source: Limitations · p. 9

A Efficiency in NLP

• The work focuses on inference efficiency, proposing model cascading across models of varying capacities to output predictions efficiently and accurately.Source: A Efficiency in NLP · p. 12

• The evaluation covers diverse NLU datasets spanning natural language inference, duplicate detection, and sentiment classification.Source: B Dataset Statistics · p. 12

C.1.1 Medium and Large

• Figure 6 compares MaxProb and Random accuracy-cost curves for BERT-medium (M1) and BERT-large (M2); MaxProb clearly outperforms Random and has considerably higher AUC.Source: §C.1.1 · p. 12

C.1.2 Mini and Large

• Figure 7 compares MaxProb and Random accuracy-cost curves for BERT-mini (M1) and BERT-large (M2); MaxProb clearly outperforms Random and has considerably higher AUC.Source: §C.1.2 · p. 12

C.2.1 Medium and Base

• The analysis compares M1 and M2 contributions when M2’s accuracy matches the cascading system’s, with MNLI analyzed in section 4.2 and other datasets shown here.Source: §C.2.1 · p. 12

D.1.1 Mini, Medium, and Base

• Figure 9’s contribution analysis considers M1, M2, and M3 when M3 matches the cascade’s accuracy; MNLI is analyzed in section 4.3, with other datasets shown here.Source: §D.1.1 · pp. 12–13
• Figure 6 shows accuracy-cost curves for K=2 with M1 as BERT-medium and M2 as BERT-large.Source: Figure 6 · p. 13
• Table 4 presents statistics for the evaluation datasets considered in the work.Source: Table 4 · p. 13

D.2 Overall Efficiency and Accuracy Improvement

• Figure 7 shows K=2 accuracy-cost curves with M1 as BERT-mini and M2 as BERT-large.Source: Figure 7 · p. 14
• Figure 8 depicts M1 and M2 contributions for K=2 with BERT-medium and BERT-base models.Source: Figure 8 · p. 14
• Figure 9 depicts M1, M2, and M3 contributions for K=3 with BERT-mini, BERT-medium, and BERT-base models.Source: Figure 9 · p. 15
• 88.93% computation improvement over M3 on QQP and 2.18% accuracy improvement over M3 on CB are reported for K=3.Source: §D.2 · p. 13, Figure 10 · p. 15