Source-linked AI summary
The Right Tool for the Job: Matching Model and Instance Complexities
Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, Noah A. Smith
TL;DR
Growing NLP models increase inference costs, motivating a way to match each instance with an appropriately efficient model. The paper adds layer-wise BERT classifiers whose calibrated confidence scores trigger early exits, and reports improved speed/accuracy tradeoffs across five benchmarks with minimal overhead. A single trained model controls the tradeoff at inference time, although fixed exit choices can reduce performance or incur classifier overhead.
Problem
Larger NLP models increase inference costs, while instances differ in difficulty and may not require the most expensive model.
Method
The method attaches classifiers to intermediate BERT-large layers and uses calibrated confidence scores to exit early for simple instances or continue to deeper classifiers.
Results
The method delivers a better speed/accuracy tradeoff in almost all cases across three text-classification and two NLI benchmarks, reaching similar performance at up to five times faster inference.
Takeaways & Limitations
A single trained model can control the inference speed/accuracy tradeoff with one confidence threshold and requires less than 0.005% additional parameters.
Takeaways & Limitations
Always selecting the most efficient classifier can substantially reduce performance, while always selecting the most accurate classifier can be slower because earlier classifiers still run.
Abstract
from arXiv · showhide
As NLP models become larger, executing a trained model requires significant computational resources incurring monetary and environmental costs. To better respect a given inference budget, we propose a modification to contextual representation fine-tuning which, during inference, allows for an early (and fast) "exit" from neural network calculations for simple instances, and late (and accurate) exit for hard instances. To achieve this, we add classifiers to different layers of BERT and use their calibrated confidence scores to make early exit decisions. We test our proposed modification on five different datasets in two tasks: three text classification datasets and two natural language inference benchmarks. Our method presents a favorable speed/accuracy tradeoff in almost all cases, producing models which are up to five times faster than the state of the art, while preserving their accuracy. Our method also requires almost no additional training resources (in either time or parameters) compared to the baseline BERT model. Finally, our method alleviates the need for costly retraining of multiple models at different levels of efficiency; we allow users to control the inference speed/accuracy tradeoff using a single trained model, by setting a single variable at inference time. We publicly release our code.
1 Introduction
The paper targets rising NLP inference costs by matching instances of different difficulty with models of different computational expense. Its BERT-large approach uses layer-wise classifiers and confidence-based early exits, improving the speed/accuracy tradeoff with minimal overhead.
- Motivation: Inference costs rise with larger AI models and can limit adoption on real-time devices.Unlike training, inference costs recur for every production instance.
- Approach: The method matches simple instances with smaller models and difficult instances with larger models to reduce computation while retaining performance.This treats model size and instance difficulty as complementary sources of efficiency.
- Approach: BERT-large is modified with classifiers attached to selected layers, and confidence determines whether each instance exits early or continues deeper.Earlier classifiers are more efficient but potentially less accurate than later classifiers.
- Results: Across three text-classification and two NLI benchmarks, the method improves the speed/accuracy tradeoff in almost all cases.Some cases achieve similar performance while running as much as five times faster than the strongest baseline.
- Efficiency and flexibility: The approach requires less than 0.005% additional parameters and supports the speed/accuracy curve through one inference-time confidence threshold.Unlike distillation or pruning, it does not require a separately trained model for each operating point.
- Resources: The authors publicly release the code.
2 Premise: Models Vary in Size, Examples Vary in Complexity
The paper rests on two premises: larger NLP models are generally more accurate but slower, while inputs differ in processing difficulty. It therefore proposes assigning simpler instances to smaller models and complex instances to larger ones.
- Models Vary in Size: As NLP models grow larger, they generally become more accurate on downstream tasks but slower to run.
- Examples Vary in Complexity: Inputs vary in difficulty because their length and linguistic phenomena affect the processing required for analysis.
- Examples Vary in Complexity: A short sentence such as “The movie was awesome” is simpler to process than a longer sentence containing misleading phrases and figurative speech.
- Implication: The proposed solution speeds inference by matching simple instances with small models and complex instances with large models.
3 Approach: The Right Tool for the Job
The approach matches each instance with the least expensive classifier likely to solve it by using intermediate BERT classifiers and confidence-based early exits. Calibration supports these decisions, while shared computation and a single inference threshold preserve efficiency and flexibility.
- Motivation: The method approximates an oracle that identifies the cheapest accurate model for each instance.It exploits variation in both model cost and instance difficulty.
- Suite of models: Intermediate classifiers exit earlier and are more efficient but potentially less accurate than deeper classifiers.BERT’s multilayered structure supplies a sequence of classifiers with increasing computational cost.
- Confidence scores for early exit decisions: Softmax confidence scores determine whether an instance exits at the current classifier or continues to a more expensive one.Representations and predictions are computed sequentially by layer depth.
- Calibration: Temperature calibration divides logits by a learned parameter T before softmax, without changing predictions, to support early-exit decisions.The temperature is selected using development data.
- Discussion: The model reuses preceding computation, adds less than 0.005% parameters, and controls the speed/accuracy tradeoff with one inference-time confidence threshold.Four output layers add 6K to 12K parameters, while fine-tuning remains as fast as for the standard model.
- Discussion: Preliminary experiments found no benefit from feeding earlier classifier outputs into later classifiers through stacking.The authors therefore did not pursue stacking further.
4 Experiments
The experiments evaluate the approach across three text-classification datasets and two NLI benchmarks against standard and intermediate-layer BERT baselines. The setup uses batch size one for inference and varies a confidence threshold to measure the speed/accuracy tradeoff.
- Datasets: The evaluation covers three text-classification datasets and two English natural-language-inference benchmarks.The datasets are AG News, IMDB, SST, SNLI, and MultiNLI.
- Baselines: The baselines include standard BERT-large and three increasingly large efficient BERT models, each with one output layer.These baselines provide separate speed/accuracy points rather than one model spanning the tradeoff.
- Baselines: The oracle upper bound selects the fastest classifier that correctly solves each instance, or the fastest classifier available when none succeeds.It replaces confidence-based early-exit decisions with instance-level correctness information.
- Experimental setup: Inference uses batch size 1 because discrete early exits may differ across instances in a batch.The authors note that larger batches can use budgeted batch-classification methods.
- Experimental setup: Efficiency is measured as the average runtime of one test instance, with experiments repeated five times and reported using mean and standard deviation.This runtime measure is computed across the test set.
- Experimental setup: Lower confidence thresholds produce earlier exits, while threshold 1 always uses the most expensive and accurate classifier.Threshold 0 always selects the most efficient classifier.
5 Results
Across five datasets, the method generally improves the inference speed/accuracy tradeoff, while retaining similar performance and requiring comparable fine-tuning resources. Its single-model threshold also provides flexible control over this tradeoff, although extreme exit choices can reduce performance or add runtime overhead.
- 40% faster inference on IMDB and 80% faster inference on AG remain within 0.5% of the standard model.
- The method provides a similar or better speed/accuracy tradeoff in almost all cases, outperforming efficient baselines on SST and medium-budget NLI settings.
- An oracle selecting among classifiers outperforms the original baseline by 1.8% on AG to 6.9% on MNLI while running 4–6 times faster.
- With always-early or always-late exits, performance can drop relative to corresponding baselines, while always-late selection can also be slower because earlier classifiers still run.
- Fine-tuning costs are comparable to the standard BERT-large model: the approach is not slower in four of five cases and is slightly faster in three.
- A single trained model controls the speed/accuracy tradeoff through the confidence threshold, avoiding separate retraining for each operating point.
- The method also improves the speed/accuracy tradeoff for tinyBERT, while BERT-large performs best overall, especially at higher budgets.
6 A Criterion for “Difficulty”
The method’s confidence is only weakly aligned with several external difficulty measures, though it tracks consistency reasonably and varies systematically across labels. These results suggest that confidence captures a distinct notion of instance difficulty, with some evidence of class-specific bias.
- Consistency and difficulty: The method’s confidence is reasonably correlated with prediction consistency across datasets, but its relationship with other difficulty measures is generally weak.The paper compares confidence with document length, consistency, hypothesis-only predictions, and inter-annotator consensus.
- Shorter is easier?: |ρ| < 0.2 across datasets for confidence versus document length.Confidence is weakly negative with length on four datasets but positively correlated on AG.
- Comparison with hypothesis-only criteria: Confidence correlations with hypothesis-only predictions decrease for larger classifiers, while the overall difficulty definitions are not strongly correlated.The paper interprets this as larger classifiers handling harder instances beyond hypothesis-only artifacts.
- Inter-annotator consensus: Confidence correlates weakly with inter-annotator consensus on SNLI and MNLI, at 0.08 and 0.14 for the most efficient classifiers.Inter-annotator consensus is treated as a difficulty measure, with higher consensus indicating easier instances.
- Confidence across labels: Confidence varies across labels, and larger models often narrow these gaps without substantially changing their relative ordering.The paper reports this pattern across datasets and model sizes.
- Confidence across labels: High confidence sometimes reflects genuinely easy classes, but can also reflect overconfidence or underconfidence tied to model biases.Sports in AG and positive in IMDB combine high confidence with high F1, while other labels show mismatches.
7 Related Work
Prior efficiency methods mainly reduce model size or allocate fewer resources, while this work emphasizes document-level early exits. Related transformer and vision studies pursue similar dynamic computation strategies.
- Model compression: Model distillation trains a smaller student to mimic a larger teacher, while pruning removes network weights to produce a smaller, potentially faster model.These approaches target model size and inference efficiency through different mechanisms.
- Model compression: Quantization reduces numerical precision to speed numerical operations and reduce model size.It is another model-size-based efficiency strategy.
- Input-adaptive computation: Some methods allocate fewer resources to selected input components, whereas this method does so at the document level rather than for individual tokens.The comparison distinguishes document-level early exiting from token-level resource allocation.
- Dynamic early exits: Concurrent transformer studies introduced early stopping, control symbols, and other inference-time cost optimizations for sequence-to-sequence or transformer models.These works explore dynamic computation within transformer architectures.
- Dynamic early exits: Computer vision research has dynamically skipped layers or learned early-exit policies, reporting substantial computational gains.These studies provide an earlier precedent for adaptive depth in neural architectures.
8 Conclusion
The paper presents early exiting as a way to improve inference efficiency in pretrained language models by matching computation to instance difficulty. Across five datasets, it reports faster inference with similar performance and minimal training overhead.
- 8 Conclusion: The method improves the inference speed/accuracy tradeoff for pretrained language models by making early exits on simple instances.Harder instances continue through more layers, while simple instances avoid unnecessary computation.
- 8 Conclusion: Up to 80% faster inference was achieved on five text classification and NLI datasets while maintaining similar performance.The comparison is against the standard approach using BERT-large experiments.
- 8 Conclusion: The approach requires neither additional training time nor a significant number of additional parameters compared with the standard approach.The conclusion describes the added training and parameter requirements as negligible.
- 8 Conclusion: A single trained model controls the speed/accuracy tradeoff at inference time without retraining for each operating point.The control is achieved by selecting different confidence thresholds.
A Implementation Details
The experiments use standardized dropout and a single GPU, with separate fine-tuning durations for classification and NLI tasks and randomized validation-based hyperparameter selection.
- A Implementation Details: Both the proposed model and baselines use dropout 0.1 and are evaluated on a single Quadro RTX 8000 GPU.The implementation uses AllenNLP, with calibration code based on Guo et al. (2017).
- A Implementation Details: Text classification models are fine-tuned for 2 epochs, while NLI models are fine-tuned for 4 epochs.These task-specific training durations are used for the model and baselines.
- A Implementation Details: Ten random-search trials select the learning rate and random seed on the validation set.The learning rate is chosen from {0.00002, 0.00003, 0.00005}.
B Validation Results
Figure 7 reports validation accuracy and processing time for the proposed approach, standard and efficient baselines, and an oracle; the paper also lists implementation links.
- Figure 7 presents the validation results of the experiments.
- Figure 7 compares validation accuracy and processing time across the approach, standard baseline, efficient baselines, and oracle.Accuracy is higher toward the left, and higher values are better.
- The paper lists AllenNLP and temperature_scaling implementation links.