Source-linked AI summary
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
Zhengbao Jiang, Jun Araki, Haibo Ding, Graham Neubig
TL;DR
Language models can answer many QA questions but may assign unreliable confidence to their predictions, raising the problem of knowing when an answer is correct. The paper evaluates calibration in T5, BART, and GPT-2 across diverse QA tasks and tests fine-tuning, post-hoc adjustment, paraphrasing, and retrieved context. The methods improve calibration without sacrificing accuracy, although extractive and free-form QA impose important evaluation and calibration limitations.
Problem
The paper asks whether LM probability estimates align with the actual probability that QA answers are correct, despite models failing on many queries.
Method
The paper evaluates T5, BART, and GPT-2 on diverse QA datasets and calibrates them using fine-tuning, post-hoc confidence modification, paraphrasing, and retrieved context.
Results
Fine-tuning and post-hoc methods improve calibration without sacrificing accuracy, while paraphrasing answers and retrieving evidence further improve calibration.
Takeaways & Limitations
LM confidence estimates are sensitive to both input and output phrasing, while post-processing confidence is reported as universally helpful.
Takeaways & Limitations
Free-form abstractive QA is not evaluated because correctness is difficult to assess when paraphrases of correct answers remain valid.
Abstract
from arXiv · showhide
Recent works have shown that language models (LM) capture different types of knowledge regarding facts or common sense. However, because no model is perfect, they still fail to provide appropriate answers in many cases. In this paper, we ask the question "how can we know when language models know, with confidence, the answer to a particular query?" We examine this question from the point of view of calibration, the property of a probabilistic model's predicted probabilities actually being well correlated with the probabilities of correctness. We examine three strong generative models -- T5, BART, and GPT-2 -- and study whether their probabilities on QA tasks are well calibrated, finding the answer is a relatively emphatic no. We then examine methods to calibrate such models to make their confidence scores correlate better with the likelihood of correctness through fine-tuning, post-hoc probability modification, or adjustment of the predicted outputs or inputs. Experiments on a diverse range of datasets demonstrate the effectiveness of our methods. We also perform analysis to study the strengths and limitations of these methods, shedding light on further improvements that may be made in methods for calibrating LMs. We have released the code at https://github.com/jzbjyb/lm-calibration.
1 Introduction
Language models capture substantial factual and commonsense knowledge but remain unreliable on some queries, making confidence calibration important for deployment. The paper finds that T5, BART, and GPT-2 are poorly calibrated on diverse QA tasks, then evaluates methods that improve calibration.
- LMs capture factual, commonsense, and numerical knowledge, including through prompting large pretrained models to perform knowledge-based tasks.
- LMs still fail on uncommon facts and complex reasoning, while superficial training signals may not generalize to unseen cases.
- The paper defines the central problem as determining whether LM probability estimates align with the actual probability that an answer is correct.
- Despite high QA performance, T5, BART, and GPT-2 tend to be poorly calibrated, with confidence poorly corresponding to answer correctness.
- The proposed methods combine parameter fine-tuning, post-hoc confidence adjustment, paraphrased answers, and retrieved context to improve calibration.
- Experiments show that fine-tuning and post-hoc methods improve calibration without sacrificing accuracy, while retrieval and multiple answer phrasings address input-output sensitivity.
2 LM-based Question Answering
The paper treats QA as generative sequence-to-sequence prediction, evaluating calibration across multiple-choice and extractive settings. It uses candidate-answer probabilities for calibration, with extractive QA requiring computationally manageable span selection.
- LMs can solve some NLU tasks directly by predicting missing words or prompt continuations, beyond their use for pretrained representations.
- The paper differs from prior calibration work by calibrating generative LMs themselves rather than representations used to predict task labels or answer spans.
- Experiments cover diverse QA datasets spanning factual and commonsense knowledge, with broad domain coverage.
- Different QA formats are converted into a unified sequence-to-sequence task that maps question X to an answer continuation Y.
- Multiple-choice QA: For multiple-choice QA, the model scores candidate answers appended to the question and returns the highest-probability candidate.
- Multiple-choice QA: Normalized candidate probabilities provide a confidence estimate for the selected answer relative to the candidate list.
- Extractive QA: For extractive QA, every passage span is a candidate, but calibration uses a manageable candidate set because enumerating all spans is computationally costly.
- Free-form abstractive QA is excluded because evaluating correctness is difficult when paraphrases of correct answers remain valid.
3 Background on Calibration
Calibration asks whether a model’s confidence estimates align with the actual probability that its answers are correct. Expected calibration error approximates this alignment by comparing accuracy and confidence across confidence buckets, while reliability diagrams visualize the comparison.
- A model is perfectly calibrated when predictions assigned probability p are correct with probability p.
- Expected calibration error (ECE) measures calibration as a weighted average of the discrepancy between bucket accuracy and confidence.
- Each confidence bucket contains predictions within an interval, with acc(B_m) denoting average accuracy and conf(B_m) denoting average confidence.
- Reliability diagrams plot bucket accuracy as bar heights, and perfect calibration places all bars on the diagonal.
- State-of-the-art LM-based QA methods can be extraordinarily poorly calibrated, with probability estimates barely correlated with output correctness.
4 Calibrating LMs for Question Answering
The paper calibrates language models for QA using fine-tuning, post-hoc probability adjustment, and changes to candidate outputs or inputs. These approaches target confidence estimation while preserving the underlying model’s accuracy and address sensitivity to candidate wording and available evidence.
- Calibration methods comprise fine-tuning LM parameters and post-hoc methods that keep the model fixed while manipulating confidence or inputs.
- 4.1 Fine-tuning-based Calibration: Fine-tuning objectives use candidate answers: softmax maximizes the correct candidate’s normalized probability, while margin objectives separate it from negative candidates.
- 4.2 Post-hoc Calibration: Post-hoc calibration includes temperature scaling over candidate log probabilities and feature-based decision trees that estimate confidence from model and input features.
- 4.2 Post-hoc Calibration: Decision-tree features include candidate-distribution entropy, input perplexity, and input and output lengths.
- 4.2 Post-hoc Calibration: Paraphrasing candidate answers addresses wording sensitivity by aggregating probabilities across multiple paraphrases.
- 4.2 Post-hoc Calibration: Retrieving a relevant Wikipedia article and appending its first paragraph augments inputs with additional evidence for prediction and confidence estimation.
5 Experiments
Experiments evaluate calibration across diverse QA datasets, models, calibration methods, model sizes, paraphrase counts, and dataset splits. Calibration generally improves without sacrificing accuracy, with especially strong gains on multiple-choice QA and smaller gains on extractive QA.
- Baseline calibration: UnifiedQA achieves strong QA accuracy but remains poorly calibrated, with ECE higher than 0.2 on MT-test and excessive confidence near 0 or 1.The model often assigns high confidence to wrong answers.
- Calibration methods: Both fine-tuning and post-hoc methods improve ECE while maintaining accuracy on multiple-choice QA; Combo reduces ECE from 0.095 to 0.044, over 53%.Combo combines margin-based fine-tuning, temperature scaling, paraphrasing, and input augmentation.
- Dataset effects: Calibration gains are smaller on extractive QA, where candidate confidence entropy is higher than on multiple-choice QA: 0.40 versus 0.13.The authors hypothesize that span-decoded candidates are harder to calibrate than manually curated multiple-choice candidates.
- Calibration methods: Softmax- and margin-based objectives improve ECE on multiple-choice datasets by explicitly comparing candidate-answer probabilities.These objectives are argued to suit questions with potential candidates better than maximum-likelihood fine-tuning.
- Calibration methods: Post-hoc temperature scaling and feature-based decision trees rescale over-confident predictions into more reasonable confidence ranges and improve calibration across datasets.These methods manipulate confidence information while keeping the language model fixed.
- Input augmentation: More paraphrases generally improve calibration, but gains beyond 10 are subtle, making 5–10 paraphrases a practical cost-performance tradeoff.Paraphrasing helps mainly for shorter questions: average lengths are 187 for improved candidates versus 320 for candidates without improvement.
6 Related Work
Prior NLP work studied calibration in structured prediction and language understanding tasks, while LM probing examined what models know across syntactic, semantic, factual, and commonsense dimensions. This paper extends that focus to whether LMs know answers with confidence.
- Prior NLP calibration studies covered structured prediction and language understanding tasks, including extractive question answering and text classification.
- LM probing research examined syntactic, semantic, factual, commonsense, and other properties of pretrained language models.
- Unlike earlier probing work focused on what LMs know, this paper also examines when they do not know an answer with confidence.
7 Conclusion
The paper finds that strong LM-based QA models are poorly calibrated despite impressive performance, then evaluates fine-tuning and post-processing methods to improve confidence estimates. Results show these methods improve calibration, while the authors identify finer-grained calibration and free-form QA evaluation as important future directions.
- Despite impressive performance, LM-based QA models tend to have poorly calibrated probability estimates.
- The paper evaluates fine-tuning, probability post-processing, input augmentation, and candidate-answer paraphrasing to improve confidence calibration.
- Experimental results demonstrate the effectiveness of the proposed calibration methods, while further analysis reveals challenges for future work.
- Future work could develop calibration methods finer-grained than holistic calibration across an entire dataset.