Source-linked AI summary
Reducing conversational agents' overconfidence through linguistic calibration
Sabrina J. Mielke, Arthur Szlam, Emily Dinan, Y-Lan Boureau
TL;DR
Neural dialogue research has focused heavily on factual accuracy, while transparency about ignorance is less studied. This paper tests and improves linguistic calibration by predicting answer correctness and using those predictions in controllable generation, finding poor baseline calibration but greatly improved alignment.
Problem
The paper addresses whether dialogue agents’ verbalized confidence or doubt matches the likelihood that their answers are factually correct.
Method
The authors train a correctness calibrator from dialogue-model information and use its predictions with confidence-controlled generation to adjust expressed certainty.
Results
State-of-the-art conversational models are poorly linguistically calibrated, but correctness likelihood can be well predicted and used to greatly improve calibration.
Takeaways & Limitations
Calibrated language makes model limitations more transparent in dialogue without addressing the models’ broader accuracy and generative-model problems.
Takeaways & Limitations
Improved calibration does not make current models suitable for most applications because confident answers remain often incorrect and broader issues persist.
Abstract
from arXiv · showhide
While improving neural dialogue agents' factual accuracy is the object of much research, another important aspect of communication, less studied in the setting of neural dialogue, is transparency about ignorance. In this work, we analyze to what extent state-of-the-art chit-chat models are linguistically calibrated in the sense that their verbalized expression of doubt (or confidence) matches the likelihood that the model's responses are factually incorrect (or correct). We find that these models are poorly calibrated, yet we show that likelihood of correctness can accurately be predicted. By incorporating such metacognitive features into the training of a controllable generation model, we obtain a dialogue agent with greatly improved linguistic calibration. While improving neural dialogue agents' factual accuracy is the object of much research, another important aspect of communication, less studied in the setting of neural dialogue, is transparency about ignorance. In this work, we analyze to what extent state-of-the-art chit-chat models are linguistically calibrated in the sense that their verbalized expression of doubt (or confidence) matches the likelihood that the model's responses are factually incorrect (or correct). We find that these models are poorly calibrated, yet we show that likelihood of correctness can accurately be predicted. By incorporating such metacognitive features into the training of a controllable generation model, we obtain a dialogue agent with greatly improved linguistic calibration.
1 Introduction
Neural chit-chat agents can express confidence or doubt without matching the factual correctness of their answers. The paper proposes predicting correctness and controlling linguistic confidence to improve this alignment.
- A confident response can be factually wrong, as illustrated by a chatbot claiming that 1 kg of feathers is heavier than 1 kg of stone.The example comes from BST 2.7B.
- The paper frames linguistic calibration as a way to make model limitations more transparent directly in dialogue.This supports truthful expression of what the model knows, even when factual accuracy is high or questions lack known answers.
- The paper asks whether dialogue agents’ verbalized confidence or doubt corresponds to the likelihood that their answers are correct.It also asks whether controlled generation can use this information to achieve better verbalized metacognition.
- The authors annotate a state-of-the-art chit-chat model’s responses for factual correctness and linguistic confidence, then find the model poorly calibrated.They report that linguistic confidence does not match factual correctness.
- A correctness predictor trained from the chit-chat model’s representations is used within controllable generation to greatly improve calibration.The proposed pipeline adjusts linguistic confidence according to predicted correctness probability.
2 Related Work
Related work examines factual knowledge, overconfidence, probabilistic calibration, and controlled response generation in neural language and dialogue models. These strands motivate predicting correctness and conditioning responses on desired certainty.
- Open-domain dialogue agents are neural encoder-decoder models trained on large internet-scale conversational datasets to generate responses from dialogue history.The paper distinguishes them from general-purpose language models and QA models trained to produce factual answers.
- Prior work finds overconfidence in humans and poor probabilistic calibration in modern neural networks, including models whose confidence estimates mismatch empirical correctness.Related QA research also explores abstaining when errors are likely.
- Existing findings on QA calibration are mixed: one cited result reports 63.1% accuracy among GPT2’s 1% most confident Natural Questions answers.This contrasts with cited evidence that another model is poorly calibrated on QA probability estimates.
- Controlled response generation methods manipulate attributes such as length or sentiment, including approaches that concatenate learned control tokens for multiple attributes.The paper applies this general strategy to expressed certainty.
3 Quantifying Linguistic Confidence
The section operationalizes linguistic confidence and correctness for closed-book TriviaQA answers, then evaluates human annotation and automatic proxies. It finds that confidence classification is usable but has difficulty separating nuanced uncertainty categories.
- Closed-book QA: Closed-book TriviaQA removes supporting evidence and yields 76,523 training and 9,961 validation question-answer pairs.Gold-answer aliases are retained after evidence documents are removed.
- Annotation scheme: Human annotators label each response for linguistic confidence and correctness, producing 13 taxonomy classes before binary correctness is used for modeling.Correctness maps OTHER and WRONG to incorrect, and EXTRA and RIGHT to correct.
- Annotation scheme: Annotation reliability is limited by answers such as “whetstone,” which can be judged incorrect against the alias list despite being factually plausible.Annotators also disagree about certainty and correctness for this example.
- Automatic annotation: A match-based scorer predicts binary correctness by checking whether tokenized answer aliases appear in the generation, providing an automatic proxy for human labels.The method is used when human annotation is cost-prohibitive.
- Automatic annotation: A BERT classifier trained on 2,000 annotated question-prediction pairs provides usable linguistic-confidence predictions, especially for distinguishing HI from DK/LO.It struggles to distinguish DK from LO, likely because many responses combine uncertainty with a subsequent claim.
- Vanilla model: The vanilla BST 2.7B model achieves only 4.8% test accuracy, yet answers 29.45% of questions confidently and gets just 14% of those confident answers correct.The model uses recommended standard parameters, including a minimum generated-answer length of 20 tokens.
4 Re-calibrating chatbots’ language
The pipeline improves linguistic calibration by combining a correctness calibrator with controllable generation. It adjusts confidence control tokens to align verbalized confidence with predicted correctness while managing content changes caused by confidence control.
- Pipeline overview: The base dialogue models are overconfident on TriviaQA, motivating a pipeline designed to align linguistic confidence with answer correctness.
- Pipeline overview: The pipeline trains a calibrator to predict answer correctness from the question, answer, and vanilla model representations, then uses that probability to control generation.The calibrator outputs logits for correct and incorrect classes; the generation model receives linguistic-confidence control tokens.
- Pipeline overview: The controllable generator is used with the calibrator by selecting confidence tokens that match the calibrator’s predicted correctness probability.This produces a calibrator-controlled chatbot whose linguistic confidence is intended to align with answer correctness.
- Stage 1: confidence controllable model: Stage 1 fine-tunes a confidence-controllable model on BST and TriviaQA examples labeled with BERT-based linguistic confidence, enabling selected confidence levels at inference.Incorrect and OT-labeled examples are discarded, and the remaining examples are oversampled to match the VALID SET certainty distribution.
- Stage 2: confidence-and-content controlled model: Stage 2 adds a <SAME> or <DIFF> content-control token because changing linguistic confidence can also change response content.The training target is the response produced by the Stage 1 model with the selected confidence token; examples where both responses are incorrect are discarded.
5 Results
The experiments evaluate annotation reliability, calibrator correctness prediction, controllable generation, and final linguistic calibration. The calibrator-controlled chatbot substantially improves alignment between expressed confidence and correctness while preserving overall accuracy.
- 5.1 Data collection and annotation: The TEST SET contains 4793 examples after majority-label filtering, with human annotations covering vanilla and controlled generations under DK, LO, and HI settings.The original evaluation design collected 5000 question-response pairs per model and confidence setting; filtering reduced this to 4793.
- 5.1 Data collection and annotation: Annotation agreement is 43.60% unanimous for linguistic confidence and 94.35% for binary correctness, with at least two annotators agreeing on 97.60% and 99.40%, respectively.Majority labels reduce the VALID SET from 2000 to 1793 examples and the TEST SET from 5000 to 4793.
- 5.2 Calibrator training results: The calibrator predicts correctness probability well enough to align expressed confidence with a more realistic likelihood of answering correctly.On the TEST SET, observed correctness is compared with calibrator-predicted correctness probability.
- 5.2 Calibrator training results: The calibrator achieves an ECE of 0.018 and an MCE of 0.292 under the reported calibration metrics.These metrics compare predicted likelihood with empirical correctness across probability bins; lower distances are better.
- 5.2 Calibrator training results: Questions alone predict correctness almost as reliably as the full calibrator, while a fine-tuned BERT model predicts empirical correctness reasonably accurately from words.The authors leave better use of all available information for future work.
- 5.3 Controllable generation training results: The controllable model separates confidence categories: 96.27% of HI-forced generations are judged HI, while 98.79% of DK-forced and 99.12% of LO-forced generations are judged not HI.Among vanilla-correct answers, 88.46% remain correct under HI forcing, and the controlled model loses less than one perplexity point on original validation tasks.
- 5.4 Evaluating the calibrator-controlled chatbot: Correctness among human-rated highly confident answers rises nearly threefold from 13.7% to 38.9% with calibrator-controlled generation.The improvement comes from answering fewer questions confidently; overall accuracy changes from 4.8% to 5.1%, not significantly under a paired permutation test.
6 Conclusion
The study finds that linguistic calibration can be greatly improved using correctness predictions, but confident answers remain frequently incorrect. It therefore presents calibration as increased transparency rather than a sufficient remedy for broader deployment concerns.
- BlenderBot is poorly linguistically calibrated, expressing confidence for answers that are very likely incorrect.
- A trained calibrator can predict correctness likelihood well and support alignment between expressed confidence and realistic answer accuracy.
- Controlled generation using calibrator predictions greatly improves the model’s linguistic calibration.
- Calibration does not address low accuracy or the broader issues of generative models, so it is not sufficient for most applications beyond entertainment and research.
- Inference-time control tokens allow confidence expression to be adjusted for sensitivity, conversational openness, or low-stakes casual settings.
- The test-set examples include factual errors involving religion, geography, sports, history, and science questions.