Source-linked AI summary
Large Language Models Are Overconfident in Their Own Responses
Mario Sanz-Guerrero, Manuel Mager, Katharina von der Wense
TL;DR
Instruction-tuned LLMs are miscalibrated, but the role of chat formatting and model answer ownership remains unclear. The paper separates these effects and finds an ownership bias, then uses user-framed answers to reduce overconfidence and improve calibration without retraining.
Problem
The study asks what drives instruction-tuned LLM miscalibration and whether end users can mitigate it easily.
Method
The authors compare base and instruction-tuned models with and without chat templates, evaluating confidence through three elicitation methods.
Results
Models were more miscalibrated on self-generated answers, with average differences of 9.8% in ECE and 8.8% in Brier score for P(True).
Takeaways & Limitations
Presenting the model’s answer as user input during confidence elicitation reduces overconfidence and recovers calibration comparable to base models without retraining.
Takeaways & Limitations
The results may not generalize to all proprietary closed-source models because their post-training recipes may differ.
Abstract
from arXiv · showhide
Prior work has shown that instruction-tuned large language models (LLMs) are less well calibrated than their base pre-trained counterparts. However, little is known about the frequently used chat template's effect on the calibration of conversational LLMs. In this work, we investigate the mechanisms driving this miscalibration by decoupling the effects of the post-training algorithm and the chat format. We find that, while instruction tuning fundamentally harms calibration, the chat template aggravates the issue through an "ownership bias" -- models are significantly more confident in their own answers than in identical answers provided by a user. Extensive experiments across six recent open-weight LLMs, three benchmarks, and three confidence elicitation methods show that models assign up to 26% higher confidence to their own responses. Leveraging this insight, we propose a simple inference-time strategy: framing the model's answer as user input during confidence elicitation. This approach significantly reduces overconfidence and improves calibration by up to 26% without the need for retraining, narrowing the gap between base and instruction-tuned models.
1 Introduction
Reliable confidence estimation is essential for trustworthy LLM deployment, yet instruction-tuned models are significantly miscalibrated despite base models generally being well calibrated. The paper separates instruction tuning from chat-template effects and finds post-training is the main source of miscalibration, while the chat template further worsens it.
- Motivation: Well-calibrated models should be highly confident only when correct, but instruction-tuned LLMs show significant miscalibration whereas base LLMs are generally well calibrated.Reliable confidence estimation is framed as crucial for safe and trustworthy AI, especially in high-stakes applications.
- Ownership bias: LLMs are overconfident in their own answers regardless of whether those answers are correct, contributing to miscalibration.The figure illustrates this behavior using real outputs from Llama 3.1 (8B).
- Research questions and contributions: The study isolates instruction tuning from chat-template effects and identifies post-training as the main source of reduced calibration.The chat template, frequently introduced during instruction tuning, further aggravates the calibration problem.
2 Related Work
Prior work links LLM miscalibration to post-training and several mechanisms, while exploring verbalized confidence and resource-intensive mitigation methods. This work instead identifies overconfidence in assessing the model’s own answers and proposes an inference-time remedy without retraining.
- The Impact of Post-training on LLM Calibration: Post-training through SFT and RLHF significantly degrades the generally well-calibrated probabilities of pre-trained LLMs.
- Mechanisms of Miscalibration: Reward-model bias toward high-confidence responses and preference collapse have been proposed as mechanisms underlying miscalibration in instruction-tuned LLMs.
- Verbalized Confidence and Elicitation: Verbalized probability and confidence prompting have been studied as alternatives because post-trained LLM logits are poorly calibrated.
- Mitigation Strategies: Existing mitigation strategies include disagreement-aware alignment, reward modeling, calibration-aware fine-tuning, and auxiliary uncertainty models, whereas this work uses inference-time framing without retraining.
3 Finding the Root of Miscalibration
Instruction tuning is the main driver of miscalibration, while applying a chat template further worsens calibration despite modestly improving accuracy. The study isolates these effects by comparing base, instruction-tuned without chat formatting, and instruction-tuned chat-formatted models.
- Methods: The study compares base models, instruction-tuned models without chat templates, and instruction-tuned models with chat templates to isolate each factor.Experiments use paired base and instruction-tuned versions across model families and scales.
- Results: 3.7% accuracy gain from instruction tuning accompanies a 13.1% ECE increase and a 6.5% Brier score increase.Instruction tuning improves accuracy but harms calibration.
- Results: +1.1% accuracy gain from applying the chat template adds 2.74% to ECE and 1.5% to Brier score.Chat formatting further aggravates the calibration harm caused by instruction tuning.
- Implications: Instruction tuning is the main driver of miscalibration, while the chat template also contributes to the problem in widely used conversational systems.These findings motivate methods for obtaining model confidence more reliably.
4 Does Explicitly Asking for Confidence Alter the Miscalibration Trends?
Explicitly eliciting confidence does not change the main miscalibration trend: instruction-tuned models remain significantly less calibrated than base models, with or without a chat template. The results therefore identify post-training as the root cause of miscalibration.
- Evaluation: The evaluation forces confidence estimates for each of four possible answers, enabling calibration assessment for both correct and incorrect answers using ECE and Brier score.Standard accuracy is not computed because the model estimates confidence for every option independently.
- Results: Instruction-tuned models remain significantly worse calibrated than base models across all three confidence-elicitation methods, regardless of the chat template.The methods are P(True), Verbalized Percentage, and Verbalized Linguistic.
5 Are LLMs Overconfident in Their Own Answers?
LLMs are more miscalibrated and more confident when evaluating answers presented as their own responses than identical answers framed as user input. This ownership bias appears across confidence elicitation methods and tasks, and reframing answers as user input reduces overconfidence.
- Results: Models were more miscalibrated when they provided the answer themselves: all calibration deltas were positive, with average differences of 9.8% in ECE and 8.8% in Brier score.P(True) produced the smallest differences for both calibration metrics, and most differences were statistically significant.
- Results: Across all three confidence estimation methods, calibration was better when answers were presented as user messages rather than assistant responses.This pattern was observed in the reliability diagrams for all models and methods.
- Results: 26.8% was the largest average confidence increase for assistant-provided answers, compared with 15.8% for the smallest-difference method, P(True).Confidence distributions showed a clear shift toward higher confidence when the assistant provided the answer.
- Mitigation: Reframing an answer as user input significantly reduces overconfidence and can recover calibration comparable to, or sometimes better than, base models.The authors propose this as an inference-time strategy for obtaining a more trustworthy confidence estimate without relying on the model’s ownership of the answer.
- Ownership bias: Assistant responses received higher confidence than user-framed answers for both correct and incorrect answers, including erroneous high confidence for incorrect self-generated answers.Most confidence comparisons lay above the equal-confidence diagonal, especially for incorrect answers whose user-framed confidence was close to 0.
- Cross-task generality: The same pattern extends across GSM8K, TruthfulQA, and open-ended MMLU, where user-framed answers were consistently better calibrated and less overconfident.On GSM8K, self-generated answers produced up to 19.5% higher confidence and a 14.2% increase in ECE.
6 Conclusion
The chat format is not the main driver of instruction-tuned LLM miscalibration, but it critically shapes confidence perception through an ownership bias. Models are significantly more confident in their own responses than in identical user-provided answers, regardless of correctness.
- The chat format itself is not the main driver of instruction-tuned LLM miscalibration.
- Models exhibit an inherent ownership bias, assigning significantly higher confidence to their own responses than to identical answers provided by a user.
- This ownership bias holds regardless of whether the response is correct.
- The authors build on this ownership-bias finding to propose a simple inference-time strategy.
Limitations
The study’s evidence is concentrated on open-weight LLMs and objective question answering, limiting how confidently its findings generalize to closed-source models and subjective tasks. Its mitigation reduces miscalibration at inference time but does not change model weights or address alignment-induced causes.
- Model coverage: Most experiments use open-weight LLMs, so the persistence and degree of overconfidence in all proprietary closed-source models cannot be guaranteed.The analysis includes GPT-5.2 and a proprietary model, but closed-source post-training recipes may differ.
- Mitigation scope: The proposed framing of model outputs as user inputs is an inference-time mitigation that reduces miscalibration without changing model weights or root causes.It leverages chat format and narrows the gap between base and instruction-tuned models, but does not address overconfidence introduced during alignment.
- Task coverage: The evaluation remains limited to objective question answering, leaving overconfidence in subjective tasks such as open-ended generation unresolved.Subjective tasks make correctness less clearly defined and confidence estimation more ambiguous.
A Confidence Elicitation Prompts
The section specifies confidence-elicitation prompts for assistant- and user-provided answers across three methods, with confidence extracted from the most probable completion. It also describes label-length normalization and the adaptation of these prompts to open-ended questions.
- Confidence Elicitation Prompts: Prompts measure confidence in assistant- and user-provided answers using P(True), Verbalized Percentage, and Verbalized Linguistic methods.The full prompts for the latter two methods appear in Figures 8 and 9, while Figure 3 presents the P(True) prompts.
- Confidence Extraction: Confidence is extracted from the completion with the highest log-probability among the available options.This procedure is used in all confidence-elicitation settings.
- Confidence Extraction: Normalized Contextual Calibration subtracts each option’s neutral-context log-probability from its original-prompt log-probability to address label-length bias.The neutral context omits the question, correcting the tendency to favor shorter labels.
- Open-Ended Tasks: Open-ended tasks reuse the multiple-choice templates, replacing the option list with one assistant- or user-provided answer for confidence judgment.This setup represents a closed-book, open-ended, zero-shot question-answering scenario.
B Full Results
Table 6 reports full MMLU calibration results across base, instruction-tuned, and chat-format settings. It evaluates both ECE and Brier score under three confidence elicitation methods and compares assistant-versus-user answer positions.
- Calibration results: Table 6 evaluates calibration on MMLU using ECE and Brier score across three confidence elicitation methods.The table provides full results for all models.
- Calibration results: The comparison includes base models without instruction tuning, instruction-tuned models without chat format, and instruction-tuned models with chat format.These settings isolate the effects of instruction tuning and chat formatting.
- Calibration results: Chat-format instruction-tuned models are evaluated when answering as either the assistant or the user.This enables comparison of the two answer positions within the chat setting.