PaperlayerView exact revision on arXiv

Source-linked AI summary

Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Models

Prateek Chhikara

arXiv:2502.11028v3cs.CLcs.AI

TL;DR

The paper examines LLM confidence miscalibration and the limited evidence on explicit distractors in structured QA. It compares nine models across three factual QA datasets in free-generation and distractor-augmented settings. Distractors improve calibration overall, especially for smaller models, but calibration gains vary and the study’s findings are conditional on its generator–judge pair.

  • Problem

    How explicit distractors affect calibration and confidence ranking in modern LLMs remains insufficiently examined despite their prevalence in structured QA.

    Source: §1 · p. 2

  • Method

    The study benchmarks nine LLMs on three factual QA datasets, comparing free-generation with distractor-augmented prompts and elicited confidence.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Structured distractors significantly improve calibration, particularly for smaller models, though large models can show degraded calibration on simpler queries; GPT-4o-mini accuracy rose from 8.5% to 47.4% (+461%) on SimpleQA.

    Source: §4.4 · p. 9, §5 · p. 11

  • Takeaways & Limitations

    The authors conclude that trustworthy deployment calls for a multifaceted strategy combining robust RLHF, optimized prompting, and post-hoc calibration adjustments.

    Source: §5 · p. 11

    Results are conditional on using GPT-4o-mini as the fixed distractor generator and judge, so the authors identify alternative generators and independent judges as important future work.

    Source: §6 · p. 11

Abstract

from arXiv · show

Large Language Models (LLMs) show remarkable proficiency in natural language tasks, yet their frequent overconfidence-misalignment between predicted confidence and true correctness-poses significant risks in critical decision-making applications. We present a comprehensive analysis on calibration in LLMs across nine LLMs and three factual Question-Answering (QA) datasets, systematically comparing standard free-generation settings against structured distractor-augmented prompts. Our evaluation reveals that explicitly incorporating distractors can substantially mitigate miscalibration, achieving relative accuracy improvements up to 460% and ECE reductions up to 90%. Despite general trends, we uncover nuanced findings: large RLHF-tuned models display inherent calibration strengths but can paradoxically suffer increased miscalibration on easier queries, whereas smaller models benefit disproportionately from distractor prompts but remain significantly miscalibrated. Through detailed analyses across question types, we identify persistent calibration failures, particularly in person-based queries. We conclude with concrete recommendations-targeted fine-tuning, structured prompting, and strategic model choice-to ensure reliable, trustworthy LLM deployments.

1 Introduction

Confidence miscalibration can make LLM outputs risky in consequential decisions, while the effects of explicit distractors on calibration remain insufficiently studied. The paper compares free-generation and distractor-augmented QA across models, datasets, and question types.

1 Introduction

  • Confidence misaligned with correctness can erode trust and expose high-stakes users to costly or hazardous errors.The passage illustrates this with a confidently incorrect answer reported at 93% confidence.Source: §1 · p. 2
  • The study addresses whether plausible distractors improve LLM calibration, a gap in evidence for structured, distractor-rich QA settings.Classical post-hoc calibration methods were not originally designed for these conditions.Source: §1 · p. 2
  • The authors introduce a benchmark of nine LLMs across three factual QA datasets, comparing free-generation with distractor-augmented evaluation.The framework assesses accuracy and Expected Calibration Error, with additional analyses of question types, scale, tuning, and architecture.Source: §1 · p. 2

2 Related Work

Prior calibration work spans confidence elicitation, training-time strategies, pretraining and alignment, and post-hoc adjustment. The paper identifies structured distractor effects as an underexplored part of calibration research.

Intrinsic calibration

  • Prior work studies verbalized confidence and aggregated outputs as intrinsic calibration approaches, especially for black-box models.The cited approaches include prompting models to report confidence and combining multiple outputs.Source: §2 · p. 2

Fine-tuning strategies

  • Calibration-aware RLHF and Mixup-style augmentation aim to reduce ECE, but reported approaches remain sensitive to distribution shifts.The passage characterizes this sensitivity as a limitation of these fine-tuning strategies.Source: §2 · p. 2

Calibration during pre-training and alignment

  • Prior findings address calibration during pretraining and alignment, while post-hoc temperature scaling has often failed to align confidence with factual-QA accuracy.The passage notes that explicit distractor-based prompting had not been explored in this line of work.Source: §2 · p. 3

Post-hoc calibration

  • Post-hoc methods such as temperature scaling remain underexplored for contemporary LLMs, and cited studies report persistent overconfidence at larger scales.The passage describes a potential degradation in calibration with increasing model size.Source: §2 · p. 3

Contribution

  • The paper fills a gap in systematic evidence on structured distractors and examines calibration challenges across question types.Its analyses include person, date, and place queries in practical distractor settings such as retrieval-augmented generation and multiple-choice contexts.Source: §2 · p. 3

3 Experimental Setup

The experiments compare nine models on three factual QA datasets using free-generation and distractor-augmented prompts, elicited confidence, and common evaluation criteria. The setup measures answer accuracy and calibration, alongside instances helped or harmed by distractors.

3.1 Evaluation Datasets:

  • The evaluation uses 4,326 SimpleQA fact-seeking question-answer pairs, 2,922 FaVIQ R-set test examples labeled “supports,” and the first 1,000 TriviaQA validation examples from rc.web.no_content.The FaVIQ labels indicate that the provided answer is correct; TriviaQA’s no-content selection tests settings without retrieved supporting context.Source: §3.1 · p. 3

3.2 Evaluation Methods

  • The study compares free-generation with a choice-list setting that adds three plausible, factually incorrect distractors to each question.GPT-4o-mini generated distractors matched to answer type and comparable in specificity and context; confidence is elicited as a 0–100 self-report for cross-model comparison.Source: §3.2 · p. 4

3.3 Selected LLMs

  • Nine models span GPT-4, LLaMA, Gemma-2, and Qwen-qwq families, with differences in scale, architecture, and fine-tuning represented in the model taxonomy.The set includes dense and MoE models and varied SFT/RLHF regimes.Source: §3.3 · pp. 4–5, Table 1 · p. 5

3.4 Evaluation Criteria

  • GPT-4o-mini judges responses as CORRECT, INCORRECT, or NOT_ATTEMPTED, with correctness requiring the gold target’s key information and no contradiction.Smaller LLM judges sometimes misclassified responses or hesitated to assign NOT_ATTEMPTED.Source: §3.4 · pp. 5–6

3.5 Evaluation Metrics:

  • ECE measures confidence–accuracy misalignment using empirical bins of size 0.1; Dhelped and Dharmed count answers corrected or disrupted by distractors.Figure 2 plots actual accuracy against predicted confidence for the normal and distractor settings on SimpleQA.Source: §3.5 · p. 6, Figure 2 · p. 7

4 Experimental Results and Analysis

Across three benchmarks, baseline calibration varies with task difficulty and model family; distractors generally improve accuracy and often reduce ECE, but calibration effects depend on model size, tuning, and question type.

4.1 Quantifying Baseline Calibration of LLMs

  • GPT-4o reaches 35% accuracy with ECE 0.45 on SimpleQA, versus 90% accuracy with ECE ≈0.07 on TriviaQA; calibration loss remains similar across model sizes on hard SimpleQA.Small LLaMA-3 models have single-digit SimpleQA accuracy and ECE approaching 0.8; scaling them or using other open models yields only incremental gains.Source: §4.1 · pp. 7–8

4.2 Effects of Structured Distractors on Accuracy and Confidence

  • Across models, distractors boost accuracy and usually reduce ECE, but GPT-4o and GPT-4-turbo show ECE increases on TriviaQA despite 3–5% accuracy gains.On SimpleQA, distractors often halve ECE by up to ∆ECE ≈0.4 and more than double accuracy for smaller variants; Dna falls to zero on TriviaQA.Source: §4.2 · p. 8

4.3 Influence of Fine-Tuning Regime and Model Architecture

  • RLHF's calibration effect varies by model; on SimpleQA D, SFT-only Qwen-qwq-32B exceeds smaller LLaMA models in accuracy and ECE.LLaMA-3.1-8b outperforms the earlier 8B variant, while Gemma2-9b-it has elevated deferrals; larger LLaMA-4-Scout-17b trails GPT-4o-mini, but LLaMA-3-70b surpasses it.Source: §4.3 · pp. 8–9

4.4 Effect of Model Size within LLM Families

  • On SimpleQA, GPT-4o-mini gains 461% accuracy while GPT-4o compresses ECE by 92% to 0.037; large accuracy lifts do not ensure calibration in smaller models.GPT-4o-mini remains above 0.32 ECE, and LLaMA-3-8b remains at 0.36 despite an 819% accuracy gain; returns attenuate on FaVIQ and TriviaQA.Source: §4.4 · p. 9, Figure 3 · p. 9

4.5 Performance Across Question Types

  • Person queries have the largest relative ECE drop (69%), versus Place's 49%, though Person queries remain most challenging and Place performance is stronger.The analysis covers Date, Number, Person, and Place query types in N and D settings.Source: §4.5 · pp. 10–11, Figure 4 · p. 10, Table 3 · p. 11

5 Conclusion

The study finds widespread LLM overconfidence and improved calibration with structured distractors, especially for smaller models. Calibration nevertheless remains complex across model sizes and query categories.

5 Conclusion

  • Structured distractors significantly improve calibration, particularly for smaller models, against a backdrop of widespread overconfidence.The study also reports degraded calibration for large models on simpler queries.Source: §5 · p. 11
  • Large models can show degraded calibration on simpler queries, despite the general benefits of distractors.Source: §5 · p. 11
  • Systematic miscalibration across query categories shows that calibration challenges extend beyond simple accuracy.Source: §5 · p. 11
  • The conclusion calls for a multifaceted strategy combining robust RLHF, optimized prompting, and post-hoc calibration adjustments.It also presents the evaluation framework and guidelines as tools for future research.Source: §5 · p. 11

6 Limitations

The distractor-augmented evaluation fixes GPT-4o-mini as generator and judge for comparability, but results are conditional on that pair. The authors identify cross-model distractor sources and independent judges as future work.

6 Limitations

  • Results are conditional on using GPT-4o-mini as both the fixed distractor generator and judge, so generator/judge invariance is not claimed.Source: §6 · p. 11
  • The fixed generator–judge pair reduces rubric drift and style confounds but risks bias from an “AI-generates/AI-judges” loop.Source: §6 · p. 11
  • Type- and format-matched prompts, overlap and plausibility checks, and three reviewers’ human spot-checks mitigate subjectivity.Disagreements were resolved.Source: §6 · p. 11
  • Future work should explore cross-model distractor sources and independent judges, including human-only adjudication.Source: §6 · p. 11

A Custom Prompts

The appendix describes custom prompts for free-generation and distractor settings, eliciting both answers and confidence. The distractor workflow supplies plausible incorrect options matched to the expected answer type.

A Custom Prompts

  • The Normal-setting prompt elicits an answer and a confidence score from 0 to 100 in JSON format.Source: A Custom Prompts · p. 15
  • The Distractor-setting prompt gives the model a question and options, requires an answer from that list, and asks for a confidence score.The instructions say to lower confidence when uncertain and consider information reliability, recency, and domain knowledge.Source: A Custom Prompts · p. 16
  • For each question-answer pair, the workflow generates three wrong answers designed to be plausible, answer-type matched, distinct, and similarly specific.Source: A Custom Prompts · p. 17

B Same LLM judge as Prediction LLM

The authors compare performance when the prediction model judges its own answers and use GPT-4o-mini as a consistent judge across models. A small human evaluation reports substantial agreement with GPT-4o-mini, while reliability diagrams compare Normal and Distractor settings.

B Same LLM judge as Prediction LLM

  • The authors switched to GPT-4o-mini as judge across models after finding that self-judges sometimes misclassified smaller-model responses and missed NOT_ATTEMPTED labels.They note this discrepancy by comparing SimpleQA Nnone counts in Tables 2 and 4.Source: B Same LLM judge as Prediction LLM · p. 18
  • Table 4 covers SimpleQA accuracy, non-attempts, ECE, and helped or harmed instance counts in Normal and Distractor settings, using the prediction model as judge.Source: Table 4 · p. 19
  • Figure 5 plots actual accuracy against predicted confidence for Normal and Distractor settings, with bar labels showing correctly predicted instance counts.Its judge is the same as the prediction model.Source: Figure 5 · p. 19
  • On 100 responses independently classified by three annotators, GPT-4o-mini’s Cohen’s kappa was 0.82, indicating substantial agreement.Minor inconsistencies remained in ambiguous cases.Source: B Same LLM judge as Prediction LLM · p. 19

C Reliability Diagrams of Selected Models on FaVIQ and TriviaQA dataset

Figures 6 and 7 present reliability diagrams comparing calibration in N and D settings on TriviaQA and FaVIQ, respectively.

C Reliability Diagrams of Selected Models on FaVIQ and TriviaQA dataset

  • Figure 6 plots actual accuracy against predicted confidence for N and D settings on TriviaQA.The diagram presents calibration performance for the TriviaQA dataset.Source: Figure 6 · p. 20
  • Figure 7 plots actual accuracy against predicted confidence for N and D settings on FaVIQ.The diagram presents calibration performance for the FaVIQ dataset.Source: Figure 7 · p. 21
Loading 2502.11028v3…