Source-linked AI summary
Stable Miscalibration in Large Language Models: A Practical View of High-Confidence Errors
Akira Okutomi
TL;DR
High-confidence errors may be locally stable rather than fragile, raising questions about what confidence and perturbation responses reveal about their reliability. The paper combines a label-aware output audit with hidden-state sensitivity probes, finding internal stabilization without a clear fragility gap between confidently correct and overconfidently wrong items.
Problem
The paper asks whether high-confidence errors are locally stable miscalibrated mistakes rather than necessarily fragile failures, and what measurable confidence and perturbation behavior reveals about their reliability.
Method
The study combines a label-aware audit score of confidence variation and overconfident mistakes with hidden-state sensitivity probes under self-critical prompting.
Results
Across three open-weight models, self-critical prompting lowers layer-wise hidden-state sensitivity, but overconfidently wrong items show no clear sensitivity gap from confidently correct items.
Takeaways & Limitations
Prompt-induced local quieting is a stability signal rather than evidence of calibrated correctness, so some high-confidence errors may be stable and miscalibrated.
Takeaways & Limitations
The audit score is a labeled ranking signal rather than an absolute instability measure, and direct labeled baselines rank self-critique gains more strongly.
Abstract
from arXiv · showhide
High-confidence errors in large language models are often treated as evidence of fragile internal inference. We study a different possibility: stable miscalibration, where a confident wrong answer remains locally stable under small perturbations. We combine two diagnostics: a label-aware output-level audit score that ranks domains by confidence variation and overconfident mistakes under a forced-answer baseline, and an internal sensitivity probe that measures hidden-state movement. On a multi-domain binary factual audit set, this audit score tracks where abstention-aware self-critique reduces decision loss, although direct labeled baselines rank the same gain more strongly. Internally, self-critical prompting consistently reduces hidden-state sensitivity across layers in three open-weight models. This supports prompt-induced local stabilization rather than a purely output-level abstention pattern, but it does not imply calibration: audit-defined overconfident errors are not clearly more locally sensitive than confidently correct answers, so some high-confidence errors may be stable and miscalibrated rather than simply fragile.
1. Introduction
The introduction frames high-confidence errors as potentially stable miscalibration rather than only fragile failures. It proposes combining confidence-based decision audits with perturbation behavior to study practical uncertainty without requiring a complete mechanistic explanation.
- Motivation and hypothesis: High-confidence errors may remain locally stable under small perturbations, so correctness and internal stability need not coincide.A linear–Gaussian closed-loop abstraction motivates stable miscalibration as a useful uncertainty hypothesis.
- Motivation and hypothesis: The paper treats high-confidence errors as failures of epistemic awareness when the model behaves as though unreliable evidence were sufficient.This distinction matters because stable errors can affect decisions without being unusually fragile.
- Contributions: A label-aware audit score uses confidence variation and overconfident mistakes under a forced-answer baseline to rank domains where abstention-aware self-critique helps.Because it uses observed correctness, the score is a labeled audit diagnostic rather than an unlabeled uncertainty measure.
- Contributions: Across three open-weight models, self-critical prompting lowers layer-wise hidden-state sensitivity, indicating prompt-induced local stabilization internally.The result supports local stabilization but does not by itself establish calibration.
- Results and practical lesson: Audit-defined overconfidently wrong items show no clear sensitivity gap relative to audit-defined confidently correct items, so sensitivity does not separate them.The practical lesson is that confidence responses can inform decision-relevant uncertainty even when internal mechanisms remain partly understood.
2. Related Work
Related work motivates abstention-aware decision policies, highlights persistent overconfidence, and uses perturbation-based reliability probes. This paper combines these perspectives to rank intervention opportunities and test whether overconfident mistakes are locally fragile.
- Abstention-aware prediction: Abstention-aware prediction treats refusal as a decision option whose usefulness depends on calibration and task value.The paper's policy-aware loss therefore evaluates whether a policy answers in the right places, not only whether it is accurate.
- Calibration: Calibration research shows that language models can remain overconfident, while verbalized confidence may not reliably produce good decisions.This work instead asks whether observable confidence behavior can rank where intervention is likely to help, rather than proposing a new calibration benchmark.
- Perturbation and consistency probes: Perturbation and consistency probes study reliability through internal representations or output changes under small input variation.The paper combines both views through an output-level domain-ranking score and a layer-wise internal probe of overconfident mistakes.
3. A Practical View of Stable Miscalibration
Stable miscalibration describes locally damped behavior whose fixed point can still produce systematically wrong decisions. The practical audit therefore targets policy-dependent confidence movement and confident-error mass, using a labeled domain score to identify intervention-relevant overconfidence.
- Stable miscalibration: The linear–Gaussian model is only a simple abstraction: when ρ(A) < 1, perturbations damp locally, yet the fixed point can still disagree with truth.This defines stable miscalibration as compatibility between local stability and systematic miscalibration.
- Observable signals: The practical view tracks two observable signals: confidence movement under nearby decision policies and mass of confident mistakes under the reference policy.The domain score is designed to capture both signals.
- Domain audit score: Hproxy combines confidence movement with an overconfident-wrong indicator based on forced-answer baseline C0, cautious-abstention policy C1, and self-critical abstention policy C2.Reported P(correct) values pi,c map abstentions to 0.5, while OCW requires an answered, high-confidence, incorrect C0 response.
- Domain audit score: Because OCW uses correctness, Hproxy is a labeled audit diagnostic rather than an unlabeled deployment estimator or replacement for C0 error rate and C0 Brier risk.A larger score indicates policy movement, overconfident failure, or both.
- Interpretation: Domains with similar average accuracy can receive different scores, prioritizing those exposed to costly overconfidence and likely to benefit from intervention.The score asks which domain is most intervention-relevant, rather than which is hardest in the abstract.
4. Experimental Setup
The experiments use a frozen 532-item binary factual audit set spanning 11 domains, evaluated under paired forced-answer, abstention, and self-critical policies. A separate perturbation probe measures local hidden-state sensitivity in three open-weight models using audit-defined error groups.
- Audit set: The frozen audit set contains 532 short binary factual items across 11 domains, with 35–64 items per domain.The reproduction bundle includes item strings, domain labels, gold labels, and policy outputs.
- Audit set: The 11 domains include medical epidemiology, social stats, geo travel, cultural industry, dev region, entertainment event, literature media, macro index, history diplomacy, sports, and technical standard.Every item is evaluated under the same three policies, enabling paired domain-wise comparisons.
- Policies: C0 forces “Yes” or “No”, C1 permits abstention for under-specified items, and C2 adds a self-critical check before deciding.All policies use identical item wording; only the decision policy changes.
- Confidence and loss: Policy gains use paired domain-wise changes in Brier-style squared loss relative to C0, assigning abstentions 0.5 and a fixed squared-loss penalty of 0.25.The loss is computed on the probability assigned to the positive “Yes” label.
- Internal probe: The internal probe tests Llama-3.1-8B-Instruct, DeepSeek-R1-Distill-Llama-8B, and Qwen2.5-7B-Instruct with Gaussian embedding noise σ = 0.01 averaged over 40 trials per item.Confidently correct and overconfidently wrong groups come from the frozen gpt-4.1-mini C0 audit logs.
5. Results
The output-level audit score identifies domains where abstention-aware self-critique improves policy-aware decision loss, although labeled error diagnostics rank those gains more strongly. Internal probes show prompt-induced hidden-state stabilization across three models, without clear evidence that audit-defined overconfident errors are uniquely fragile or calibrated.
- Output-level results: Hproxy’s domain ranking correlates positively with C2 gain over C0, with Spearman ρ = 0.71.Higher proxy values tend to coincide with larger reductions in policy-aware loss.
- Output-level results: Predictive entropy and confidence variation do not rank C2 gains, whereas label-aware C0 error and Brier risk outperform Hproxy as pure rankers.Hproxy is interpreted as a structured diagnostic of intervention value rather than a performance-optimized predictor.
- Output-level results: C1 reduces the overconfident-wrong rate from 0.211 to 0.028 at 0.427 coverage, while C2 reaches 0.064 at 0.571 coverage.C1 is more conservative; C2 preserves more answers while still reducing overconfident errors.
- Output-level results: C2’s strongest paired improvements occur in medical epidemiology (−0.101), social stats (−0.099), and geo travel (−0.064), while three domains worsen.Entertainment event, literature media, and technical standard worsen by +0.039, +0.049, and +0.064, respectively.
- Internal probe: Self-critical prompting lowers layer-wise hidden-state sensitivity across all three representative models, supporting internal stabilization rather than only output-level abstention.Lower sensitivity indicates reduced response to injected local perturbations, not calibrated correctness.
- Internal probe: OCW–CC sensitivity gaps remain small, with bootstrap intervals including zero and standardized gaps of about 0.13 pooled standard deviations or less.Gaussian sweeps and semantic rewrites show the gap does not steadily grow, so the items are not clearly uniquely fragile.
6. Discussion and Limitations
The discussion emphasizes that the findings are narrow, benchmark-dependent, and insufficient to treat local stability as calibrated correctness. It frames stable miscalibration as a trust risk and recommends periodic labeled audits to guide targeted interventions.
- Limitations: The study uses a frozen 532-item binary factual audit set, so its rankings may not transfer to larger datasets, multi-class tasks, or open-ended generation.The audit set is author-curated rather than publicly benchmarked with independent multi-annotator validation, so label errors or item artifacts may affect rankings.
- Limitations: Hproxy is a labeled ranking signal rather than an absolute instability measure or label-free deployment estimator, and direct labeled baselines rank C2 gain more strongly.This limitation constrains how the audit score should be interpreted beyond the study’s labeled evaluation setting.
- Interpretation: The absence of a clear CC–OCW gap does not show that hidden-state instability never matters; it shows no evidence for local fragility under the present probe.Prompt-induced local quieting should therefore be treated as a stability signal, not evidence of calibrated correctness.
- Practical implications: Stable miscalibration creates trust risk because locally stable answers can still be confidently wrong in high-impact domains.The audit is recommended as a periodic labeled evaluation workflow that ranks domains by overconfident-error mass and policy movement to target abstention, retrieval, or human review.
Impact Statement
The work targets improved reliability under uncertainty through high-confidence-error and abstention-aware auditing, while cautioning that its diagnostics support review rather than guarantee correctness. Reproducibility materials and methodological disclosures are provided.
- Intended impact: The diagnostics are intended for labeled auditing, risk reporting, and human review—not as guarantees of correctness or ways to conceal uncertainty.The stated impact is improving reliability under uncertainty.
- Reproducibility: The reproducibility artifact is archived at OSF, with README guidance and a main reproduction command.The artifact specifies “python scripts/reproduce eiml.py” as the main command.
- Reproducibility: The default reproduction path uses frozen CSV inputs and does not call external APIs.This makes the default path self-contained with respect to external API access.
- Disclosure: LLMs assisted with language editing and synthetic item drafting, while the author reviewed all claims, labels, code, and results.The disclosure distinguishes assistance from author review.
- Disclosure: The views are solely the author’s, and no employer, client, proprietary, or personal data was used.The statement disclaims representation of affiliated organizations.
Appendix
Appendix robustness checks find no stable positive OCW sensitivity gap in the four-domain DeepSeek subset, and the audit-defined gap remains small across denser Gaussian sweeps and semantic rewrites. The appendix also documents the audit materials and prompt conditions used for the probe.
- Robustness checks: The four-domain DeepSeek subset shows no stable positive OCW sensitivity gap: the signed final-layer gap ranges from −0.185 to +0.009 across eight scales.σ ranges from 0.0025 to 0.05.
- Audit materials: The OSF repository contains the audit data, prompts, examples, and policy outputs, while C0, C1, and C2 denote forced-answer, cautious-abstention, and self-critical conditions.The internal probe compares standard and self-critical prompts.
- Robustness checks: The audit-defined OCW–CC gap remains small and does not steadily grow under denser Gaussian sweeps or semantic rewrites in DeepSeek-R1 and Qwen2.5-7B.These checks are shown for both models across both perturbation families.