Source-linked AI summary
Different Facets of Verbalised Overconfidence: an Interpretability Study
Davide Mazzaccara, Leonardo Bertolazzi, Raffaella Bernardi
TL;DR
LLMs often express certainty when evidence warrants hedging or abstention, but prior work has treated these uncertainty expressions separately. This study compares verbal markers, abstention, and numeric scores in controlled reasoning scenarios and finds that Qwen3-4B defaults to certainty, with numeric confidence especially prone to overconfidence.
Problem
Prior work provides limited integrated evidence on overconfidence across verbal markers, abstention, and numeric confidence scores.
Method
The study evaluates Qwen3-4B on controlled reasoning stories that test whether expressed uncertainty matches logically certain or uncertain scenarios across three formats.
Results
Qwen3-4B expresses certainty in 14%, 27%, and 80% of uncertainty cases for verbal, abstention, and numeric settings, respectively.
Takeaways & Limitations
Certainty is the model’s broad default mechanism, while uncertainty is a sparse override whose features generalize across expression formats and can reduce some overconfident errors.
Takeaways & Limitations
The analysis is restricted to Qwen3-4B in non-thinking mode, so its interaction with larger models and chain-of-thought reasoning remains untested.
Abstract
from arXiv · showhide
Large language models tend to overconfidence, giving assertive answers when the evidence suggests hedging or abstention. Using controlled reasoning scenarios that manipulate logical necessity and possibility, we study this behavior in Qwen3-4B, across three ways to express uncertainty: verbal epistemic markers, abstention, and numeric confidence scores. Our results confirm this tendency toward overconfidence, particularly when the model is prompted to output a numeric confidence score. At the interpretability level, we propose a method that differentially identifies transcoder features responsible for uncertainty and certainty. Our analysis reveals Qwen3-4B's default mechanism favors certainty generation through a broad coalition of shared features, while uncertainty is implemented as a sparse override mediated by a small set of dedicated features. Intervening on these uncertainty features both causally proves this imbalance underlying overconfidence and also mitigate overconfident errors. The same set of features generalise across the three uncertainty-expression settings, languages, and an out-of-distribution modality task.
1 Introduction
This study investigates verbal overconfidence in Qwen3-4B by comparing verbal epistemic markers, abstention, and numeric confidence scores. It uses controlled stories that distinguish logically derivable certainty from unresolved uncertainty to examine how the model expresses confidence and uncertainty.
- Motivation: The study targets models’ difficulty accurately expressing uncertainty despite strong performance in generating appropriate outputs.This challenge matters because large language models increasingly operate in practical and critical domains, including medical and intelligence settings.
- Study design: It compares three expressions of verbalised uncertainty: verbal epistemic markers, abstention, and numeric confidence scores.The comparison is intended to avoid treating abstention as separate from verbalised confidence.
- Study design: Controlled stories provide evidence and a question, with certainty appropriate when the answer is logically derivable and uncertainty otherwise.The verbal setting illustrates certainty with assertive “is” and uncertainty with the epistemic marker “might”; abstention is another uncertainty expression.
- Research questions: Using Qwen3-4B on controlled stories, the study asks whether overconfidence appears across verbal confidence, abstention, and numeric confidence scores.This is the first stated research question and defines the behavioral scope of the investigation.
- Research questions: The study also asks what mechanism the model uses to express certainty versus uncertainty.This is the second stated research question and motivates analysis beyond behavioral outputs.
2 Experiment 1
Experiment 1 tests Qwen3-4B’s uncertainty behavior using controlled stories across verbal markers, abstention, and numeric confidence settings. The model overexpresses certainty in uncertainty scenarios, especially when prompted for numeric confidence.
- Experimental design: The experiment uses 450 premise–question pairs: 300 uncertainty scenarios and 150 certainty scenarios, reframed as open-ended question answering.The scenarios come from Li et al. (2025) and span three uncertainty-expression settings.
- Results: In uncertainty scenarios, Qwen3-4B incorrectly expresses certainty in 14% of verbal, 27% of abstention, and 80% of numeric cases.In certainty scenarios, the model consistently expresses certainty.
- Results: The numeric setting produces the strongest overconfidence, with the model tending to state “5” (certain) across scenarios.Most mean probability mass falls on the “5” token even in uncertainty scenarios, while it is near zero in certainty scenarios.
3 Experiment 2
Experiment 2 identifies certainty- and uncertainty-related transcoder features through complementary selectivity and causal analyses, revealing a broad default certainty mechanism versus a sparse uncertainty override. Interventions causally validate this asymmetry and generalize uncertainty control across verbal, abstention, numeric, and modality settings.
- Feature identification: Features were identified using contrastive activation selectivity and causal contribution analyses, retaining both condition-selective features and features that directly move certainty–uncertainty logits.Selectivity used Cohen’s d with Mann–Whitney U and Benjamini–Hochberg FDR; causal scores measured attribution toward certainty versus uncertainty token logits.
- Circuit asymmetry: Certainty is generated by a broad coalition of shared middle-layer features, whereas uncertainty relies on a small dedicated override that supplies most of its uncertainty-token logit push.Certainty-specific features contribute almost nothing directly to the certainty token, while uncertainty-specific features provide the bulk of the uncertainty-token push.
- Causal validation: Suppressing the top-20 uncertainty features reduced correct uncertainty expression from 99% to 0%, while suppressing the top-20 certainty features left certainty at 100%.Random-feature baselines were flat in both conditions.
- Cross-setting generalization: 30 of 82 committed-answer errors flipped to “I don’t know” under a ×3 boost, raising abstention accuracy from 72.7% to 82.7%.The same uncertainty features generalized from might/could expressions to abstention, while numeric confidence changed monotonically but negligibly.
- Cross-modal generalization: In a modality task, uncertainty-feature boosts shifted probable-item responses only from Somewhat Certain to Somewhat Uncertain, while improbable-item responses showed a stronger asymmetric shift.The probable-item shift left Very Certain intact and saturated immediately across ×1–×5; interventions also produced lexical reframings of answers.
4 Conclusions
The study compares three forms of verbalised uncertainty and finds overconfidence across all settings, with numeric confidence expressions most prone to it. It also introduces a method for identifying transcoder features associated with uncertainty and certainty expressions.
- The study connects verbal epistemic markers, abstention, and numeric confidence scores as three forms of verbalised uncertainty.This comparison provides a unified framework for examining how models express uncertainty.
- Overconfidence occurs across all three uncertainty-expression settings, with numeric confidence expression the most prone to overconfidence.The behavioral experiment directly compares the model’s tendency toward overconfidence across these settings.
- The study proposes a method to identify transcoder features associated with uncertainty and certainty verbal expressions.This interpretability method supports analysis of the mechanisms underlying the model’s verbalised uncertainty.
Limitations
The analysis is limited to Qwen3-4B in non-thinking mode, leaving its interaction with model scale and chain-of-thought reasoning for future work.
- Model scope: The study examines only Qwen3-4B in its non-thinking mode.The authors note that larger models and chain-of-thought reasoning have been shown to mitigate overconfidence, but their interaction with these findings remains unstudied.
A Experiment 1 … B Results on the Verbal Setting
Experiment 1 evaluates certainty and uncertainty responses in Qwen3-4B using controlled prompts across verbal, abstention, and numeric-confidence settings. The supplied passages specify the verbal prompt designs and implementation, but provide no results passage for Section B.
- A Experiment 1: Qwen3-4B is evaluated with greedy decoding in non-thinking mode through the Huggingface Transformer library.The implementation uses empty <think></think> blocks.
- A Experiment 1: Across settings, a marker ** enforces the expected output format, while certainty and uncertainty scenarios differ by one premise.The three settings vary only in system instruction and assistant prefill; stories and questions remain shared within each setting.
- A.1 Verbal Prompt: In the verbal setting, the prefill “The stamp **” elicits certainty markers such as “is” or “must” and uncertainty markers such as “could,” “might,” or “may.”The prompt asks whether the stamp is in the yellow case under either determined or undetermined premises.
- A.1 Verbal Prompt: The certainty verbal scenario states that the stamp is neither in the white nor black case, making the yellow-case question determined.The assistant prefill is “The stamp **”.
- A.1 Verbal Prompt: The uncertainty verbal scenario states only that the stamp is not in the black case, leaving the yellow-case question unresolved.It uses the same story and question with one changed premise.
- A.2 Abstention Prompt: The abstention prompt instructs the model to answer “I don’t know” when the answer cannot be determined.The supplied passages show determined and undetermined stamp-location variants with a bare assistant prefill marker.
- A.3 Numeric Confidence Score Prompt: The numeric-confidence prompt requests a score from 1 (uncertain) to 5 (certain) for the yes/no question.Both premise variants use the assistant prefill “Confidence: **”.
B.1 Behavioral Overconfidence
Qwen3-4B performs perfectly on necessity prompts but substantially worse on possibility prompts, where some incorrect responses are assertive epistemic errors. Its certainty expression relies on assertive sentences rather than epistemic verbs, consistent with overconfidence affecting the necessity–possibility performance gap.
- Overall performance: Qwen3-4B answers all 150 necessity prompts correctly (100%) but only 206 of 300 possibility prompts (68.7%).This establishes a pronounced performance gap between logically necessary and merely possible scenarios.
- Possibility errors: Among 94 non-correct possibility responses, 44 are enumerations, 17 repeat constraints, and 33 are assertive epistemic errors.Enumerations and repetitions are excluded from accuracy computation; errors emit an assertive is/did when the answer should be epistemic.
- Template effects: Accuracy drops are concentrated in the novelist (71%) and traveling (58%) templates.These template-specific drops are shown for the verbal setting.
- Certainty expression: In correct certainty scenarios, the model uses assertive sentences and never epistemic verbs such as must or should.The authors interpret this certainty-expression pattern as evidence that apparent necessity advantage may reflect overconfidence.
B.2 Errors as under-fired possibility circuits · B.3 Ablation Intervention
Errors under-fire the possibility-specific circuit and instead rely more heavily on necessity-specific features, unlike the possibility prototype and enumeration cases. This imbalance is reflected both in attribution mass and in the fraction of possibility-specific features that activate.
- B.2 Errors as under-fired possibility circuits: Errors allocate 16.9% of attribution mass to possibility-specific features versus 30.2% in the possibility prototype, while necessity-specific mass rises to 18.5% versus 5.3%.This distinguishes errors from the other possibility-only pools.
- B.2 Errors as under-fired possibility circuits: Only 1,002 of 2,302 possibility-specific features are active in the error pool, corresponding to 43.5%.The passage identifies this as evidence that the possibility circuit is under-used in errors.
- B.2 Errors as under-fired possibility circuits: Enumeration resembles a properly fired possibility pass, unlike errors that under-use the possibility-specific coalition and over-use necessity-specific features.The comparison covers the qualitative pattern reported for the three possibility pools.
B.4 Feature Visualisation
Feature visualisation identifies a small, consistent Constraints-zone coalition feeding the certainty prototype, with the L27 “is” feature showing especially strong certainty-specific behavior. This feature activates far more often in certainty than uncertainty samples and has the largest direct assertive-logit weight among Constraints-zone features.
- B.4 Feature Visualisation: The certainty prototype receives a small but consistent coalition of lexical-syntactic Constraints-zone features labelled “not” and “is”.
- B.4 Feature Visualisation: L27/F67512 (“is”) appears in at least 50% of certainty samples, with d = +0.33 and padj = 6e−15.
- B.4 Feature Visualisation: The L27 “is” feature has the largest direct assertive-logit weight among Constraints-zone features: +0.031 in certainty versus +0.021 in uncertainty.It activates with selection frequency 0.98 in certainty and 0.40 in uncertainty.
C Multilingual Experiments
The multilingual experiments translated controlled stories into Chinese and Italian to test whether Qwen3-4B’s overconfidence and uncertainty features generalize across languages. Qwen3-4B exceeded 90% accuracy in certainty scenarios but remained overconfident in uncertainty scenarios, with English-derived uncertainty-feature interventions evaluated cross-lingually.
- Experimental setup: Controlled stories were translated into Chinese and Italian, with Qwen3-4B response labels assigned by Claude Haiku as in the English experiment.The multilingual verbal setting tested cross-language overconfidence and feature generalization.
- Results: Accuracy exceeded 90% in certainty scenarios for both Chinese and Italian, while uncertainty-scenario accuracy was 44% and 64%, respectively.English performed best, followed by Chinese and then Italian, possibly reflecting the multilingual composition of Qwen3-4B’s pretraining data.
- Feature transfer: The experiment applied a ×3 boost to the top-20 uncertainty features identified in English to test cross-lingual transfer.Figure 7 reports this intervention on errors only.
D Implementation Details & AI Assistance
The experiments used two Nvidia GeForce RTX 3090 24GB GPUs and took approximately 72 hours across six settings, with Claude Code and ChatGPT assisting drafting and editing.
- Implementation Details & AI Assistance: Experiments ran on two Nvidia GeForce RTX 3090 24GB GPUs for approximately 72 hours total across six settings.The reported runtime was 12 hours × 6 settings.
- Implementation Details & AI Assistance: Claude Code drafted parts of the experiments, while ChatGPT provided grammatical corrections and rephrasing of short passages.