PaperlayerView exact revision on arXiv

Source-linked AI summary

How do LLMs Compute Verbal Confidence

Dharshan Kumaran, Arthur Conmy, Federico Barbero, Simon Osindero, Viorica Patraucean, Petar Veličković

arXiv:2603.17839v3cs.CLcs.AIcs.LG

TL;DR

The paper investigates when LLMs compute verbal confidence and whether it reflects token-level fluency or a richer evaluation of answer quality. Using causal interventions and representation analyses across several models and tasks, it finds evidence that confidence is computed during answer generation, cached, and later retrieved, with cached representations explaining variance beyond token log-probabilities. The authors conclude that verbal confidence reflects automatic self-evaluation, while noting that cached retrieval may be one of several overlapping mechanisms.

  • Problem

    Little is understood about how verbal confidence is internally generated, including whether it is computed on demand and whether it reflects token log-probabilities or richer answer-quality evaluation.

  • Method

    The paper uses activation steering, patching, noising, swapping, attention blocking, linear probing, and variance partitioning to examine confidence representations and their information flow.

  • Results

    Cached confidence representations explain substantial variance in verbal confidence beyond token log-probabilities, supporting cached retrieval and a richer evaluation of question-answer fit.

  • Takeaways & Limitations

    Verbal confidence reflects automatic self-evaluation rather than post-hoc reconstruction, with implications for understanding LLM metacognition and improving calibration.

  • Takeaways & Limitations

    Cached retrieval is dominant rather than exclusive in the studied settings, and confidence may also arise from distributed or overlapping circuits.

Abstract

from arXiv · show

Verbal confidence -- prompting LLMs to state their confidence as a number or category -- is widely used to extract uncertainty estimates from black-box models. However, how LLMs internally generate such scores remains unknown. We address two questions: first, when confidence is computed -- just-in-time when requested, or automatically during answer generation and cached for later retrieval; and second, what verbal confidence represents -- token log-probabilities, or a richer evaluation of answer quality? Focusing on Gemma 3 27B (across TriviaQA, BigMath, and MMLU), Qwen 2.5 7B, and the reasoning model Magistral Small 24B, we provide convergent evidence for cached retrieval. Activation steering, patching, noising, and swap experiments reveal that confidence representations emerge at answer-adjacent positions before appearing at the verbalization site. Attention blocking pinpoints the information flow: confidence is gathered from answer tokens, cached at the first post-answer position, then retrieved for output. Critically, linear probing and variance partitioning reveal that these cached representations explain substantial variance in verbal confidence beyond token log-probabilities, suggesting a richer answer-quality evaluation rather than a simple fluency readout. These findings demonstrate that verbal confidence reflects automatic, sophisticated self-evaluation -- not post-hoc reconstruction -- with implications for understanding metacognition in LLMs and improving calibration.

1. Introduction

The paper asks how LLMs generate verbal confidence and whether it reflects meaningful answer evaluation or merely generation fluency. It frames these questions around the timing of confidence computation and the distinction between token log-probabilities and richer question-answer assessment.

  • Motivation: Verbal confidence is increasingly used to estimate uncertainty, but little is known about how LLMs internally generate these scores.Understanding this mechanism bears on whether confidence reflects answer-quality evaluation or surface-level generation fluency.
  • Research questions: The first question is whether confidence is computed only when requested or automatically during answer generation and later retrieved.The paper contrasts just-in-time computation at the verbalization site with cached computation at answer-adjacent positions.
  • Research questions: The second question is whether verbal confidence reads out token log-probabilities or represents a richer evaluation of question-answer fit.This distinction separates a fluency-based account from an evaluation that uses partially independent signals.
  • Theoretical framing: The paper connects this distinction to first-order and second-order theories of confidence from decision neuroscience.First-order accounts treat confidence as a direct readout of the decision signal, whereas second-order accounts posit a distinct evaluation of the decision.
  • Theoretical framing: A second-order account could support error detection after an answer is produced, whereas a pure first-order account links confidence and accuracy to the same underlying signal.The paper presents this as a key empirical implication of the competing theories.

2. Experiments

Across convergent interventions, confidence representations emerge at PANL before being transferred to CC for verbalization, while decoding and causal tests indicate they encode more than answer-token likelihoods.

  • Activation patching: Activation patching partially restores corrupted confidence from PANL and nearly completely restores it from CC, with PANL recovery preceding CC recovery.PANL recovery peaks at layer 25, whereas CC recovery rises sharply after layer 30; partial PANL recovery is consistent with distributed, overlapping mechanisms.
  • Activation interventions: The causal interventions support PANL and CC as necessary components of faithful verbal confidence reporting, although single-position interventions produce only partial disruption.Mean ablation disrupts confidence at PANL and CC but not at PANL+1, with partial effects possibly reflecting distributed encoding.
  • Activation interventions: Steering, patching, noising, and swaps identify PANL and CC as causally involved in verbal confidence, with PANL effects consistently appearing earlier.PANL effects emerge earlier across interventions, consistent with confidence being encoded there before transfer to CC.
  • Activation swap experiment: Cross-confidence PANL swaps shift reported confidence toward the donor trial, beyond same-confidence controls, ruling out a purely content-correlated PANL representation.High→Low swaps decrease confidence and Low→High swaps increase it; effects peak at layer 26 across three metrics.
  • Decoding confidence information: PANL and CC activations explain substantial unique variance in correctness and verbal confidence beyond log-probabilities, supporting a distinct question-answer-fit evaluation.Length-normalized mean answer log-probability remains a strong correctness indicator, but interventions at the answer-colon position have null effects.

3. Related Work

Prior work links cached internal representations to later retrieval and uses classifiers to decode latent properties from LLM hidden states.

  • LLMs can automatically cache attributes at the last subject token, then retrieve them for output rather than recomputing them at the prediction site.This factual-recall pattern resembles the paper’s cached-retrieval hypothesis for confidence.

4. Conclusion

The paper finds that verbal confidence in both reasoning and non-reasoning LLMs is cached during answer generation and later retrieved for verbalization. These cached representations contain information beyond token log-probabilities, consistent with richer answer-quality evaluation.

  • Verbal confidence reflects cached retrieval rather than just-in-time computation in reasoning and non-reasoning LLMs.Confidence representations emerge during answer generation before the model knows a rating will be requested.
  • Cached confidence representations explain substantial variance beyond token log-probabilities, suggesting evaluation of question-answer fit rather than fluency alone.They also emerge at a position distinct from where the answer itself is generated, consistent with a second-order account.
  • Automatic confidence computation parallels factual-recall findings in which LLMs enrich subject representations before extracting a queried attribute.Related work also suggests that immediate post-answer cached representations may contribute to error detection.

Impact Statement

The paper situates verbal confidence within broader work on uncertainty, latent representations, mechanistic interventions, and confidence theories. Its supported impact is improved mechanistic understanding of how confidence signals may relate to reliability and error detection.

  • The work may support more reliable uncertainty estimation by identifying where confidence signals form and where they may become misaligned with answer correctness.This matters for high-stakes deployments in which misleading confidence can distort user trust.
  • Verbal confidence is motivated as a practical uncertainty measure because most deployed black-box models do not expose token-level probabilities.Token likelihoods and sampling-based consistency are alternative approaches, but their applicability differs by access requirements.
  • Prior work shows that LLM hidden states encode output quality, truthfulness, and uncertainty in ways that can diverge from surface-level confidence.These findings provide context for studying internal confidence representations.
  • Activation steering, patching, noising, and swapping provide intervention frameworks for probing or testing causal roles of internal representations.Attention blocking supplies a related way to distinguish cached storage from direct output access.
  • Second-order confidence accounts posit signals distinct from decision-driving signals, enabling evaluation beyond the information directly used to produce a decision.Such partially independent signals can support error detection after a response is produced.

B. Supplemental Figures

The supplemental figures document the prompts, calibration, intervention designs, attention analyses, and decoding experiments used to study verbal confidence across models.

  • Prompt designs: The categorical prompt isolates PANL, PANL+1, confidence-colon, first-confidence-colon, and last-answer-token positions for mechanistic analysis.Each confidence class begins with a unique first token, enabling analysis of first-token identity and logits.
  • Intervention experiments: Activation steering, noising, patching, swapping, and attention blocking test how confidence representations are formed, transferred, and verbalized.The intervention figures report confidence changes, logit-difference changes, and first-token change rates across selected positions.
  • Variance analysis: PANL activations explain substantial unique variance in verbal confidence beyond answer-token log-probability summaries, reaching R²unique = 0.38 at peak layer ∼40.Variance partitioning controls for individual baselines and all six baselines combined.
  • Additional model analyses: Additional figures show numeric-confidence prompting and reasoning-model experiments involving Magistral Small 24B’s reasoning trace, answer, confidence report, patching, noising, and decoding.Magistral decoding probes residual-stream activations at PANL, PANL+1, CC, the final answer token, and intermediate reasoning-trace positions.
  • Calibration: The supplemental calibration figures report Gemma categorical confidence with ECE = 0.12 and AUROC = 0.71, and numeric confidence with ECE = 0.16 and AUROC = 0.73.The figures also show the distributions of responses across ten confidence bins or classes.
  • Calibration: Qwen’s categorical confidence ratings have ECE = 0.06 and AUROC = 0.65, with the response distribution shown alongside calibration.No temperature scaling was applied because the analysis targeted raw verbal confidence signals.

C.1. Experiments with Categorical Confidence Prompt in Gemma 3 27B

The Gemma experiments combine activation interventions, probing, and log-probability baselines to locate and characterize confidence representations. They test answer-token dependence and whether activations explain verbal confidence beyond token likelihoods.

  • Technical details: Categorical confidence was extracted from the first generated token in single forward passes, with argmax outputs verified against generation and valid confidence classes.
  • Technical details: Answer-generation trials were produced in a preliminary phase using a confidence-instruction prompt, with duplicate TriviaQA questions removed.
  • Activation steering: Activation steering compared high- and low-confidence residual-stream activity at answer-adjacent and confidence-reporting positions across layers.High- and low-confidence vectors were constructed from contrasting trial groups and scaled relative to residual norms.
  • Activation steering: Steering used residual-stream vectors defined as high-confidence mean activity minus low-confidence mean activity, with the low-confidence vector as its inverse.The vectors were scaled to 3% of the residual norm and multiplied by empirically selected constants.
  • Activation patching: Answer-token corruption replaced answer embeddings with calibration-set mean activations before selectively restoring clean activations at chosen positions and layers.The calibration set contained 100 trials, split evenly between high- and low-confidence cases.
  • Activation patching: Patching evaluated PANL, the confidence colon, and PANL+1, while restricting the main analysis to high-confidence trials because corruption should be most detectable there.The intervention restored one position-layer activation while other positions remained corrupted.
  • Decoding confidence information: Linear probes used logistic regression for correctness AUROC and Ridge regression for verbal-confidence R2 across layers and token positions.Activations were collected from 3,000 TriviaQA trials and z-scored before fitting.
  • Variance partitioning using log-probability baselines: Variance partitioning compared activation-only, log-probability-only, and combined Ridge models to quantify activation information beyond answer log-probability baselines.The combined-baseline analysis was the most conservative test because it credited baselines with redundant information across six summaries.

C.3. Experiment using Qwen 2.5 7B

Qwen 2.5 7B was evaluated with the same categorical-confidence procedures on TriviaQA, using a dense layer sweep and 3,000 activation-collection trials.

  • Qwen 2.5 7B showed reasonable calibration, with ECE = 0.06 and AUROC = 0.65 on the categorical prompt.The model was evaluated greedily across all 28 layers using the same TriviaQA subset and procedures as Gemma.

C.4. Experiments using Magistral Small 24B Reasoning

The Magistral experiments adapt the intervention framework to chain-of-thought reasoning, where confidence-relevant information spans the full reasoning trace and final answer. They collect activations across trace positions and use matched corruption, noising, and swapping procedures.

  • Model and setup: Magistral Small 2506 is a 24B-parameter, 40-layer decoder-only transformer evaluated with attention weights exposed for mechanistic experiments.
  • Activation collection and decoding: The Magistral study used 5,000 TriviaQA questions for behavior and a stratified 3,000-trial subset for activation analyses.The behavioral set contained 4,998 valid trials after filtering.
  • Common adaptations: Patching, noising, and swapping followed the Gemma framework but used chain-of-thought prompts with PANL defined as the response-block terminating newline.The response block included the reasoning trace and extracted answer before PANL.
  • Activation patching: Because confidence-relevant content spans the reasoning trace in chain-of-thought settings, patching corrupted embeddings across the question and entire response block.PANL and downstream class tokens were excluded from the corruption scope.
  • Activation noising: Noising used balanced high- and low-confidence calibration trials at PANL and PANL+1, with CC analyzed separately as a control.The test set contained 400 trials selected from the confidence extremes.
  • Activation swap: Activation swaps used a 2×2 recipient-by-donor confidence design at PANL and PANL+1, matching donors and recipients on question and reasoning-trace length.Each confidence pool contained 400 source trials, and matching used ten quantile bins on both length axes.
  • Activation collection and decoding: Activations were collected at ten evenly spaced reasoning-trace positions plus PANL, PANL+1, the last answer token, and a question-token control.The trace positions ranged from the reasoning-trace start to the final answer token.

D.1. Attention Blocking with Categorical Confidence Prompt

Categorical-prompt attention blocking supports cached retrieval: answer tokens supply confidence information, PANL caches it, and later tokens retrieve it for verbalization. The experiments also show that redundant routing can obscure a direct PANL-to-CC pathway.

  • Blocking CC’s attention to question tokens, answer tokens, or both produced only approximately 10% change, ruling out confidence computation from scratch at CC.
  • Blocking CC’s direct attention to PANL had no effect under the categorical prompt, consistent with redundant routing through intermediate template tokens.
  • Blocking attention from later tokens to PANL produced a 20% change rate and reduced logit differences, confirming PANL as a confidence cache.This held even when direct CC→PANL blocking was ineffective.
  • Answer tokens are the primary confidence-information source: blocking the last answer token changed outputs 50%, while preserving PANL→last A reduced this to 20%.
  • The results support a pathway in which confidence originates at answer tokens, is read and cached by PANL, then flows to CC for verbalization.

D.2. Generalization to a Reasoning Model: Magistral Small 24B

Experiments on Magistral Small 24B extend the cached-retrieval account to reasoning traces. Confidence information remains available at the post-answer-newline position, although reasoning introduces distributed and redundant encoding.

  • Patching PANL restored confidence after corrupting the question and full response, whereas patching PANL+1 had no effect.This replicates Gemma’s central patching result in a reasoning model.
  • Noising PANL alone had no effect beyond PANL+1, while noising CC substantially disrupted verbal confidence.The contrast indicates that reasoning-trace representations distribute confidence information redundantly.
  • Cross-confidence swaps at PANL produced systematic directional shifts, with no effect at PANL+1.The L→H direction dominated, plausibly because high-confidence classes concentrated probability mass near a ceiling.
  • Linear probes decoded confidence from PANL and later CC layers, with decoding increasing across the reasoning trace and peaking at the final trace token.The control position QTT explained negligible variance.
  • Across patching, swapping, and decoding, Magistral supports cached retrieval after extended reasoning: confidence is encoded at the post-answer-newline position following the final answer.

D.3. Gemma 3: Variance partitioning using log-probability baseline

Variance partitioning shows that cached PANL representations encode verbal confidence beyond multiple token log-probability summaries. Intervention controls further indicate that this result is not explained by generic out-of-distribution activation shifts.

  • The six log-probability baselines jointly explained R2_CV = 0.100 of verbal-confidence variance, despite individual values ranging from 0.025 to 0.101.
  • PANL activations explained R2_unique = 0.38 at peak layer 40 beyond the combined six-baseline set.This exceeded the variance explained by any individual log-probability summary by more than three times.
  • The PANL representation was therefore not reducible to token-level probability signals measured by means, extremes, variability, or token position.
  • Across steering, patching, and noising, cosine similarity to clean activations exceeded 0.99 and norm ratios stayed within 0.91–1.10.These shifts remained within the observed natural activation distribution.
  • At Layer 25, patching PANL yielded 24.3% confidence recovery versus −1.4% at PANL+1 despite identical pre-patch cosine similarity of 0.999.The dissociation separates position-specific causal effects from generic activation drift.

D.5. Limitations and Future Work

The paper bounds its claims to fixed prompt wording and leaves some retrieval-routing details unresolved. Future work must test framing sensitivity and identify the heads and intermediate positions mediating longer-prompt retrieval.

  • The cached-retrieval mechanism was tested across categorical and numeric formats, but not under small paraphrases or persona-style framing.
  • Whether framing changes confidence upstream at PANL or biases verbalization at CC remains open.
  • Attention blocking rules out just-in-time computation at CC but does not identify the precise retrieval route in prompts with many intermediate template tokens.
  • Future work should identify attention heads mediating PANL→CC retrieval and characterize routing through intermediate positions in longer prompts.
Loading 2603.17839v3…