PaperlayerView exact revision on arXiv

Source-linked AI summary

When Words Fall Short: Iterative Synergy Between Verbalized Reasoning and Hidden Features for LLM Confidence Estimation

Yekun Xu, Ante Wang, Jingyi Ren, Xuanyi Chen, Weizhi Ma, Yang Liu

arXiv:2609.34454v1cs.CL

TL;DR

Reliable confidence estimation matters because LLMs can give unwarranted confidence to incorrect answers, and the relative strengths of estimator and verbalization approaches remain in question. The paper proposes IPoET, which alternates policy optimization with estimator updates to combine reasoning traces and hidden features. Across diverse datasets and backbones, it reports stronger in-domain performance than baselines and superior or comparable out-of-domain performance.

  • Problem

    LLMs can produce incorrect answers with unwarranted confidence, while the relative strengths of estimator-based and verbalization-based confidence methods remain contested.

    Source: §1 · pp. 1–2

  • Method

    IPoET alternates policy optimization with estimator updates, using estimator feedback in policy learning and hidden states from current-policy rollouts to refresh the estimator.

    Source: §1 · p. 2

  • Results

    Across diverse datasets and backbones, IPoET outperforms baselines in-domain and achieves superior or comparable out-of-domain confidence estimation.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The results highlight the value of internal representations and suggest iterative policy-estimator training as a promising direction for trustworthy reasoning models.

    Source: §6 · p. 10

Abstract

from arXiv · show

Confidence estimation is crucial for developing trustworthy large language models (LLMs), with most methods following estimator-based or verbalization-based paradigms. While recent research increasingly focuses on improving verbalized self-reports of confidence, we challenge the prevailing view that this approach surpasses independent confidence estimators. Our empirical study shows that a dedicated confidence estimator can substantially outperform verbalized confidence, indicating that LLMs' internal representations contain richer confidence signals. Building on this finding, we propose Iterative Policy-Estimator Training (IPoET), a framework that synergizes the complementary strengths of verbalized reasoning traces and informative representations. IPoET alternates policy optimization with estimator updating, integrating estimator-derived confidence feedback into policy learning and refreshing the estimator on new policy rollouts. Experiments across diverse datasets and Qwen and Llama backbones demonstrate that, by iteratively exploiting richer hidden features and adapting to the evolving policy distribution, IPoET consistently outperforms both estimator- and verbalization-based baselines in-domain and achieves superior or comparable results across all out-of-domain metrics. For more details, refer to https://github.com/xyk829/ipoet.

1 INTRODUCTION

Reliable confidence estimation matters because incorrect LLM answers can be delivered with unwarranted confidence, limiting use in high-stakes settings. This paper revisits estimator and verbalization approaches, then proposes IPoET to combine hidden-feature signals with verbalized reasoning.

  • High-stakes deployment is limited by LLMs that give unwarranted confidence to hallucinated or incorrect answers.The introduction names medicine, law, and finance as settings where decision errors carry significant consequences.Source: §1 · p. 1
  • Hidden features provide confidence signals, but standalone estimators cannot fully use contemporary LLM reasoning capabilities.The paper identifies these complementary strengths as motivation to combine the paradigms.Source: §1 · p. 2
  • IPoET alternates policy optimization using estimator feedback with estimator updates on hidden states from the current policy.The framework aims to keep the estimator aligned with the evolving policy distribution while using reasoning traces as richer signals.Source: §1 · p. 2
  • Estimators can substantially outperform verbalized confidence when overfitting is controlled, helping explain differing conclusions in prior comparisons.Extended optimization can reduce training error while degrading confidence estimation on unseen data.Source: §1 · pp. 1–2
  • Across diverse datasets and backbones, IPoET outperforms baselines in-domain and achieves superior or comparable out-of-domain confidence estimation.The paper also reports competitive task accuracy.Source: §1 · p. 2

2 EMPIRICAL STUDY

The empirical study defines confidence estimation and contrasts hidden-state estimators with verbalized confidence methods. It revisits their relative performance, examines estimator overfitting, and motivates iterative updates that connect reasoning-oriented policy training with hidden-state estimation.

2.1 PRELIMINARIES

  • Confidence estimation assigns a score from 0 to 1 representing the likelihood that the model’s answer is correct.Correctness is defined by whether the generated answer matches the ground-truth answer.Source: §2.1 · p. 3
  • An independent estimator predicts confidence from the input and response using a linear head on the last non-padding token’s hidden state.The estimator is trained on collected rollouts with a mean squared error objective.Source: §2.1 · p. 3
  • Verbalization methods generate an answer and natural-language confidence, typically using reinforcement learning to improve reasoning traces and confidence.The described approach augments correctness reward with a Brier-style term over verbalized confidence.Source: §2.1 · p. 3
  • Prior studies reported advantages for RL-trained verbalized confidence, including calibration gains under shift and improved discrimination and generalization.The cited comparison motivates revisiting estimator training dynamics and overfitting.Source: §2.1 · p. 3

2.2 ESTIMATOR-BASED VS. VERBALIZATION-BASED

  • Out-of-domain Brier score worsens from an early minimum of 0.184 to 0.258 with extended training, while training Brier score keeps decreasing.In-domain validation first improves and then worsens; the authors adopt the 1× training configuration without additional checkpoint selection.Source: §2.2 · p. 3
  • The estimator improves AUROC by 0.161 in-domain and 0.078 out-of-domain, while reducing Brier score by 0.049 and 0.020, respectively, versus RL-trained verbalized confidence.Figure 1a presents the in-domain and out-of-domain comparison.Source: §2.2 · p. 4
  • Hidden-state estimators and reasoning-linked verbalized methods have distinct strengths, motivating iterative updates of generation and confidence estimation.IPoET’s overview alternates policy and estimator optimization, with each component frozen while the other is updated.Source: §2.2 · p. 4, Figure 2 · p. 4

3 METHOD

IPoET couples reasoning-oriented policy optimization with hidden-state confidence estimation through alternating updates. The estimator both predicts confidence and supplies feedback for generating responses whose correctness is easier to assess.

3.1 ITERATIVE POLICY-ESTIMATOR TRAINING

  • An RLVR warm-up creates a stable answer format, and its rollouts train the initial confidence estimator with supervised regression.The process then freezes the estimator during policy updates and freezes the policy while training the estimator on fresh rollouts.Source: §3.1 · pp. 4–5
  • IPoET alternates policy improvement with estimator refinement on current-policy rollouts, allowing both components to co-adapt across rounds.The estimator serves as both a confidence predictor and feedback source for reasoning generation.Source: §3.1 · p. 5

3.2 REWARD FUNCTION

  • The policy reward combines format and answer-accuracy terms with an estimator-derived negative Brier-style reward from hidden representations.Confidence estimates matching correctness earn higher reward, discouraging overconfidence and underconfidence.Source: §3.2 · p. 5

4 EXPERIMENTS

Across the evaluated datasets and backbones, IPoET performs strongly on in-domain confidence estimation and ranks first or second on out-of-domain confidence metrics. Experiments also examine iterative training, estimator feedback, computational cost, update schedules, and training rounds.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses HotpotQA and Big-Math training, Qwen and Llama backbones, and benchmarks spanning factual, commonsense, expert, and mathematical reasoning.Task accuracy and confidence estimation are measured separately; confidence metrics are AUROC, Brier score, and ECE.Source: §4.1 · pp. 5–6

4.2 MAIN RESULTS

  • On out-of-domain benchmarks, IPoET ranks first or second on every confidence-estimation metric, with results varying across evaluation settings.Under both training datasets, it has the highest AUROC and lowest Brier score and ECE on Qwen.Source: §4.2 · p. 6, Table 1 · p. 7
  • IPoET leads AUROC and Brier score across four in-domain settings, with up to 0.043 AUROC gains and 0.015 Brier-score reductions over RLVR + Estimator.It also has the highest task accuracy and lowest ECE in both Llama settings; RLVR + Estimator exceeds RLCR on AUROC, Brier score, and ECE across all four settings.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 ABLATION AND ANALYSIS

  • Policy updates with a fixed estimator improve AUROC and reduce Brier score in both domains; the in-domain ECE falls from 0.087 to 0.055 while accuracy rises 1.1 percentage points.The subsequent estimator update further improves confidence quality, especially out of domain; out-of-domain accuracy is preserved.Source: §4.3 · pp. 7–8, Figure 3 · p. 8

4.2 MAIN RESULTS

  • With the same update budget, iterative training outperforms joint training on accuracy, AUROC, and Brier score in-domain and out-of-domain; a step-aligned RLCR control indicates gains are not from update count alone.Joint optimization lowers answer accuracy relative to the starting policy, whereas iterative training improves task performance.Source: §4.2 · p. 6, Table 2 · p. 7, §4.3 · p. 8

4.3 ABLATION AND ANALYSIS

  • IPoET takes 500.38 s on 1,000 HotpotQA examples versus RLCR’s 554.60 s, a 9.8% reduction despite its additional estimator.The estimator scores each response in one forward pass, while RLCR generates additional confidence-related reasoning autoregressively.Source: §4.3 · p. 8
  • With equal update budgets, 2×200 performs better in three of four schedule comparisons; Round 3 adds only marginal out-of-domain AUROC and slightly worsens Brier score.Round 1 to Round 2 substantially improves AUROC and Brier score in both domains, so Round 2 is used in the main experiments.Source: §4.3 · pp. 8–9, Table 3 · p. 8, Figure 4 · p. 9

5 RELATED WORK

Related work divides confidence estimation into methods using implicit signals and methods eliciting explicit verbal confidence. IPoET draws on hidden representations while connecting confidence estimates to reasoning-oriented policy training.

Confidence Estimation from Implicit Information

Verbalized Confidence Estimation

6 CONCLUSION

The paper argues that IPoET combines hidden-state confidence estimation with policy reasoning, outperforming estimator- and verbalization-based baselines in-domain and matching or surpassing them out-of-domain. It also reports its research-materials, ethics, and reproducibility practices.

6 CONCLUSION

  • IPoET combines estimator and verbalization strengths, improving in-domain confidence estimation and achieving comparable or better out-of-domain performance with competitive task accuracy.The authors identify internal representations as valuable for confidence estimation and describe iterative policy-estimator training as a promising direction for trustworthy reasoning models.Source: §6 · pp. 9–10
  • Generative AI tools assisted manuscript wording and figure presentation, but not method implementation, dataset generation, or mathematical proofs.The authors say they reviewed AI-assisted work and checked revised figures against experimental data.Source: AI USE STATEMENT · p. 10
  • The study uses publicly available mathematical-reasoning and question-answering benchmarks, with no human participants or private personal data.The authors report no direct ethical risks beyond standard concerns associated with LLM research and use.Source: ETHICS STATEMENT · p. 10
  • The authors provide source code and describe where to find training, experimental, prompt, and evaluation details.Code: https://github.com/xyk829/ipoet.Source: REPRODUCIBILITY STATEMENT · p. 10

A TRAINING DETAILS

The appendix reports the hardware and training configuration for policy and estimator training, along with the policy-training prompt template.

A TRAINING DETAILS

  • Training used eight 80GB NVIDIA A100 GPUs; HotpotQA and Big-Math each required approximately 15 hours.These are the reported setup and runtime figures.Source: A TRAINING DETAILS · p. 15
  • Policy training used GRPO with eight sampled responses per prompt, temperature 0.7, and a 1e-6 learning rate for both datasets.Batch sizes were 64 for HotpotQA and 72 for Big-Math; advantages were not normalized by group standard deviation.Source: A TRAINING DETAILS · p. 15
  • Estimator training used a per-device batch size of 1 with gradient accumulation over 8 steps, and a lower learning rate in round two.The rate decreased from 5×10−6 to 1 × 10−6 for HotpotQA and from 5 × 10−6 to 5 × 10−7 for Big-Math.Source: A TRAINING DETAILS · p. 15
  • The policy-training prompt template is shown in Figure 5.The supplied passage identifies the figure as a prompt template but provides no further template content.Source: A TRAINING DETAILS · p. 15, Figure 5 · p. 15

B DETAILS OF EVALUATION METRICS

The appendix specifies accuracy protocols and describes AUROC, Brier score, and ECE evaluation, including the confidence-bin setup for ECE.

B DETAILS OF EVALUATION METRICS

  • Accuracy uses exact match for HotpotQA variants, a math-verification library for math datasets, and an LLM judge for CommonsenseQA, GPQA, SimpleQA, and TriviaQA.The judge model family differs from the evaluated backbone family, runs at temperature 0, and judges answer correctness without thinking traces.Source: B DETAILS OF EVALUATION METRICS · p. 15
  • AUROC is computed by sweeping the confidence threshold and integrating the resulting ROC curve.The text defines TPR and FPR as the true positive rate and false positive rate.Source: B DETAILS OF EVALUATION METRICS · p. 15
  • The Brier score is computed from confidence scores c_i and correctness labels z_i.Source: B DETAILS OF EVALUATION METRICS · p. 15
  • ECE uses 10 confidence bins and compares average correctness with average confidence within each bin.The passage introduces ECE as the aggregation of these bin-level quantities.Source: B DETAILS OF EVALUATION METRICS · p. 16
  • Table 4 compares IPoET with frontier commercial models, using HotpotQA results and a math average across MATH-500, GSM8K, and Big-Math.Source: Table 4 · p. 16

C COMMERCIAL MODEL COMPARISON

Across the HotpotQA and math settings, IPoET achieves the best reported AUROC, Brier score, and ECE among the evaluated commercial and paper models. The authors report that commercial models' verbalized confidence is less reliable in these settings.

C COMMERCIAL MODEL COMPARISON

  • IPoET ranks best on AUROC, Brier score, and ECE across both reported settings among the evaluated models.The HotpotQA comparison uses an IPoET model trained on HotpotQA; the math comparison uses one trained on Big-Math and averaged over MATH-500, GSM8K, and Big-Math.Source: C COMMERCIAL MODEL COMPARISON · p. 16
  • The comparison includes GPT-5 mini, Claude Haiku 4.5, DeepSeek-V4-Flash, and Gemini 3 Flash.The authors describe these commercial models as having reasonable confidence-estimation ability, while their verbalized confidence is less reliable in these settings.Source: C COMMERCIAL MODEL COMPARISON · p. 16
  • The evaluated versions were gpt-5-mini-2025-08-07, claude-haiku-4-5-20251001, deepseek-v4-flash, and gemini-3-flash-preview-nothinking.Source: C COMMERCIAL MODEL COMPARISON · p. 16

Related papers

Loading 2609.34454v1…