PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Models

Shuoyuan Wang, Beier Luo, Hao Zeng, Chengyao Yu, Songxin Zhang, Zejian Xie, Bingyi Jing, Hongxin Wei

arXiv:2609.32470v1cs.LG

TL;DR

Large reasoning models often verbalize overconfident scores, while confidence-aware RL is constrained by the model’s pre-RL confidence distribution. The paper proposes CalibSFT to shape a broader calibrated prior before RL; across 16 benchmarks it improves calibration and discrimination across RL methods while preserving comparable accuracy.

  • Problem

    Confidence-aware RL learns from on-policy rollouts, but concentrated pre-RL confidence priors limit exploration and constrain calibration.

    Source: §2.2 · p. 3, §3 · p. 4

  • Method

    CalibSFT builds confidence targets from sampled-response success rates and correctness, balances target values, and supervises confidence on all responses while restricting reasoning supervision to correct ones.

    Source: §1 · p. 2, §4 · p. 5, §4.1 · p. 5

  • Results

    Across 16 benchmarks, CalibSFT improves calibration and discrimination across five confidence-aware RL methods without compromising task accuracy.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    CalibSFT also provides practical benefits for selective prediction and model routing.

    Source: §1 · p. 2

    The theoretical analysis assumes confidence and correctness are conditionally independent given the context representation, and target probabilities are estimated from base-model rollouts.

    Source: §3 · p. 4, §4.1 · p. 5

Abstract

from arXiv · show

Large reasoning models (LRMs) often suffer from overconfidence when expressing their uncertainty. Confidence-aware reinforcement learning (RL) offers a promising way to optimize calibration. However, it relies on on-policy rollouts and is thus constrained by the model's pre-RL confidence distribution, which we term confidence prior. In this work, we reveal that off-the-shelf LRMs exhibit a confidence prior heavily concentrated on a few high values, which persists throughout RL. Theoretically, we prove that this concentration suppresses policy gradient updates for rarely sampled confidence values and inflates the lower bound on expected Brier risk. To overcome this exploration bottleneck, we propose CalibSFT, a plug-and-play supervised fine-tuning stage that shapes a calibrated confidence prior with broad support before RL. For each question, CalibSFT constructs confidence targets combining its success rate with response-level correctness, which provably preserves proper-scoring optimality, and then balances training responses across the confidence spectrum to enable diverse confidence exploration during RL. To learn from incorrect responses without imitating their reasoning, CalibSFT introduces correctness-conditional supervision, guiding confidence across all responses while supervising reasoning only on correct ones. Across 16 mathematical and general reasoning benchmarks, incorporating CalibSFT reduces calibration errors and improves discrimination across five representative RL algorithms while preserving comparable accuracy. Furthermore, CalibSFT delivers practical benefits for downstream selective prediction and model routing. Our code is available at https://github.com/ml-stat-Sustech/verbalized-confidence-training.

1 INTRODUCTION

Large reasoning models can verbalize confidence in a single black-box generation, but often remain overconfident on incorrect answers. This section introduces CalibSFT to broaden the confidence prior before RL and reports calibration and discrimination gains across benchmarks and RL methods.

1 INTRODUCTION

  • Reliable confidence should match the probability of correctness, helping indicate when answers can be trusted for selective prediction and model routing.Accuracy gains alone do not ensure reliable confidence because models can remain overconfident on incorrect answers.Source: §1 · p. 1
  • Verbalized confidence can be obtained in a single black-box generation, but off-the-shelf models often assign concentrated high scores regardless of correctness.Existing post-training approaches include supervised fine-tuning on constructed targets and reinforcement learning guided by calibration rewards.Source: §1 · p. 1
  • On-policy confidence-aware RL is constrained by the pre-RL confidence prior, which concentrates on a few high values and limits exploration.The concentration persists during policy optimization and raises the lower bound on expected Brier risk.Source: §1 · p. 2
  • CalibSFT shapes a broad-support confidence prior before RL using mixed success-rate and response-correctness targets, target-balanced sampling, and correctness-conditional supervision.The mixed targets preserve question-level optimal confidence under strictly proper scoring rules; reasoning and answer supervision is restricted to correct responses.Source: §1 · p. 2
  • Across 16 benchmarks, CalibSFT improves calibration and discrimination across five confidence-aware RL methods without compromising task accuracy.The reported benefits also extend to different confidence formats, model families, selective prediction, and model routing.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries define verbalized confidence and the metrics used to evaluate it, then describe confidence-aware RL as reward optimization with on-policy rollouts. The on-policy constraint means optimization learns only from sampled confidence values.

2 PRELIMINARIES

  • An LRM generates a reasoning trace, answer, and confidence c in [0, 1], interpreted as the estimated probability that the answer is correct and conditioned on the question, reasoning, and answer.An automated verifier supplies the correctness label.Source: §2 · p. 2

2.1 CALIBRATION METRICS

  • Calibration is evaluated with ECE and Brier score, while AUROC measures discrimination between correct and incorrect answers.ECE assesses bin-level calibration; Brier score measures response-level squared error, and zero ECE need not imply discrimination.Source: §2.1 · p. 3

2.2 CONFIDENCE-AWARE REINFORCEMENT LEARNING

  • Confidence-aware RL adds a reward depending on confidence and correctness to the correctness reward, and typically optimizes it with GRPO using normalized group-wise advantages.Because optimization is on-policy, it learns only from confidence values sampled during rollouts.Source: §2.2 · p. 3

3 MOTIVATION

The motivation section shows that confidence priors concentrate on high values and remain narrow during RL, limiting exploration and discrimination. Its analysis connects rare-value gradient suppression to increased Brier risk and weak confidence sensitivity to task difficulty.

3 MOTIVATION

  • Across three base models, fewer than 5% of responses have confidence below 0.5, and confidence mass concentrates on a few high values for correct and incorrect responses.As a result, RL starts from a concentrated prior and rarely samples low confidence values early in training.Source: §3 · p. 3, Figure 1 · p. 4
  • RLCR lowers ECE on Qwen3-8B but does not improve AUROC, while low confidence-token entropy tracks few distinct sampled confidence values and limits exploration.The text reports that restricted exploration and low entropy persist across other confidence-aware RL methods.Source: §3 · pp. 3–4, Figure 2 · p. 5
  • Under a concentrated prior, the policy-gradient norm is bounded by 2εΔh, and the gradient shifting mass away from a dominant value is bounded by εΔh.Because gradients scale with sampling probability, rarely sampled confidence values receive weaker updates, hindering escape from a suboptimal initialization.Source: §3 · p. 4
  • Across Math500 and MinervaMath, Pass@1 drops by 39.1% while mean confidence falls by only 9.7%.The paper attributes the mismatch to confidence remaining concentrated on a few high values, limiting lower confidence on harder questions.Source: §3 · p. 4, Figure 3 · p. 5
  • When confidence rarely falls near the true correctness probability, the expected Brier-loss lower bound includes excess error from that mismatch.The bound combines conditional variance with an excess term that grows as the probability of sampling confidence within δ of the true probability decreases.Source: §3 · pp. 4–5

4 METHOD

CalibSFT prepares confidence-aware RL by constructing confidence targets that retain question-level proper-scoring optimality, broadening confidence support, and supervising confidence on all responses while limiting reasoning supervision to correct ones.

4.1 PROPER-SCORING CONFIDENCE TARGETS

  • CalibSFT constructs mixed confidence targets from question-level success rates and response-level correctness, preserving each question’s optimal confidence under strictly proper scoring rules.The mixing weight balances question-level calibration with within-question discrimination.Source: §4.1 · p. 5
  • Target-balanced sampling broadens the initial confidence prior because 61% of raw targets are exactly 0 or 1.The method uses n = 50 rollouts and λ = 0.5 by default; balancing maintains comparable calibration while broadening confidence support.Source: §4.1 · p. 6

4.2 CORRECTNESS-CONDITIONAL SUPERVISION

  • CalibSFT supervises confidence on all responses but reasoning and answers only on correct responses, teaching low confidence on errors without imitating incorrect solutions.The confidence term averages over all responses, while reasoning and answer supervision averages over correct responses.Source: §4.2 · p. 6

4 METHOD

  • CalibSFT initializes confidence-aware RL without changing its objective, so the stage applies across different RL methods.The paper evaluates it across RL methods and separately ablates target construction, mixing weight, and supervision design.Source: §4 · p. 5, §4.2 · p. 6

5 EXPERIMENTS

Across mathematical and general-reasoning benchmarks, CalibSFT initialization improves calibration and discrimination across confidence-aware RL methods while maintaining or improving accuracy. The experiments also examine confidence alignment across tasks, reasoning budgets, confidence formats, and model families.

5.1 EXPERIMENTAL SETUP

  • The evaluation uses Qwen3-8B, five confidence-aware RL baselines, and ID/OOD mathematical and general-reasoning datasets, with Pass@1, AUROC, Brier score, and ECE metrics.All methods use the DAPO training framework; evaluation samples 32 responses per AIME problem and four per other benchmark problem.Source: §5.1 · pp. 6–7

5.2 OVERALL RESULTS

  • Across five confidence-aware RL baselines, CalibSFT improves calibration and discrimination on ID and OOD benchmarks without sacrificing average accuracy.Average Pass@1 improves by 1.20 in-distribution and 1.49 out-of-distribution across the five methods.Source: Table 1 · p. 7, §5.2 · p. 7

5.3 CALIBRATION ANALYSIS

  • CalibSFT improves the correlation between task-level mean confidence and accuracy across all RL baselines and both ID and OOD splits.For DCPO, Spearman correlation rises from 0.619 to 0.976 on mathematical tasks and from 0.548 to 0.643 on general reasoning.Source: §5.3 · p. 8, Table 2 · p. 8
  • At every tested AIME 2026 token budget, CalibSFT lowers RLCR’s ECE while maintaining comparable accuracy.At 10,000 tokens, ECE drops from 22.71 to 10.97.Source: Figure 5 · p. 8, §5.3 · p. 9

5.4 GENERALITY OF CALIBSFT

  • Across four confidence formats, CalibSFT improves RLCR’s AUROC and ECE on the 8 ID and 8 OOD benchmarks.The ECE reduction is largest with the fine probability grid; the formats include fine and coarse probability grids, integer scores, and linguistic labels.Source: Table 3 · p. 9, §5.4 · p. 9
  • On Gemma4-E2B-Instruct, CalibSFT lowers average ECE from 23.41 to 12.49 and raises average AUROC from 70.49 to 73.95 across six benchmarks.The evaluation covers three ID and three OOD benchmarks spanning varying difficulty.Source: §5.4 · p. 9, Table 4 · p. 9

6 CONCLUSION

CalibSFT shapes a broad, calibrated confidence prior before RL, with targets that preserve proper-scoring optimality and confidence supervision that avoids imitating incorrect reasoning. The paper connects concentrated confidence priors to weaker RL exploration and higher Brier risk.

6 CONCLUSION

  • Across 16 benchmarks, CalibSFT improves calibration and discrimination across diverse RL frameworks while supervising confidence without imitating incorrect reasoning.The conclusion presents the method as a foundation for reliable and trustworthy reasoning models.Source: §6 · pp. 9–10

A RELATED WORK

  • Verbalized confidence is produced alongside an answer in one generation, while its calibration may require an extra query or rewriting pass.The related-work discussion contrasts this setting with token probabilities, probes, and sampling-based confidence estimates, which have different access or inference-cost requirements.Source: A RELATED WORK · pp. 16–17

B.1 PROOF OF PROPOSITION 1

  • When a confidence value dominates the policy with probability 1−ε, the L1 policy-gradient bound scales with ε, suppressing updates as the policy concentrates.For confidence-aware RL, the correctness-label reward contributes zero gradient; the calibration penalty supplies the policy-dependent gradient.Source: §B.1 · p. 18

B.2 PROOF OF PROPOSITION 2

  • A confidence prior with little probability mass near the true posterior incurs excess Brier loss, even when nearby confidence values remain possible.With support restricted to a set, the distance from the true posterior to the nearest available confidence value translates into excess Brier risk.Source: §B.2 · pp. 18–19

B.3 PROOF OF PROPOSITION 3

The mixed confidence target combines question-level success with each response’s correctness. It preserves the question-level optimal confidence under strictly proper scoring rules while trading lower target variance against response-level bias.

B.3 PROOF OF PROPOSITION 3

  • The mixed target preserves the question-level success rate as the unique population minimizer under any strictly proper scoring loss.Its expected target equals the question-level success rate, so the expected loss matches that of the binary correctness label.Source: §B.3 · p. 19
  • For any 0<λ<1, target variance falls below binary-label variance while correct and incorrect responses retain a within-group margin of 1−λ.At λ=0 the target is the binary label; at λ=1 it is the question success rate and provides zero within-question discrimination.Source: §B.3 · p. 19
  • With n=50 and λ=0.5, the within-group margin is 0.50, the expected margin is about 0.51, and target variance drops by over 73% relative to binary labels.Conditioned on a response context, shrinkage toward the question success rate trades response-level bias for lower target variance while retaining a correct-versus-incorrect margin.Source: §B.3 · pp. 19–20

C.1 TRAINING PIPELINES

The training pipeline builds CalibSFT from sampled responses, balances confidence targets, and uses correctness-conditional supervision before confidence-aware RL. Evaluation uses benchmark-specific answer verification, and the confidence target can be expressed in multiple formats.

C.1 TRAINING PIPELINES

  • CalibSFT samples 50 responses per question, assigns mixed targets, balances examples across target values, and supervises reasoning only for correct responses.Incorrect responses receive confidence-token supervision only; the balanced training set contains 34,900 examples across 11,539 questions.Source: §C.1 · p. 20

C.2 EVALUATION PROTOCOL

  • Evaluation reports Pass@1 and ECE from sampled responses, using symbolic matching, benchmark scripts, or an LLM judge for correctness labels.Unless stated otherwise, RL evaluations use the final checkpoint; ECE uses 10 equal-width bins.Source: §C.2 · pp. 21–22

C.3 CONFIDENCE FORMATS

  • The confidence target can be expressed as a fine or coarse probability grid, an integer score, or a low/medium/high linguistic label.The format-general evaluation keeps training data fixed and changes only how the target is expressed.Source: §C.3 · p. 22

D.1 TRAINING DYNAMICS

Across three confidence-aware RL baselines, CalibSFT initialization sustains lower Brier score and superior AUROC while preserving task accuracy. It also broadens confidence exploration during training.

D.1 TRAINING DYNAMICS

  • With CalibSFT, all three baselines sustain lower Brier score and superior AUROC throughout training while preserving task accuracy.Without CalibSFT, ECE falls from 23.8–25.8% to 7.8–14.2%, while AUROC falls from 70.8–73.5% to 64.1–68.0%.Source: §D.1 · p. 22
  • CalibSFT increases confidence-token entropy by more than threefold and roughly doubles distinct confidence values sampled per question during training.Without CalibSFT, each question yields fewer than 5.3 distinct confidence values on average across eight rollouts.Source: §D.1 · p. 22

D.2 PER-DATASET MAIN RESULTS

Across in-distribution and out-of-distribution benchmarks, CalibSFT initialization improves calibration and discrimination on most tasks while maintaining comparable accuracy. These gains are stable across three random seeds.

D.2 PER-DATASET MAIN RESULTS

  • Standard RL improves Brier score and ECE but degrades AUROC over training, whereas CalibSFT improves all three while preserving task accuracy.The results are shown in training dynamics for RLCR, CoCA, and DCPO with and without CalibSFT initialization.Source: Figure 6 · p. 23
  • CalibSFT initialization improves calibration and discrimination on most in-domain mathematical and out-of-domain general-reasoning datasets, with largest gains on benchmarks where baseline RL is heavily overconfident.It lowers ECE in 61 of 80 method–dataset pairs; on DROP, baseline RL models have AUROC ≈59%, and CalibSFT substantially improves discrimination.Source: Table 6 · p. 24, §D.2 · p. 24, Table 7 · p. 25
  • On six benchmarks with baseline ECE above 25%, CalibSFT lowers both Brier score and ECE across all five RL algorithms.On OOD benchmarks, subsequent confidence-aware RL raises AUROC from 61.20% to over 71% for four of five methods.Source: §D.2 · pp. 24–25

D.3 ROBUSTNESS ACROSS RANDOM SEEDS

  • CalibSFT consistently outperforms RL baselines across calibration and discrimination metrics over three seeds, while Pass@1 remains comparable.The repeat runs evaluate RLCR and DCPO on Qwen3-8B using seeds 43, 44, and 45.Source: §D.3 · p. 26, Table 8 · p. 26

D.4 EVALUATION ON ADDITIONAL METRICS

CalibSFT improves fine-grained confidence quality and broadens the range of confidence values used across RL frameworks. These metrics capture calibration, ranking, and confidence diversity beyond standard measures.

D.4 EVALUATION ON ADDITIONAL METRICS

  • CalibSFT reduces overconfidence and misranking errors while improving question-level confidence alignment across baseline RL methods.Table 9 reports lower PCE, E-AURC, RCE, and capability Brier score with CalibSFT.Source: §D.4 · p. 26, Table 9 · p. 27
  • Top-1 and Top-3 track concentration in the most frequent confidence values, while effective support estimates how many values are used regularly.Effective support is the exponential of confidence entropy; unlike raw support size, it is not inflated by rare values.Source: §D.4 · p. 27
  • For RLCR OOD, effective support rises from 7.03 to 72.83, alongside broader confidence support across all five RL frameworks and both splits.Top-1 and Top-3 shares drop sharply, and both support size and effective support increase.Source: §D.4 · p. 27, Table 10 · p. 27

D.5.1 CONFIDENCE TARGET CONSTRUCTION

CalibSFT's target construction, balanced sampling, and correctness-conditional supervision are designed to provide calibrated confidence targets and broad confidence exploration. The supplied results also examine other confidence estimators, calibration across task difficulty, training cost, and downstream applications.

D.5.3 TARGET MIXING WEIGHT

  • λ = 0.5 achieves the highest AUROC on both splits and the lowest Brier score on ID benchmarks among the tested mixing weights.Any λ ≤ 0.5 preserves the ordering of correct and incorrect response targets; larger weights can reverse it.Source: Table 13 · p. 29, §D.5.3 · p. 29

D.5.1 CONFIDENCE TARGET CONSTRUCTION

  • 79.68 AUROC and 17.74 Brier score are the best post-SFT results across the tested target methods, averaged over eight mathematical benchmarks.At target construction, the mixed target separates correct from incorrect responses by construction at λ = 0.5, with AUROC 100.00 and Brier score 7.92 versus 31.67 for question-level targets.Source: §D.5.1 · p. 29, Table 12 · p. 29

D.5.2 TARGET-BALANCED SAMPLING

  • Balancing drops confidence mass at the extremes from 75.56% to 14.49% and expands effective support from 3.33 to 29.86.The change has little impact on standard calibration error and provides a broader confidence action space for downstream RL.Source: §D.5.2 · p. 29, Table 14 · p. 30, Figure 8 · p. 30

D.5.4 CORRECTNESS-CONDITIONAL SUPERVISION

  • Supervising confidence on all responses while restricting reasoning and answer supervision to correct responses gives the best AUROC, Brier score, and ECE among the tested variants.Confidence-only supervision leaves Pass@1 far behind, and omitting segment balancing worsens calibration.Source: §D.5.4 · p. 30, Table 15 · p. 30

D.6 INTEGRATION WITH OTHER CONFIDENCE ESTIMATORS

  • CalibSFT initialization improves calibration and discrimination across token-probability, self-consistency, and verbalized-confidence estimators on both splits.With token probability, ID AUROC rises from 83.60 to 87.66 and OOD Brier score falls from 31.22 to 22.80.Source: §D.6 · p. 31, Table 16 · p. 31

D.7 CONFIDENCE VISUALIZATION

  • CalibSFT reduces overconfidence on harder benchmarks and underconfidence on the easier MATH-500 benchmark.ECE falls from 23.05% to 12.00% on MinervaMath, from 52.43% to 30.89% on NQOpen, and from 7.53% to 3.89% on MATH-500.Source: §D.7 · pp. 31–32, Figure 9 · p. 32, Figure 10 · p. 32

D.8 RUNNING TIME ANALYSIS

  • CalibSFT takes 0.33 minutes per step versus 4.77 minutes for RLCR and accounts for 6.9% of RLCR's GPU-hours.The dominant cost is one-time rollout collection, which can be reused across downstream confidence-aware RL methods.Source: §D.8 · p. 33, Table 17 · p. 33

E DOWNSTREAM APPLICATIONS

  • The downstream evaluation considers selective risk control and confidence-guided routing to reduce reliance on a stronger model while bounding accuracy loss.Selective risk control seeks higher response coverage under an error-rate budget; routing seeks fewer deferrals under an accuracy-loss tolerance.Source: E DOWNSTREAM APPLICATIONS · p. 33

E.1 SELECTIVE RISK CONTROL

CalibSFT increases certified coverage under fixed selective-risk budgets and reduces routing deferrals under accuracy-loss guarantees. Both applications use confidence thresholds selected with finite-sample risk bounds.

E.1 SELECTIVE RISK CONTROL

  • Selective risk control chooses a confidence threshold to maximize coverage while keeping selective risk below a fixed budget.The threshold is selected using one-sided Clopper–Pearson upper bounds across 101 candidates, with a union-bound correction.Source: §E.1 · pp. 33–34
  • At a 10% ID risk budget, CalibSFT raises RLCR coverage from 31.4% to 62.6%; at a 20% OOD budget, it yields 16.8%–22.8% coverage where all five baselines yield zero.Certified coverage is higher across methods and risk budgets, and tracks the perfect-ranking reference more closely.Source: Table 18 · p. 34, §E.1 · pp. 34–35, Figure 11 · p. 34

E.2 CONFIDENCE-GUIDED MODEL ROUTING

  • Confidence-guided routing retains a local answer above a threshold and otherwise defers to Qwen3.7-Flash, with a bound on accuracy loss.The routing threshold is certified using upper bounds across 102 candidates.Source: §E.2 · p. 35
  • CalibSFT lowers routing deferrals by an average of 12.3 and 27.7 percentage points at ε = 1% and ε = 2%, respectively.At ε = 2%, DCPO's deferral rate falls from 82.9% to 33.3%, while token savings rise from 5.6% to 43.5%; accuracy remains close to the 82.58% full-deferral baseline.Source: §E.2 · p. 35, Table 19 · p. 35

Related papers

Loading 2609.32470v1…