PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty

Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, Jacob Andreas

arXiv:2507.16806v2cs.LGcs.AIcs.CL

TL;DR

Correctness-only RL can worsen calibration by rewarding guessing, motivating methods that train reasoning models to express uncertainty. RLCR combines correctness and Brier calibration rewards, and it improves calibration while preserving accuracy across in-domain and out-of-domain evaluations, with additional benefits from verbalized confidence at test time.

  • Problem

    Binary correctness rewards can incentivize overconfident guessing, while reasoning models may become poorly calibrated and hallucinate more after RL.

  • Method

    RLCR trains models to generate answers and confidence estimates, optimizing correctness jointly with a Brier-score calibration reward.

  • Results

    RLCR substantially improves calibration without sacrificing accuracy across in-domain and out-of-domain evaluations, outperforming ordinary RL and post-hoc confidence classifiers.

  • Takeaways & Limitations

    Verbalized confidence supports confidence-weighted test-time scaling, while RLCR improves confidence consistency across reasoning samples and contradictory answers.

  • Takeaways & Limitations

    The theoretical guarantee requires a bounded proper scoring rule, and the method mainly generates one answer with one confidence score.

Abstract

from arXiv · show

When language models (LMs) are trained via reinforcement learning (RL) to generate natural language "reasoning chains", their performance improves on a variety of difficult question answering tasks. Today, almost all successful applications of RL for reasoning use binary reward functions that evaluate the correctness of LM outputs. Because such reward functions do not penalize guessing or low-confidence outputs, they often have the unintended side-effect of degrading calibration and increasing the rate at which LMs generate incorrect responses (or "hallucinate") in other problem domains. This paper describes RLCR (Reinforcement Learning with Calibration Rewards), an approach to training reasoning models that jointly improves accuracy and calibrated confidence estimation. During RLCR, LMs generate both predictions and numerical confidence estimates after reasoning. They are trained to optimize a reward function that augments a binary correctness score with a Brier score -- a scoring rule for confidence estimates that incentivizes calibrated prediction. We first prove that this reward function (or any reward function that uses a bounded, proper scoring rule) yields models whose predictions are both accurate and well-calibrated. We next show that across diverse datasets, RLCR substantially improves calibration with no loss in accuracy, on both in-domain and out-of-domain evaluations -- outperforming both ordinary RL training and classifiers trained to assign post-hoc confidence scores. While ordinary RL hurts calibration, RLCR improves it. Finally, we demonstrate that verbalized confidence can be leveraged at test time to improve accuracy and calibration via confidence-weighted scaling methods. Our results show that explicitly optimizing for calibration can produce more generally reliable reasoning models. Code, models, and further info is available at https://rl-calibration.github.io/.

1 INTRODUCTION

Reasoning-focused RL improves difficult-task performance, but binary correctness rewards can incentivize overconfident guessing and worsen calibration. RLCR addresses this by training models to produce answers and confidence estimates, with theoretical and empirical evidence for improved calibration without sacrificing accuracy.

  • Reasoning models trained with reinforcement learning achieve state-of-the-art performance on challenging math and programming tasks.
  • Binary correctness rewards treat confident correctness and guessing identically, incentivizing overconfident guesses.
  • Reasoning models can become overconfident after RL, with worsened calibration and increased hallucination rates, especially under correctness-only rewards.
  • The paper asks whether reasoning models can optimize correctness and calibration, and whether reasoning chains can improve calibration.
  • RLCR trains models to output answers and verbalized confidence scores using a reward combining correctness with a calibration objective based on proper scoring rules.
  • RLCR matches RLVR accuracy while reducing expected calibration error from 0.37 →0.03 on HotpotQA and 0.26 →0.10 on math datasets.
  • RLCR improves calibration and test-time scaling outcomes across in-domain and out-of-domain evaluations, whereas ordinary RL can harm calibration.

2 PRELIMINARIES

This section introduces language-model output optimization, binary correctness rewards, and proper scoring rules for confidence estimates. Proper scoring rules reward confidence estimates that match true outcome probabilities.

  • Language-model training optimizes outputs against rewards comparing predicted outputs with ground-truth outputs.
  • RLVR uses a binary correctness reward that assigns one when the model output matches the ground-truth answer and zero otherwise.
  • A scoring rule maps a confidence estimate and binary outcome to a scalar score for evaluating confidence quality.
  • A proper scoring rule is optimized when reported confidence matches the true probability of the outcome.
  • Log-loss is one proper scoring rule, alongside other scoring rules that share probability-matching behavior.

3 METHOD

RLCR combines correctness rewards with Brier-based calibration rewards while training models to produce answers and confidence estimates. The resulting objective theoretically incentivizes both calibrated confidence and choosing the most likely correct answer.

  • RLCR modifies reasoning training by combining a standard correctness reward with a Brier-score-based calibration reward.
  • The method trains models to generate reasoning chains containing both answers and confidence estimates.
  • The reward penalizes incorrect answers with high confidence and correct answers with low confidence.
  • Theorem 1 states that the calibration term does not reduce accuracy under the RLCR objective.
  • For any prediction, expected RLCR reward is maximized when confidence equals its success probability.
  • Among calibrated predictions, expected reward is maximized by the prediction with the greatest success probability.
  • The theoretical result requires a bounded proper scoring rule; log loss does not incentivize correctness in the same way.

4 EXPERIMENTS

The experiments evaluate RLCR against standard RL, confidence classifiers, and test-time scaling methods across in-domain and out-of-domain tasks. RLCR improves calibration while preserving accuracy, and verbalized confidence supports stronger aggregation and modest ensemble-based calibration gains.

  • Experimental results: RLCR maintains competitive accuracy while substantially improving calibration over RLVR and other baselines in both in-domain and out-of-domain evaluations.On HotpotQA, RLCR is comparable to RLVR in accuracy; across six OOD datasets, it improves calibration while maintaining or slightly improving task accuracy.
  • Test-time scaling: Confidence-weighted majority vote outperforms vanilla majority vote, max-confidence, and likelihood-based baselines as the number of test-time samples increases.The method uses verbalized confidence as a proxy reward without additional supervision or external reward models.
  • Test-time scaling: Calibration improves modestly as the confidence-estimation ensemble grows, providing lightweight residual-noise reduction especially for harder questions.The ensemble averages confidence estimates from multiple analyses, while most questions exhibit limited uncertainty about uncertainty.
  • Confidence consistency: RLCR confidence estimates are generally self-consistent, but predicted confidence sums remain overconfident on out-of-distribution datasets.On HotpotQA, confidence sums cluster near the ideal value of 1; OOD sums exceed 1, although RLCR remains closer to the ideal than RLVR.
  • Calibration ablations: Explicit uncertainty reasoning improves calibration for both RLCR and RLVR, while calibration-aware reward learning remains more effective than prompting alone.RLVR with analysis improves over vanilla RLVR but remains behind both RLCR variants, including RLCR without analysis.
  • Calibration ablations: RLCR without uncertainty analysis achieves 61.7% accuracy versus RLVR’s 63.0%, while reducing ECE from 0.37 to 0.09 on HotpotQA.The RLCR variant uses 113 tokens versus 92 for RLVR, so its calibration gain does not require a large accuracy or token-cost change.

5 RELATED WORK

Prior work estimates uncertainty in language models through verbalized confidence, sampling agreement, internal features, and reinforcement learning. These approaches differ in supervision, computational cost, generality, and whether confidence is integrated into training.

  • Post-hoc verbalizations: Post-hoc verbalization methods ask models to report confidence after answering, but prior studies find that language models often remain overconfident.Some work fine-tunes models on empirical accuracy targets, while other studies compare verbalized confidence with conditional probabilities.
  • Sampling-based methods: Sampling-based methods use response agreement, semantic entropy, or consistency among diverse responses as confidence proxies, but can be costly or require clear ground truth.These methods estimate uncertainty from multiple generated answers rather than directly training the model’s confidence output.
  • Internal probing: Internal probing extracts confidence from token probabilities or model representations, offering fine-grained estimates but limited generality or requiring additional models.Examples include token-level uncertainty, true-or-false probability prompts, external probes, and linear probes over final-layer embeddings.
  • RL-based methods: RL-based approaches train models to produce calibrated verbal confidence using speaker-listener objectives, reward-model training, or proper scoring rules.RLCR is most closely related to prior methods that use proper scoring rules as reinforcement-learning rewards.

6 CONCLUSION

RLCR improves calibration while preserving accuracy, including across in-domain and out-of-domain settings. The results also show more coherent confidence and benefits from test-time scaling, though substantial calibration errors remain.

  • RLCR improves both accuracy and calibration while preserving task performance in- and out-of-distribution.
  • RLCR improves confidence self-consistency and benefits from test-time scaling.
  • Out-of-domain calibration error often remains high, and models may still assign high confidence to contradictory answers.

A PROOF OF THEOREM 1

The proof establishes that proper scoring rules incentivize calibrated confidence, while boundedness ensures the combined reward still favors more accurate predictions. The Brier score satisfies this requirement, whereas the logarithmic score does not.

  • Calibration incentive: For any response, expected reward is maximized when confidence q equals the response’s true success probability py if and only if the scoring rule is proper.
  • Correctness incentive: The correctness incentive holds when S(p, 1) − S(p, 0) is bounded by λ across p ∈[0, 1].
  • Theorem 1: The theorem combines calibration from proper scoring with correctness from bounded score differences.
  • Corollary 1: A bounded proper scoring rule admits a finite λ that jointly incentivizes calibration and correctness.
  • Corollary 1: An unbounded score difference can make the reward prefer a less accurate response, so no finite λ guarantees the correctness condition.
  • Examples: The Brier score is bounded, whereas the logarithmic score is unbounded and fails the finite-λ condition.

B.3 EVALUATION DATASETS

The evaluation spans factual, multi-hop, commonsense, and mathematical question-answering datasets. Answers and confidence values are extracted through method-specific pipelines, then scored with identical downstream correctness procedures.

  • Datasets: The benchmark suite includes HotPotQA, TriviaQA, SimpleQA, GPQA, CommonSenseQA, Math-500, GSM8K, and Big-Math-Digits.
  • Datasets: HotPotQA-Modified uses 500 held-out validation examples and exact-match correctness evaluation.
  • Datasets: Math-500, GSM8K, and Big-Math-Digits use math-verify for correctness evaluation.
  • Evaluation pipeline: RLCR outputs confidence directly, whereas RLVR confidence is elicited after generation and classifier methods use RLVR generations as input.
  • Evaluation pipeline: After extraction, all methods are evaluated identically using exact-match, LLM-as-a-judge, or math-verify depending on the dataset.

E.2 TOY EXPERIMENT

The toy arm experiment tests whether calibration rewards preserve accurate prediction under substantial uncertainty. Brier-based training avoids the degenerate behavior induced by logarithmic scoring, while broader comparisons show RLCR generally maintains accuracy and improves calibration.

  • E.2 Toy Experiment: The toy task samples a five-arm distribution, reveals a short sequence of IID draws, and asks the model to predict the next arm with confidence.
  • E.2 Toy Experiment: The model may abstain by outputting an invalid arm −1 with confidence q = 0, and short observations induce substantial aleatoric uncertainty.
  • E.1 Intuition: Under logarithmic scoring, the expected reward is non-monotonic in true correctness probability and can favor intentionally incorrect low-confidence answers.
  • E.2 Toy Experiment: RLCR-Log converges to predicting the invalid arm with q = 0, achieving nearly perfect calibration but 0 accuracy.
  • E.2 Toy Experiment: RLCR-Brier continues making genuine arm predictions, achieving non-trivial accuracy and stable calibration.
  • Broader comparisons: In realistic QA, RLCR-Log performs slightly worse than RLCR-Brier, but maintains reasonable accuracy without degenerate predictions.
  • Broader comparisons: Across OlMo-2 and Qwen3 evaluations, RLCR matches RLVR in accuracy while improving calibration in in-domain and out-of-domain settings.

H.1 CONFIDENCE DISTRIBUTIONS ACROSS DIFFERENT INPUTS

RLCR produces confidence estimates that vary across inputs and remain closer to ideal calibration than RLVR or a post-hoc BCE classifier, both in- and out-of-distribution. Confidence variability is lower when the answer is fixed than across different reasoning trajectories.

  • Evaluation Displays: Calibration charts combine accuracy-per-bin curves with confidence-frequency histograms, allowing calibration and confidence diversity to be assessed together.Ideal calibration places the curve near the orange dashed diagonal, while histograms reveal whether confidence values are diverse or clustered.
  • In-Distribution HotpotQA: RLCR uses a broad confidence range with substantial mid-confidence mass, while its accuracy-per-bin curve closely tracks ideal calibration on in-distribution HotpotQA.RLVR concentrates nearly all predictions at 0.9–1.0, whereas the BCE classifier improves alignment but still underperforms RLCR.
  • Out-of-Distribution Evaluation: RLCR maintains a healthy 0–1 confidence spread and better calibration than RLVR or its BCE classifier variant across six out-of-distribution datasets.RLVR often collapses confidence to 1.0; verbalized confidences of 0.8–0.9 correspond to only around 0.3 average accuracy.
  • Per-Input Confidence Variability: Answer-independent confidence varies more than answer-conditioned confidence, whose narrower distribution indicates reduced noise after the answer is fixed.The answer-conditioned distribution still has non-trivial spread rather than collapsing near zero.

I COMPARISON TO RL-CALIBRATION BASELINES

RLCR outperforms calibration-oriented baselines while preserving accuracy, whereas calibration-only and abstention-based objectives introduce distinct weaknesses. A prompt adjustment also substantially recovers SFT+RLCR’s out-of-distribution accuracy degradation.

  • Overall Baseline Comparison: All calibration baselines outperform vanilla RLVR on calibration but underperform RLCR on both accuracy and calibration.The comparison initializes calibration-focused baselines from an accuracy-trained RLVR model for a fairer reasoning-setting evaluation.
  • Calibration RL: Calibration RL with full-sequence rewards collapses to near-zero accuracy by producing empty or trivial answers with confidence 0.Calibration-only optimization can achieve perfect Brier calibration through a degenerate reward-maximizing policy.
  • Calibration RL: Restricting calibration rewards to analysis and confidence prevents collapse and preserves accuracy comparable to RLCR, but calibration remains weaker.The authors hypothesize that joint accuracy-calibration optimization supplies complementary gradient signals absent from this baseline.
  • Abstention-RL: Abstention-RL underperforms RLCR because thresholded abstention rewards do not teach fine-grained confidence estimation and may suppress exploration.This method rewards whether confidence exceeds λ rather than estimating confidence continuously.
  • SFT+RLCR Analysis: 43.8% out-of-distribution accuracy rises to 49.8% after a prompt clarification for SFT+RLCR, implicating formatting biases as the main reversible failure source.The authors still hypothesize that KL regularization or more diverse RL data could mitigate remaining effects.

K HOTPOT TRAINING RESULTS

During HotpotQA training, RLCR’s correctness and calibration rewards improve together, while completion lengths gradually increase as uncertainty reasoning improves.

  • Training Curves: Both correctness and calibration rewards increase smoothly during RLCR training, indicating joint improvement in accuracy and calibration.The corresponding Brier reward is shifted upward by 1 in the figure for visual clarity.
  • Completion Lengths: Completion lengths gradually increase during RLCR training as uncertainty reasoning improves.The figure contrasts this length progression with the training trajectories of RLCR and RLVR.

L FULL RESULTS: MODELS TRAINED ON HOTPOTQA

Across full evaluations of models trained on HotpotQA, RLCR generally outperforms baselines on average while maintaining accuracy, but out-of-distribution calibration remains variable and incomplete.

  • Dataset-Specific Results: RLVR’s uniformly inflated 85–100% confidence can appear calibrated on high-accuracy CommonsenseQA, creating a spurious alignment rather than genuine uncertainty modeling.All methods achieve roughly 90% accuracy on CommonsenseQA, making overconfidence deceptively favorable there.
  • Limitations: All methods, including RLCR, have significant room for improvement in out-of-distribution calibration.The authors believe longer RL training and more diverse training data could strengthen robustness and calibration.
  • Reported Evaluations: Tables 8–11 report performance on SimpleQA, Trivia, CommonsenseQA, GPQA, Math-500, GSM8K, HotpotQA, and HotpotQA-Modified, with bootstrap confidence-interval error margins.Best values are bolded in the dataset-specific tables.

M FULL RESULTS: MODELS TRAINED ON MATH

The reported results cover multiple question-answering datasets, including MATH-500, GSM8K, SimpleQA, TriviaQA, CommonsenseQA, GPQA, Big-Math Digits, and HotpotQA. The examples also illustrate how confidence and reasoning quality vary across model outputs and solution interpretations.

  • Benchmark coverage: Performance is reported on MATH-500, GSM8K, SimpleQA, TriviaQA, CommonsenseQA, GPQA, Big-Math Digits, and HotpotQA.The tables report means with error margins given as half-widths of 95% bootstrap confidence intervals, with best values bolded.
  • Confidence examples: A model can produce an incorrect answer with high confidence, such as answering Lisa Levin with confidence 0.95 when the ground truth is Annick Bricaud.The example pairs the incorrect answer with a numerical confidence estimate.
  • Confidence examples: More cautious reasoning can acknowledge uncertainty when supporting information does not establish the answer, as in the David W. Schiedek example with confidence 0.25.The analysis explicitly questions the answer’s accuracy and notes that the relevant information was unavailable.
  • Model comparison: The SFT+RLCR model is described as producing more specific uncertainty-aware reasoning than vanilla RLCR and outperforming it on Math calibration.The comparison attributes the difference to more effective uncertainty-aware chain-of-thought reasoning.
  • Reasoning reliability: One movie-counting solution reports 76 after adding back two Bella-and-Chris movies, while another review says that addition is redundant and the derivation is unclear.The passages present both the claimed total and a critique of the overlap accounting.
Loading 2507.16806v2…