PaperlayerView exact revision on arXiv

Source-linked AI summary

RL-ARC: Calibrating Large Reasoning Models via Reasoning-guided Uncertainty

Gukhyeon Lee, SangKeun Lee

arXiv:2610.11352v1cs.AIcs.CLcs.LG

TL;DR

RLVR can degrade calibration, and calibration-aware methods can retain OOD overconfidence while sacrificing reasoning performance. RL-ARC uses reasoning confidence alongside answer confidence to calibrate predictions, and improves calibration over existing reasoning-training approaches without substantially sacrificing accuracy.

  • Problem

    RLVR can reinforce overconfidence, while calibration-aware methods may retain OOD overconfidence and degrade reasoning performance.

    Source: §1 · pp. 1–2

  • Method

    RL-ARC jointly uses reasoning and answer confidence, modulating answer-confidence calibration according to the assessed reliability of the reasoning process.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    RL-ARC consistently improves calibration over existing reasoning-training approaches across ID and OOD benchmarks without substantially sacrificing accuracy.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results highlight reasoning confidence as an important signal for calibration-aware training of reasoning models.

    Source: §1 · p. 2, §5 · p. 9

    Experiments are limited to two training settings using limited reasoning samples, and the study mainly focuses on LMs, particularly the Qwen family.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Language models (LMs) are commonly trained with Reinforcement Learning with Verifiable Rewards (RLVR) to enhance their reasoning capabilities. However, since RLVR does not explicitly account for calibration during training, it can lead to severe calibration degradation, including overconfidence. Recent calibration-aware training methods for LMs, which incorporate objectives for uncertainty estimation into training, improve calibration but still exhibit overconfidence under distribution shift, while sacrificing reasoning performance. To this end, we propose RL-ARC, a calibration-aware training framework that jointly leverages reasoning confidence and answer confidence. Specifically, RL-ARC leverages reasoning confidence as an auxiliary signal for calibrating answer confidence, applying it as reasoning-guided regularization for correct cases and as an overconfidence penalty for incorrect cases. Comprehensive results across ID and OOD settings show that, beyond improving calibration, RL-ARC enables reasoning models to adaptively estimate confidence based on the given question without substantially sacrificing reasoning performance, thereby highlighting the importance of reasoning confidence for training reliable reasoning models.

1 Introduction

RLVR and prior calibration-aware training leave reasoning models vulnerable to overconfidence, particularly under distribution shift. RL-ARC uses reasoning confidence to calibrate answer confidence and improves calibration while limiting accuracy loss.

1 Introduction

  • RLVR reinforces overconfident reasoning, while prior calibration objectives can leave flawed reasoning and OOD overconfidence unresolved and degrade reasoning performance.These limitations matter for reliability in high-stakes settings where prediction reliability is essential.Source: §1 · pp. 1–2
  • RL-ARC uses reasoning confidence to modulate answer-confidence calibration, regularizing correct predictions and penalizing overconfidence in incorrect predictions.The framework jointly leverages reasoning and answer confidence.Source: §1 · p. 2
  • RL-ARC consistently outperforms existing calibration-aware methods in calibration while substantially mitigating accuracy degradation across ID and OOD benchmarks.The evaluation includes mathematical reasoning benchmarks and OOD complex-reasoning and factual-QA tasks.Source: Figure 1 · p. 1, §1 · p. 2

2 Related Work

Prior uncertainty-estimation work includes sample-based, probability-based, and verbalized-confidence approaches, each with stated limitations. Calibration-aware training addresses answer confidence but can retain OOD overconfidence and an accuracy-calibration trade-off; RL-ARC also accounts for reasoning-process correctness.

2.1 Uncertainty Estimation for Language Models

  • Uncertainty estimation approaches use multiple generated samples, token-probability distributions, or verbalized confidence, with respective costs or applicability limits.Verbalized confidence is simple and human-interpretable, but can still be overconfident and miss its own mistakes.Source: §2.1 · p. 2

2.2 Training for Calibration

  • Calibration-aware training aligns confidence with prediction correctness, but remains prone to OOD overconfidence and an accuracy-calibration trade-off.These methods incorporate uncertainty-estimation objectives into reasoning training.Source: §2.2 · pp. 2–3
  • RL-ARC considers reasoning-process correctness alongside prediction correctness, improving calibration across ID and OOD settings while maintaining competitive accuracy.Its overview distinguishes treatment of correct predictions from incorrect ones.Source: §2.2 · p. 3, Figure 2 · p. 3

3 RL-ARC: Reinforcement Learning with Auxiliary Reasoning Confidence

RL-ARC combines reasoning and answer confidence in calibration-aware reinforcement learning. Its objective uses reasoning confidence to guide answer confidence differently for correct and incorrect predictions.

3 RL-ARC: Reinforcement Learning with Auxiliary Reasoning Confidence

  • The section introduces RL-ARC as calibration-aware reinforcement learning intended to improve calibration without substantially sacrificing accuracy.It frames the method’s goal as calibrated outputs in both ID and OOD settings while mitigating the accuracy-calibration trade-off.Source: §3 · p. 3, §3.2 · p. 4

3.1 Preliminaries

  • RLVR rewards prediction correctness, while the Brier Score measures squared error between confidence and correctness; their combination rewards accurate, appropriately confident predictions.The resulting total reward subtracts the Brier score from the RLVR reward.Source: §3.1 · pp. 3–4

3.2 Reasoning-aware Calibration

  • RL-ARC elicits self-assessed confidence in both the reasoning process and prediction, then uses both in calibration-aware reinforcement learning.Reasoning confidence reflects whether the reasoning is correct or truthful; answer confidence reflects whether the prediction is correct.Source: §3.2 · p. 4
  • RL-ARC aligns answer confidence with reasoning confidence for correct predictions and penalizes high answer confidence for incorrect predictions.For correct predictions, it minimizes misalignment between the two confidences; for incorrect ones, reasoning confidence adds a penalty to answer confidence.Source: §3.2 · p. 4

4 Experiments

The experiments compare RL-ARC with uncertainty-estimation and calibration baselines across in-distribution and out-of-distribution benchmarks. They assess calibration, accuracy, question-adaptive confidence, and reasoning reliability, finding that RL-ARC improves calibration while largely preserving reasoning performance.

4 Experiments

  • The experiments ask whether RL-ARC improves calibration while mitigating the accuracy-calibration trade-off, adapts confidence to each question, and encourages more reliable reasoning.The stated research questions cover both ID and OOD settings.Source: §4 · p. 4

4.1 Experimental Setups

  • The experiments use arithmetic benchmarks similar to training and OOD reasoning and factual QA tasks, compare six confidence and calibration baselines, and evaluate accuracy, calibration, and reasoning reliability.The main training dataset is Big-Math; experiments also use HotpotQA as an alternative training distribution and evaluate Qwen and Llama backbones.Source: §4.1 · pp. 5–6

4.2 Main Results

  • Across ID and OOD benchmarks, RL-ARC improves calibration over calibration-aware methods while largely preserving performance gains from training.The comparison covers Math and OOD benchmarks; the supplied passages report no specific table values.Source: Table 1 · p. 5, §4.2 · p. 6

4.3 General Applicability to Architectures and Training Distributions

  • Across Qwen- and Llama-based models and different training distributions, RL-ARC improves calibration while mitigating accuracy degradation, including on OOD benchmarks.The Llama-8B model still exhibits OOD accuracy degradation, though RL-ARC mitigates it.Source: §4.3 · p. 6, Table 2 · p. 6, Table 3 · p. 7

4.4 Reliability on Out-of Distribution (OOD)

  • Across complex reasoning and factual QA benchmarks, RL-ARC improves OOD calibration without substantial accuracy degradation; it achieves the best accuracy and calibration on factual QA.The factual QA benchmarks are designed to evaluate overconfidence on obscure factual knowledge.Source: §4.4 · p. 7, Table 4 · p. 7

4.5 Overconfidence Mitigation

  • On unseen tasks, RL-ARC aligns confidence better with correctness and assigns more samples to low- and mid-confidence bins; its selective risk is lower or competitive, especially at low coverage.At low coverage, retained high-confidence predictions are more likely to be correct, and RL-ARC more effectively assigns lower confidence to incorrect predictions.Source: §4.4 · p. 7, §4.5 · pp. 7–8, Figure 3 · p. 8

4.6 Component Analysis in RL-ARC

  • The overconfidence penalty improves Brier Score and ECE but can sacrifice accuracy, AUROC, and AURC, while reasoning-guided regularization supports accuracy, AUROC, and AURC.Combining the objectives and appropriately balancing their scales improves calibration without substantially sacrificing accuracy; increasing reasoning-confidence influence does not necessarily improve calibration further.Source: §4.6 · pp. 8–9, Table 5 · p. 8, Figure 5 · p. 9

4.7 Reliability of Reasoning

  • RL-ARC improves answer and reasoning calibration and achieves the largest reduction in shallow reasoning, without direct supervision for reasoning correctness.The reported reasoning-calibration gain is substantially larger than that achieved by the calibration-aware training method.Source: §4.7 · p. 9, Figure 6 · p. 9

5 Conclusion

RL-ARC uses confidence in reasoning to calibrate answer confidence, improving calibration across ID and OOD settings while maintaining accuracy comparable to correctness-focused training. Its analysis indicates that models can adapt confidence to the question.

5 Conclusion

  • RL-ARC uses verbalized confidence from both reasoning and prediction, with reasoning confidence serving as an auxiliary signal for calibrating answer confidence.The framework is evaluated on ID arithmetic tasks and OOD complex-reasoning and factual-QA tasks.Source: §5 · p. 9
  • Across ID and OOD settings, RL-ARC improves calibration over existing reasoning-training approaches while achieving accuracy comparable to correctness-only methods.Evaluations include arithmetic reasoning, complex reasoning, and factual question answering.Source: §5 · p. 9
  • Analysis indicates that RL-ARC enables models to adapt confidence to the given question, highlighting reasoning confidence's importance in calibration-aware training.Source: §5 · p. 9

Limitations

The experiments cover two training settings and use GRPO throughout; the authors also identify limits in model scope and reasoning-correctness evaluation. They propose broader training data, stronger RL algorithms, and more accurate evaluation methods as future directions.

Limitations

  • Experiments are limited to two training settings, using either a limited set of mathematical-reasoning samples or complex-reasoning samples.Source: Limitations · p. 10
  • All models are trained with GRPO to ensure a fair comparison with baselines.Source: Limitations · p. 10
  • The work focuses mainly on LMs, particularly the Qwen family, leaving multimodal reasoning calibration for future work.Source: Limitations · p. 10
  • LLM-as-a-judge can err when assessing reasoning correctness, and more accurate evaluation methods are left for future work.Source: Limitations · p. 10

A.1 Dataset Details

The evaluation spans mathematical reasoning, complex reasoning, and factual question answering, with Big-Math also used for training. Dataset descriptions specify evaluation sizes and assessment methods.

A.1 Dataset Details

  • Big-Math contains 250,000 problems, of which 30,000 appropriately difficult problems are used for training; evaluation uses 1,000 problems.Predictions are verified with math-verify.Source: §A.1 · p. 13
  • GSM8K (1,319), MATH-500 (500), AMC237 (40), AIME24 (30), and AIME25 (30) evaluate mathematical reasoning using math-verify.Source: §A.1 · p. 13
  • StrategyQA (229) and GPQA (448) use LLM-as-a-judge, while HotpotQA uses 1,000 questions assessed by exact match.These datasets evaluate complex reasoning.Source: §A.1 · p. 13
  • SimpleQA (4,326) and Natural Questions (3,610) use LLM-as-a-judge; TriviaQA uses 2,000 closed-book questions assessed by LLM-as-a-judge.Source: §A.1 · p. 13

A.2 Implementation

Training uses filtered Big-Math data and common hyperparameters across methods. RL-ARC's default positive and negative coefficients are 0.4 and 0.15.

A.2 Implementation

  • Training uses Big-Math, excluding questions LLaMA-8B solves easily and retaining only questions with numerical answers.Source: §A.2 · p. 13
  • All methods use the same hyperparameters, with training on four NVIDIA H100 or A100 GPUs.Source: §A.2 · p. 14
  • RL-ARC's default λpos and λneg values are 0.4 and 0.15, respectively.Detailed settings are listed in Table 6.Source: §A.2 · p. 14

A.3 Metrics Details

The section defines calibration, confidence-ranking, selective-prediction, and reasoning-quality metrics, then describes a prompt template that elicits confidence in both reasoning and answers.

A.3 Metrics Details

  • Brier Score assesses probabilistic predictions through the squared difference between verbalized confidence and correctness.The prediction, ground-truth answer, correctness indicator, and self-assessed answer confidence define the metric.Source: §A.3 · p. 14
  • ECE measures calibration by comparing average accuracy and confidence within bins, using 10 bins in this work.The ECE calculation weights each bin's accuracy-confidence gap by its share of the samples.Source: §A.3 · p. 14
  • AUROC measures how well confidence distinguishes correct from incorrect predictions, while AURC measures risk as coverage varies.For AURC, samples are sorted by confidence and risk is computed over the selected top-coverage subsets.Source: §A.3 · p. 14
  • Reasoning Calibration measures the gap between reasoning confidence and judged reasoning correctness; Shallow Reasoning measures correct predictions supported by incorrect reasoning on OOD benchmarks.Reasoning correctness is assessed using an LLM-as-a-judge, and Shallow Reasoning is computed as a proportion of cases.Source: §A.3 · p. 14

A.4 Calibration-aware Reasoning Training Prompt Template

  • The calibration-aware reasoning-training prompt elicits confidence in both the reasoning process and the prediction.The detailed template is shown in Figure 9.Source: §A.4 · p. 14

B.1 Overall Results for Each Task

Overall benchmark results are reported for several model and task families; the confidence-pattern analysis contrasts ID and OOD behavior across RL-ARC, RLCR, and RLVR.

B.2 Confidence-aware Patterns

  • In OOD settings, RL-ARC distributes predictions across broader confidence levels and assigns lower or mid confidence to uncertain predictions, unlike RLVR and RLCR's concentration at high confidence.In ID settings, RL-ARC and RLCR show comparable confidence-allocation patterns, and both improve confidence behavior over standard RLVR.Source: §B.2 · pp. 15–16, Figure 7 · p. 15, Figure 8 · p. 15

B.3 Additional Analysis of Calibration Gains

The analyses report improved selective reliability, calibration gains beyond confidence-distribution flattening, and an integration strategy whose effects vary by correctness and auxiliary-signal scale.

B.3 Additional Analysis of Calibration Gains

  • RL-ARC has lower ECE and Brier than confidence-shuffled controls with the same confidence histogram and bin occupancy across ID and OOD benchmarks.The control indicates that the calibration gains are not attributable solely to a broader or flatter confidence distribution.Source: §B.3 · p. 16

B.4 Quantitative Reliability

  • RL-ARC achieves the lowest AURC in ID and OOD settings, improves AURC and ECE over RLCR while maintaining competitive ID accuracy, and achieves the best OOD accuracy.The reported comparison spans Math (ID) and OOD benchmarks.Source: Table 8 · p. 16, §B.4 · p. 16

B.5 Performance Sensitivity to Reasoning Confidence Integration

  • Compared with arithmetic-mean and geometric-mean integration, RL-ARC's distinct objectives for correct and incorrect cases improve calibration while mitigating the accuracy-calibration trade-off.The strategy uses reasoning-guided regularization for correct cases and an additional overconfidence penalty for incorrect cases.Source: §B.5 · p. 16, Table 9 · p. 17

B.6 Performance Sensitivity to Auxiliary Signal Scales

  • Across auxiliary-signal scales, RL-ARC maintains a better accuracy-calibration trade-off, with optimal configurations predominantly positive-dominant (λpos/λneg > 1).The positive-case regularizer primarily affects accuracy and AUROC, while the negative-case overconfidence penalty mainly improves calibration, particularly in positive-dominant settings.Source: §B.6 · p. 16, Table 10 · p. 17
Loading 2610.11352v1…