PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards

Zhengzhao Ma, Xueru Wen, Boxi Cao, Yaojie Lu, Hongyu Lin, Jinglin Yang, Min He, Xianpei Han, Le Sun

arXiv:2603.09117v3cs.LGcs.AIcs.CL

TL;DR

RLVR improves LLM reasoning but causes severe over-confidence in incorrect answers, reflecting a conflict between accuracy and calibration optimization. DCPO decouples these objectives and improves calibration while preserving reasoning performance.

  • Problem

    RLVR causes severe calibration degeneration, with models becoming over-confident in incorrect answers and limiting practical LLM applicability.

  • Method

    DCPO decouples reasoning-quality and confidence-estimation optimization using distinct rewards and advantages with group-based supervision.

  • Results

    DCPO achieves the best tradeoff between reasoning performance and calibration compared with previous strong baselines.

  • Takeaways & Limitations

    The findings highlight decoupled optimization as important for improving model calibration while preserving reasoning performance.

  • Takeaways & Limitations

    The analysis assumes that confidence and model accuracy are positively related.

Abstract

from arXiv · show

Reinforcement Learning from Verifiable Rewards (RLVR) significantly enhances large language models (LLMs) reasoning but severely suffers from calibration degeneration, where models become excessively over-confident in incorrect answers. Previous studies devote to directly incorporating calibration objective into existing optimization target. However, our theoretical analysis demonstrates that there exists a fundamental gradient conflict between the optimization for maximizing policy accuracy and minimizing calibration error. Building on this insight, we propose DCPO, a simple yet effective framework that systematically decouples reasoning and calibration objectives. Extensive experiments demonstrate that our DCPO not only preserves accuracy on par with GRPO but also achieves the best calibration performance and substantially mitigates the over-confidence issue. Our study provides valuable insights and practical solution for more reliable LLM deployment.

1. Introduction

RLVR advances LLM reasoning but causes severe calibration degeneration, especially over-confidence in incorrect answers. DCPO addresses the identified accuracy–calibration gradient conflict by decoupling the two objectives and achieves a strong reasoning–calibration tradeoff.

  • Motivation: GRPO and related RLVR methods have advanced LLM performance in mathematical reasoning, code generation, and question answering.These methods optimize policies using automatically verifiable rewards with on-policy sampling.
  • Problem: RLVR often causes calibration degeneration, with models assigning extremely high confidence to incorrect answers.This over-confidence limits practical applicability in high-stakes domains such as healthcare, law, and finance.
  • Theoretical insight: For over-confident RLVR models, accuracy-maximizing and calibration-error-minimizing gradients are negatively aligned under the Fisher metric.This gradient conflict explains the accuracy–calibration tradeoff.
  • Method: DCPO systematically decouples reasoning accuracy and calibration objectives across generation structure, reward design, and gradient optimization.It trains confidence prediction with instance-level and rollout-group-level accuracy signals without additional annotation or an external oracle.
  • Evaluation: Experiments on mathematical reasoning and code generation show that DCPO achieves the best tradeoff between reasoning performance and calibration compared with previous strong baselines.Evaluation covers five mathematical-reasoning benchmarks and three code-generation benchmarks.

2. Preliminaries and Related Work

This section introduces GRPO as a widely used outcome-based RLVR algorithm and reviews confidence, calibration metrics, and calibration-optimization approaches for LLMs.

  • GRPO: GRPO normalizes scalar rewards within sampled response groups to compute group-relative advantages for policy optimization.Group normalization reduces reward-scale sensitivity and provides a low-variance signal, supporting GRPO’s adoption in RLVR pipelines.
  • Confidence Measurement: LLM confidence is commonly estimated through token-based generation statistics, verbalized scores, or agreement among multiple sampled outputs.Sequence probability is used for token-based confidence; verbalized confidence is human-interpretable, whereas consistency-based confidence requires additional sampling.
  • Calibration Metrics: Calibration measures alignment between predicted confidence and empirical correctness using metrics including ECE, AUROC, and PCE.PCE restricts ECE to bins where confidence exceeds accuracy, helping characterize over-confidence when ECE may decrease trivially with improving accuracy.
  • Related Work: Calibration optimization methods comprise post-hoc or inference-time approaches and calibration-aware reinforcement learning that integrates uncertainty objectives into policy optimization.Examples include inference-time distractors, external confidence predictors, and CCGSPG, which modifies the GRPO objective.

3. Empirical Analysis for Calibration Degeneration

The analysis finds pervasive, systematic over-confidence in LLMs that RLVR exacerbates, while coupled calibration optimization improves calibration at the cost of reasoning performance.

  • Baseline calibration: Across model families and scales, evaluation on mathematical reasoning benchmarks finds pervasive mis-calibration driven primarily by systematic over-confidence.The study uses a suite of mathematical reasoning benchmarks to quantify calibration behavior.
  • Baseline calibration: ECE consistently exceeds 0.3 across evaluated base models, indicating severe deviation from ideal calibration.Reliability diagrams show empirical accuracy substantially below the diagonal across most confidence bins.
  • RLVR-induced degeneration: GRPO increases average confidence from about 0.88 to above 0.98 during RLVR training while reducing confidence variance from 0.006 to about 0.001.The experiment trains Qwen3-8B on DeepScaler and evaluates calibration on AIME24 using 8 repeated samplings.
  • RLVR-induced degeneration: RLVR improves reasoning accuracy while degrading calibration by producing excessively confident predictions, including incorrect answers.These findings motivate calibration-aware strategies that explicitly control confidence during reinforcement learning.
  • Coupled optimization: RLCR and CCGSPG substantially reduce ECE and PCE relative to GRPO but both cause a noticeable drop in reasoning accuracy.The comparison on AIME24 indicates that directly coupling calibration objectives with correctness-driven optimization introduces gradient interference.

4. Theoretical Analysis

Theoretical analysis identifies trajectory-level mode collapse as a structural source of RLVR over-confidence and shows that accuracy and calibration optimization can have conflicting gradients. It further establishes group-level rollout accuracy as an unbiased, low-variance calibration signal.

  • Finding 1: Trajectory-level RLVR structurally induces over-confidence through mode collapse toward a single correct trajectory.Without explicit entropy regularization, any accuracy-optimal solution assigns probability mass 1 to one trajectory y* ∈ Y+.
  • Finding 1: Extreme logit margins remain stable under small input perturbations, producing over-confident but incorrect predictions under distribution shift.Policy-logit continuity creates neighborhoods around training inputs, while correctness is discontinuous with respect to the input.
  • Finding 2: The gradient direction for maximizing accuracy is negatively aligned with the direction for minimizing calibration error when the model is overconfident.Proposition 4.2 states this conflict under Confθ(x) > E_y∼πθ[R(y)], with a strictly negative Fisher-metric inner product.
  • Finding 3: Average correctness within a rollout group provides a more stable estimate of model uncertainty for a given input.The group-level estimator is unbiased for E[R(y)] and has variance O(1/G).
  • Finding 3: Under absolute calibration loss, group-level correctness supervision reduces gradient variance compared with instance-level correctness.The reduction follows from Proposition 4.4, with equality approached only when p ∈ {0, 1}.

5. Decoupled Calibration Policy Optimization

DCPO decouples reasoning quality from confidence estimation by assigning separate rewards, advantages, and masked token-level updates to reasoning and confidence blocks. Its design uses GRPO group sampling for stable calibration supervision, while Theorem 5.1 establishes statistically consistent uncertainty estimates without interfering with policy optimization.

  • Decoupled Calibration Policy Optimization: DCPO separates reasoning and confidence optimization by assigning distinct rewards and advantages to different output segments.The framework is designed to address the accuracy-calibration tradeoff induced by RLVR.
  • Decoupled Calibration Policy Optimization: The model generates a reasoning block containing the reasoning process and final answer, followed by a scalar confidence prediction separated by <conf>.This block-wise verbalized confidence rollout explicitly models uncertainty during reinforcement learning.
  • Decoupled Calibration Policy Optimization: DCPO interpolates group-level and instance-level accuracy as calibration targets, with λ ∈ [0, 1] controlling variance reduction versus sample-level discrimination.Group sampling from GRPO provides the calibration supervision, while confidence rewards and standardized advantages are defined from these targets.
  • Decoupled Calibration Policy Optimization: Masked optimization applies reasoning and confidence advantages exclusively to their corresponding token subsets, preventing correctness gradients from affecting confidence estimation and vice versa.A token-level mask separates reasoning tokens from confidence tokens under a shared policy.
  • Decoupled Calibration Policy Optimization: Theorem 5.1 establishes that decoupled confidence estimation yields statistically consistent uncertainty estimates without interfering with policy optimization.The result follows from independently optimizing confidence estimation and reasoning policy components.

6. Experiments

Experiments across mathematical reasoning and code generation show that DCPO preserves reasoning accuracy while improving calibration and reducing over-confidence. Ablations and training-dynamics analyses identify decoupled optimization, hybrid group-instance supervision, and on-policy calibration as critical to this tradeoff.

  • Experimental Setup: Experiments cover mathematical reasoning and code generation, evaluating accuracy alongside ECE, AUROC, and PCE.Math models use DeepScalar and benchmarks including MATH-500, AIME 2024/2025, and AMC 2023/2024; code models use PrimeIntellect-verifier and LiveCodeBench v5/v6 and HumanEval+.
  • Main Results: DCPO preserves accuracy while improving calibration across math and code tasks, achieving 41.6% accuracy and reducing AIME24 PCE from 0.505 to 0.212.On code benchmarks, DCPO reaches average accuracy 0.515 versus GRPO’s 0.514.
  • Main Results: Post-hoc calibration yields only marginal ECE improvement from 0.370 to 0.363 on AIME24, while AUROC remains 0.642 versus DCPO’s 0.914.The results indicate that post-hoc methods are insufficient to substantially correct RLVR-induced mis-calibration.
  • Ablation Studies: Removing decoupled optimization increases ECE from 0.128 to 0.258 and decreases accuracy from 60.8% to 57.3%, demonstrating gradient interference.The ablation concludes that decoupling stabilizes training and prevents calibration objectives from harming reasoning performance.
  • Ablation Studies: Hybrid supervision and on-policy calibration are necessary: removing instance-level labels raises ECE to 0.209, removing group-level labels lowers accuracy to 58.7%, and off-policy calibration lowers accuracy to 56.3%.The reported full-model reference values are ECE 0.128 and accuracy 60.8%.
  • Optimization and Confidence Dynamics: DCPO produces smoother gradient-norm trajectories and a balanced, continuous confidence distribution, unlike the fluctuations and skewed or extreme confidence patterns of comparison methods.Figure 7 compares GRPO, RLCR, DCPO-I, and DCPO; Figure 8 visualizes confidence distributions across five mathematical benchmarks.

7. Conclusion · A. Detailed Proofs for Theoretical Analysis

The paper shows that RLVR inherently induces over-confidence and creates a gradient conflict between accuracy and calibration optimization. It proposes DCPO, which improves calibration while preserving reasoning performance through decoupled optimization.

  • 7. Conclusion: RLVR inherently induces over-confidence in language models.
  • 7. Conclusion: The analysis identifies a gradient conflict between optimizing accuracy and calibration.
  • 7. Conclusion: DCPO is a decoupled confidence-aware policy optimization framework.
  • 7. Conclusion: DCPO significantly improves calibration while preserving reasoning performance.
  • 7. Conclusion: The study highlights decoupled optimization as important for model calibration.
  • A. Detailed Proofs for Theoretical Analysis: The provided passage reports the theoretical analysis and conclusions but does not state additional details from the detailed proofs.

A.1. Proof of Proposition 1 (Mode Collapse under Trajectory-Level RL) · A.2. Proof of Proposition 2 (Gradient Conflict between Correctness and Calibration) · A.3. Proof of Theorem 1 (Optimality of Decoupled Confidence Estimation)

The proofs establish that trajectory-level RL can collapse onto a single correct trajectory, creating stable low-entropy and potentially over-confident incorrect predictions under distribution shift. They further show a correctness–calibration gradient conflict and prove that decoupled confidence estimation is statistically consistent without changing generation-policy optimality.

  • A.1. Proof of Proposition 1 (Mode Collapse under Trajectory-Level RL): Without regularization, an optimal trajectory distribution can assign probability mass 1 to a single correct trajectory.Any distribution supported entirely on correct trajectories satisfies the KKT conditions, while extreme-point optima are Dirac measures.
  • A.1. Proof of Proposition 1 (Mode Collapse under Trajectory-Level RL): The mode-collapse result holds regardless of the size or structure of the correct-trajectory set and extends to bounded linearity reward objectives.
  • A.1. Proof of Proposition 1 (Mode Collapse under Trajectory-Level RL): Extreme logit margins make low-entropy solutions stable under small input perturbations, producing over-confident but incorrect predictions under distribution shift.Policy logits are continuous in the input, whereas correctness is not.
  • A.2. Proof of Proposition 2 (Gradient Conflict between Correctness and Calibration): Under the proposition’s over-confidence condition, confidence exceeds accuracy: Confθ(x) > Jacc(θ).
  • A.2. Proof of Proposition 2 (Gradient Conflict between Correctness and Calibration): The Fisher-metric inner product between the natural-gradient directions for correctness and confidence is strictly positive whenever ∇θConfθ(x) ≠ 0.The proof derives this positivity from the policy-gradient expressions and the stated assumptions.
  • A.3. Proof of Theorem 1 (Optimality of Decoupled Confidence Estimation): For a strictly proper scoring rule, the expected confidence-estimation loss is uniquely minimized at the true correctness probability p∗.The correctness reward is modeled as a Bernoulli distribution, and proper calibration estimation reaches its minimum at c = r.
  • A.3. Proof of Theorem 1 (Optimality of Decoupled Confidence Estimation): Because confidence estimation is optimized independently of generation, E[c|x] converges to p∗ without explicitly affecting πθ(y|x).
  • A.3. Proof of Theorem 1 (Optimality of Decoupled Confidence Estimation): Decoupled confidence estimation therefore provides statistically consistent calibration without altering the generation policy’s optimality conditions.

A.4. Proof of Proposition 3 (Unbiasedness and Variance of Group-Level Accuracy) … B.2. Confidence Estimation Baseline

The appendix proves that group-level accuracy is an unbiased estimator with variance O(1/G), and that it reduces calibration-gradient variance relative to instance-level supervision. Training uses specified VERL, hardware, confidence-supervision, post-training, and ConfClass procedures.

  • A.4. Proof of Proposition 3 (Unbiasedness and Variance of Group-Level Accuracy): Group-level accuracy ˜RG is an unbiased estimator of E[R(y)] with variance O(1/G).The variance therefore decays as the group size G increases.
  • A.4. Proof of Proposition 3 (Unbiasedness and Variance of Group-Level Accuracy): Using ˜RG as supervision with absolute calibration loss reduces gradient variance compared with instance-level correctness.Under group-level supervision, the subgradient is deterministic away from c = p, giving Var(g(c, ˜RG)) = 0.
  • A.4. Proof of Proposition 3 (Unbiasedness and Variance of Group-Level Accuracy): The instance-level subgradient is random because R(y) follows a Bernoulli distribution, whereas group-level supervision uses the group accuracy ˜RG = p.The comparison is strict except when p ∈ {0, 1}, where every group instance is correct or incorrect.
  • B. Training and Evaluation Details: Reinforcement learning experiments use VERL on a cluster of 8 NVIDIA A100 (80GB) GPUs.Table 4 summarizes the key hyperparameters used throughout the experiments.
  • B.1. Training Setup: GRPO and CCGSPG train on the original DeepScaler dataset, while RLCR and DCPO prompts request a confidence score after the final answer.DCPO uses tradeoff hyperparameter λ = 0.5.
  • B.1. Training Setup: Post-training calibration methods start from the GRPO checkpoint at step 120 and train for 80 additional steps using only Brier Score calibration rewards.This procedure applies to the post-training calibration methods described in the training setup.
  • B.2. Confidence Estimation Baseline: ConfClass predicts confidence with a three-layer MLP of hidden dimension 2048, trained on outputs from GRPO-120 over 20 independent runs.Top-20 token log-probabilities are inputs, group-level average accuracy is supervision, and training uses cross-entropy loss with learning rate 1 × 10−4.

B.3. Evaluation Protocol … C.2. Sensitivity to Structured Prompts

The evaluation protocol defines two confidence extraction methods, standardized sampling and calibration metrics, while additional experiments assess DCPO’s generalization and robustness to structured confidence prompts. DCPO consistently improves calibration, rapidly learns the required output format, and maintains stable formatting during training.

  • B.3. Evaluation Protocol: Responses are sampled with temperature 0.7, top-p 0.8, and top-k 20, with four independent repetitions averaged on smaller evaluation sets.Task accuracy uses OpenCompass; ECE and BS use confidence–accuracy pairs, while AUROC uses scikit-learn.
  • B.4. Verbalized Confidence Prompts: Training and evaluation prompts require the final answer in \boxed{} and a single-line confidence score between 0 and 1 after <conf>.The example format includes “<conf> Confidence: 0.83”.
  • C. Additional Experiments: Additional experiments evaluate DCPO on Llama3.1-8B-Instruct and non-thinking Qwen3-14B using the DeepScaler dataset and the main-paper settings.These experiments extend evaluation across model families and parameter scales.
  • C.1. Generalization Across Model Families and Scales: Across model families and scales, DCPO consistently improves calibration while maintaining competitive or improved accuracy.The results are presented as evidence of robustness and generalization across architectures and parameter scales.
  • C.2. Sensitivity to Structured Prompts: DCPO uses a structural penalty of −1.0, substantially larger than the calibration reward magnitude, to strongly discourage malformed confidence outputs.The penalty is designed to enforce the structured output requirement without causing instability.
  • C.2. Sensitivity to Structured Prompts: 1.60% format violations at initialization decrease to approximately 0.2% after a few training steps, indicating rapid learning and stable maintenance of the required output structure.The observed trend supports robustness to structured prompt requirements during training.

C.3. Hyperparameter λ Sensitivity · C.4. Impact on Output Length

DCPO’s λ controls the balance between group-level and instance-level calibration, with λ = 0.5 offering the strongest overall tradeoff among accuracy, calibration, and training stability. Verbalized confidence adds negligible generation overhead because DCPO responses remain similar to or slightly shorter than GRPO outputs.

  • C.3. Hyperparameter λ Sensitivity: C.3. Hyperparameter λ Sensitivity: DCPO uses λ to combine group-level and instance-level calibration objectives.The main comparisons evaluate DCPO with λ = 0.5, DCPO-I with λ = 0, and DCPO-G with λ = 1.0.
  • C.3. Hyperparameter λ Sensitivity: C.3. Hyperparameter λ Sensitivity: λ values between 0.25 and 0.5 achieve the best calibration.
  • C.3. Hyperparameter λ Sensitivity: C.3. Hyperparameter λ Sensitivity: Larger λ values reduce gradient variances and improve training stability but degrade calibration.
  • C.3. Hyperparameter λ Sensitivity: C.3. Hyperparameter λ Sensitivity: λ = 0.5 provides a robust tradeoff among accuracy, calibration, and training stability.
  • C.4. Impact on Output Length: C.4. Impact on Output Length: Both GRPO and DCPO increase generation length relative to the base model under reinforcement learning optimization.
  • C.4. Impact on Output Length: C.4. Impact on Output Length: DCPO produces responses with similar or slightly shorter length than GRPO.
  • C.4. Impact on Output Length: C.4. Impact on Output Length: Verbalized confidence introduces negligible overhead while preserving generation efficiency comparable to standard reinforcement learning.The analysis measures average output length for models trained with GRPO and DCPO.
Loading 2603.09117v3…