PaperlayerView exact revision on arXiv

Source-linked AI summary

PACR: Progressively Ascending Confidence Reward for LLM Reasoning

Eunseop Yoon, Hee Suk Yoon, Jaehyun Jang, SooHwan Eom, Qi Dai, Chong Luo, Mark A. Hasegawa-Johnson, Chang D. Yoo

arXiv:2510.22255v1cs.AIcs.CL

TL;DR

RLVR’s sparse outcome reward leaves intermediate reasoning unsupervised, motivating a denser signal for exploration and credit assignment. PACR derives stepwise rewards from the model’s evolving ground-truth confidence, and experiments and theory support its use for more effective reasoning training. Across multiple benchmarks, PACR improves training dynamics and final performance, with Dense-PACR most effective among the variants reported.

  • Problem

    RLVR provides a sparse terminal accuracy signal with no guidance for intermediate reasoning steps, while external process supervision is costly, data-hungry, and prone to misalignment.

  • Method

    PACR converts positive changes in the model’s ground-truth log-probability into dense stepwise supervision and combines it with terminal RLVR rewards.

  • Results

    Across multiple reasoning benchmarks, augmenting RLVR with PACR accelerates exploration and improves final performance, with Dense-PACR proving most effective.

  • Takeaways & Limitations

    Confidence growth provides an empirically and theoretically supported inductive bias that constrains exploration toward more logically sound and faithful reasoning paths.

  • Takeaways & Limitations

    The study is primarily confined to language models and leaves PACR’s efficacy in multimodal reasoning tasks for future work.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly improved LLM reasoning, but its sparse, outcome-based reward provides no guidance for intermediate steps, slowing exploration. We propose Progressively Ascending Confidence Reward (PACR), a dense, model-intrinsic reward computed directly from the model's evolving belief in the correct answer. PACR encodes the inductive bias that, along a well-formed reasoning trajectory, the probability of the ground-truth answer should have a generally ascending trend. We provide empirical and theoretical analysis validating that such an inductive bias constrains the exploration search space to regions richer in logically sound reasoning. We demonstrate that PACR accelerates exploration, reaches reward saturation with fewer trajectories, and yields improvements on multiple benchmarks. Our results suggest that dense, model-intrinsic shaping signals can make RLVR training more effective and reliable.

1 INTRODUCTION

RLVR improves reasoning but its terminal-only reward leaves intermediate steps unsupervised. PACR uses the model’s rising ground-truth confidence to provide dense process guidance, with empirical and theoretical support for more effective exploration.

  • Motivation: PACR addresses RLVR’s sparse terminal reward, which provides no intermediate guidance and worsens credit assignment.External process supervision can be costly, data-hungry, and misaligned.
  • Method: PACR rewards positive stepwise changes in the model’s log-probability for the ground-truth answer, using the model’s own probabilities without an external reward model.It is paired with the standard RLVR terminal accuracy reward.
  • Empirical validation: Consistent confidence ascent correlates with correctness, distinguishes logically coherent from spurious correct paths, and identifies pivotal reasoning steps.These findings provide empirical support for confidence growth as an inductive bias.
  • Theory: An oracle-policy analysis shows that idealized reasoning steps should increase or maintain ground-truth confidence, motivating PACR as a process reward.The framework includes Sparse-PACR for trajectory-level rewards and Dense-PACR for step-wise rewards.
  • Experimental results: Across multiple reasoning benchmarks, PACR accelerates exploration and achieves higher, more consistent final scores than the baseline.The result supports a more effective and reliable RLVR training process.

2 RELATED WORK

Related work frames RLVR as effective outcome-based training while highlighting the limitations of sparse rewards and external approaches to dense supervision.

  • Outcome-based RL: RLVR uses programmatically verifiable outcomes to improve LLM reasoning, including terminal rewards based on final-answer correctness.The related-work discussion contrasts outcome-based RL with denser supervision methods.
  • Outcome-based RL: RLVR can avoid a separate reward model, a distinction associated with reducing reward-hacking concerns.The passage presents this as a motivation for outcome-based verification.
  • Dense reward: Dense-reward approaches for LLM fine-tuning include external reward models trained or adapted to assign token-level or implicit rewards.These methods address trajectory-level sparsity through additional reward modeling.

3 BACKGROUND AND PROBLEM SETUP

The setup represents reasoning as segmented trajectories, measures ground-truth confidence after each step, and uses verifiable terminal rewards within GRPO-style training.

  • Reasoning trajectories and notation: A policy generates reasoning steps H and a final answer, while H≤k denotes the prefix used to track ground-truth confidence.The setup treats confidence as evolving over the reasoning trajectory.
  • Step segmentation: Reasoning traces are segmented at newlines or period-space boundaries, with fragments shorter than five tokens merged into the preceding step.This model-agnostic rule avoids overly fine-grained splits.
  • Stepwise confidence: At each step, the model’s probability of the standardized ground-truth answer is evaluated to measure confidence during reasoning.A short answer prefix is appended before evaluating the ground-truth answer tokens.
  • GRPO: GRPO samples N trajectories for each question and compares their group-relative returns rather than using a learned value function.Advantages are centered, and optionally standardized, within the sampled cohort.
  • Terminal reward: Each trajectory receives a binary terminal accuracy reward by comparing its predicted answer with the verifiable ground truth.Task-specific normalization supports exact-match evaluation before reward assignment.
  • Optimization objective: The training objective uses PPO-style clipping with a KL penalty, following Dr. GRPO’s removal of standard-deviation and length-normalization terms.Dr. GRPO is presented as a bias-mitigated GRPO variant.

4 IS GROUND-TRUTH CONFIDENCE GROWTH A USEFUL INDUCTIVE BIAS?

Empirical and theoretical analyses support ground-truth confidence growth as an inductive bias: consistent ascent correlates with correctness and coherent reasoning, while oracle-conditioned steps have non-negative expected gain.

  • Empirical observations: Correct trajectories contain a higher proportion of positive confidence-gain steps than incorrect trajectories.This links more consistent confidence ascent with final-answer correctness.
  • Empirical observations: A pivotal Vieta’s-formulas step produces a distinct confidence-gain spike of C7 = +4.83, exceeding routine algebraic steps.The example motivates using gain magnitude to identify important reasoning actions.
  • Empirical observations: Among correct-answer trajectories, coherent reasoning paths show more consistent confidence ascent than spurious paths.The comparison indicates that confidence growth reflects reasoning quality beyond the final outcome.
  • Empirical observations: LLM evaluators judge steps with higher confidence gain as more critical at a win rate significantly above chance.This quantitatively supports a relationship between gain magnitude and step importance.
  • Theoretical justification: Under the oracle policy, the expected stepwise confidence gain is non-negative because it equals a KL divergence.The oracle policy samples faithful steps conditioned on the ground-truth answer, and KL non-negativity supplies the guarantee.
  • Theoretical justification: Encouraging non-decreasing confidence therefore constrains exploration toward regions richer in logically sound reasoning.This implication follows from the theoretical guarantee for faithful reasoning steps.

5 METHOD: PROGRESSIVELY ASCENDING CONFIDENCE REWARD (PACR)

PACR complements GRPO with confidence-growth rewards derived from the model’s probability of the ground-truth answer. Sparse-PACR scores trajectory-level consistency, while Dense-PACR provides discounted, normalized step-wise advantages.

  • PACR adds a procedural reward to GRPO by using ground-truth confidence growth alongside the standard outcome-based reward.The method is designed to shape reasoning before the final answer is evaluated.
  • Sparse-PACR: Sparse-PACR assigns each trajectory a reward equal to the proportion of reasoning steps with positive confidence gains.The reward uses an indicator over stepwise confidence changes and is integrated into the trajectory advantage.
  • Dense-PACR: Dense-PACR uses each step’s confidence gain as an immediate reward and computes discounted returns for fine-grained supervision.The resulting per-step advantages are combined with the trajectory-level GRPO advantage before updating the policy.
  • Dense-PACR: Dense-PACR normalizes discounted returns across trajectories at each step with Min-Max scaling to produce a stable, purely positive advantage signal.Missing steps in shorter trajectories are assigned zero return before normalization.

6 EXPERIMENTAL SETUP

The experiments compare Sparse-PACR and Dense-PACR with Dr.GRPO across three open-source language models and five mathematical reasoning benchmarks. Training uses MATH data, grouped response sampling, and averages results across three random seeds.

  • Models and Baselines: Experiments compare Dr.GRPO with Sparse-PACR and Dense-PACR on Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen3-4B.Dr.GRPO is the bias-mitigated GRPO baseline implemented with the OAT framework.
  • Datasets and Evaluation: Models are trained on MATH and evaluated with pass@1 accuracy on MATH500, Minerva-Math, OlympiadBench, AIME 2024, and AMC 2023.The 1.5B model uses the full training set, while the 4B and 7B models use levels 3–5.
  • Training Details: Each problem produces a group of 8 sampled responses at temperature 1.0, and reported results average three runs with different random seeds.Training uses a single node with 8 NVIDIA H100 80GB GPUs.

7 RESULTS AND ABLATIONS

Across five reasoning benchmarks and all tested model sizes, both PACR variants outperform Dr.GRPO, with Dense-PACR consistently stronger than Sparse-PACR. PACR also improves early training dynamics, while non-penalizing Min-Max normalization sustains exploration better than LOO normalization.

  • 7.1 EXPERIMENTAL RESULT: Both Sparse-PACR and Dense-PACR consistently outperform Dr.GRPO across five benchmarks and all tested model sizes.Table 1 reports pass@1 accuracy and absolute changes relative to the Dr.GRPO baseline.
  • 7.1 EXPERIMENTAL RESULT: Dense-PACR consistently achieves better performance than Sparse-PACR, indicating gains from its more fine-grained reward signal.Sparse-PACR remains effective, but Dense-PACR provides more detailed feedback during training.
  • 7.2 TRAINING CURVE: PACR ACCELERATES EXPLORATION AND IMPROVES CONVERGENCE: PACR methods improve faster during early exploration and converge to higher final accuracy than the Dr.GRPO baseline.Figure 7 tracks average pass@1 accuracy and its rate of improvement for Qwen2.5-Math-1.5B.
  • 7.3 ANALYSIS ON ADVANTAGE FORMULATION: IMPACT OF PENALIZING INTERMEDIATE STEPS: LOO normalization initially accelerates learning but causes premature convergence, whereas Min-Max normalization sustains exploration and reaches higher final accuracy.Min-Max avoids process-level penalties; negative training feedback comes only from an incorrect final answer under the main GRPO reward.

8 CONCLUSION

PACR addresses sparse RLVR rewards with a dense, model-intrinsic confidence signal and improves training outcomes across reasoning benchmarks, especially with Dense-PACR.

  • PACR augments GRPO with a dense, model-intrinsic reward and achieves higher final performance across multiple reasoning benchmarks.The fine-grained Dense-PACR variant is reported as most effective.

A.1 LIMITATIONS AND FUTURE WORK

The study’s evidence is primarily limited to language models and mathematical reasoning, motivating future evaluation of PACR in multimodal tasks.

  • PACR is primarily evaluated in language models for mathematical reasoning, leaving multimodal reasoning tasks such as visual mathematics for future work.The authors specifically propose investigating PACR with Vision Language Models.

A.2 BROADER IMPACT

PACR provides fine-grained reasoning supervision from intrinsic confidence dynamics without separate reward models or manual annotation, with qualitative examples linking confidence spikes to pivotal steps.

  • PACR provides fine-grained, step-level supervision without training external process-reward models or requiring manual data annotation.The paper frames this as lowering computational and financial barriers to training reasoning agents.
  • The qualitative analyses use GPT-5 evaluation prompts for reasoning coherence and for assessing whether confidence gains correlate with pivotal reasoning steps.The corresponding prompts are shown in Figures 9 and 10.
  • Large positive spikes in stepwise confidence gain consistently align with pivotal problem-solving steps, including key formulas and critical calculations.Examples span polynomial factorization, polynomial symmetry, consecutive-integer median, and currency-exchange problems.
Loading 2510.22255v1…