PaperlayerView exact revision on arXiv

Source-linked AI summary

Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning

Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang

arXiv:2602.21420v1cs.LGcs.AI

TL;DR

RLVR can improve Pass@1 while narrowing reasoning diversity because it penalizes all incorrect rollouts uniformly, allowing overconfident errors to persist. ACE reweights negative advantages using a policy–reference confidence shift and theoretically relates this to selective regularization. Across model families and benchmarks, ACE improves the full Pass@k spectrum and composes with GRPO and DAPO.

  • Problem

    RLVR’s uniform treatment of incorrect rollouts can narrow the reasoning boundary and suppress diversity, while existing difficulty- and group-level methods overlook per-rollout error differences.

  • Method

    ACE uses c_i = log(πθ(y_i|x)/πref(y_i|x)) to amplify negative advantages for overconfident errors while largely preserving exploratory and self-correcting errors.

  • Results

    ACE consistently improves the full Pass@k spectrum across three model families and two benchmarks, including +2.2–3.0pp Pass@32 for ACE-GRPO over GRPO and 96.1% MATH-500 Pass@32 for ACE-DAPO.

  • Takeaways & Limitations

    ACE provides targeted correction that preserves Pass@1 while expanding large-k reasoning performance and remains complementary to existing diversity strategies.

  • Takeaways & Limitations

    ACE currently assumes binary rewards, and its confidence score may be unreliable when the reference model is poorly calibrated.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has become the leading paradigm for enhancing reasoning in Large Language Models (LLMs). However, standard RLVR algorithms suffer from a well-documented pathology: while they improve Pass@1 accuracy through sharpened sampling, they simultaneously narrow the model's reasoning boundary and reduce generation diversity. We identify a root cause that existing methods overlook: the uniform penalization of errors. Current approaches -- whether data-filtering methods that select prompts by difficulty, or advantage normalization schemes -- treat all incorrect rollouts within a group identically. We show that this uniformity allows overconfident errors (incorrect reasoning paths that the RL process has spuriously reinforced) to persist and monopolize probability mass, ultimately suppressing valid exploratory trajectories. To address this, we propose the Asymmetric Confidence-aware Error Penalty (ACE). ACE introduces a per-rollout confidence shift metric, c_i = log(pi_theta(y_i|x) / pi_ref(y_i|x)), to dynamically modulate negative advantages. Theoretically, we demonstrate that ACE's gradient can be decomposed into the gradient of a selective regularizer restricted to overconfident errors, plus a well-characterized residual that partially moderates the regularizer's strength. We conduct extensive experiments fine-tuning Qwen2.5-Math-7B, Qwen3-8B-Base, and Llama-3.1-8B-Instruct on the DAPO-Math-17K dataset using GRPO and DAPO within the VERL framework. Evaluated on MATH-500 and AIME 2025, ACE composes seamlessly with existing methods and consistently improves the full Pass@k spectrum across all three model families and benchmarks.

1 Introduction

RLVR improves Pass@1 but can narrow the reasoning boundary because probability mass concentrates on a small set of successful paths. ACE addresses this by selectively penalizing overconfident errors while preserving other error types.

  • RLVR models can improve Pass@1 while underperforming their base models at large-k Pass@k, indicating a narrowed reasoning boundary.The paper attributes this tension to diversity collapse: probability mass concentrates on a small number of successful reasoning paths.
  • Difficulty-based curriculum methods select prompts by difficulty but overlook distinctions among individual incorrect rollouts.
  • Standard RLVR penalizes exploratory, self-correcting, and overconfident errors uniformly, while global KL regularization suppresses beneficial exploration alongside harmful overconfidence.
  • ACE dynamically amplifies penalties for overconfident errors using a per-rollout confidence shift while leaving exploratory and self-correcting errors largely untouched.
  • ACE formalizes c_i = log(πθ(y_i|x)/πref(y_i|x)) as a per-rollout diagnostic orthogonal to prompt difficulty and reports that overconfident errors accumulate during training.
  • ACE’s gradient decomposes into a selective regularizer targeting overconfident policy regions plus a residual that partially moderates correction strength.
  • Across three model families, two benchmarks, and GRPO or DAPO, ACE improves the full Pass@k spectrum, with particularly strong gains at large k.

2 Related Work

Prior methods address difficulty, group-level weighting, KL control, or global exploration, whereas ACE differentiates incorrect rollouts using confidence shift. Its mechanism is asymmetric and targeted at overconfident errors.

  • Curriculum methods select prompts by difficulty, while advantage-shaping methods balance correct and incorrect samples at the group level; ACE operates at rollout level.
  • Global KL penalties symmetrically constrain deviations from the reference, suppressing beneficial exploration alongside harmful overconfidence.
  • DAPO promotes diversity globally through Clip-Higher, which gives low-probability exploration tokens more room for probability increase.
  • ACE’s confidence-dependent term acts as an asymmetric, adaptive auxiliary reward applied only to negative advantages.
  • ACE differentiates incorrect rollouts by confidence shift, addressing the overconfident errors identified as a mechanism of diversity collapse.

3 Preliminaries

The preliminaries define GRPO’s group-normalized objective and expose its uniform treatment of same-reward rollouts. This motivates distinguishing incorrect trajectories by their policy-relative confidence shifts.

  • GRPO computes each rollout’s advantage through group normalization within sampled rollouts.
  • Its clipped surrogate objective uses an importance-sampling ratio bounded by a clipping threshold.
  • Rollouts with identical rewards receive identical advantages, so binary-reward incorrect rollouts share the same negative advantage.
  • The only per-rollout modulation is the clipped importance ratio, which provides limited differentiation among qualitatively different errors.
  • Under standard GRPO, incorrect-rollout confidence shifts develop a heavy right tail during training, indicating that some incorrect paths become substantially more probable than under the reference.

4 The ACE Method

ACE uses the policy–reference confidence shift to reweight negative advantages: overconfident errors receive stronger penalties, while self-correcting errors remain near the GRPO baseline. Its theory connects this design to selective regularization and analyzes its gradient efficiency.

  • 4.1 Error Confidence Score: The error confidence score c_i compares policy and reference likelihoods for an incorrect rollout, with positive values indicating increased policy confidence.The score is decomposed into token-level log-ratios and is already available during standard RLVR training.
  • 4.1 Error Confidence Score: Positive, near-zero, and negative confidence shifts identify overconfident, exploratory, and self-correcting errors, respectively.
  • 4.2 The ACE Advantage: ACE restructures the negative advantage around the per-rollout confidence score and leaves positive advantages in the standard GRPO form.
  • 4.2 The ACE Advantage: Softplus is smooth and strictly increasing, producing near-zero modulation for strongly negative shifts and larger penalties for increasingly positive shifts.
  • 4.2 The ACE Advantage: ACE multiplies the standard negative advantage by 1 + α·Softplus(c_i), preserving GRPO’s difficulty scaling while adding within-group confidence differentiation.
  • 4.2 The ACE Advantage: Within one group, ACE assigns penalties |Â−|·(1+α·2.13), |Â−|·(1+α·0.69), and |Â−|·(1+α·0.05) to c_i values 2, 0, and −3, respectively.
  • 4.3 Algorithm: ACE’s algorithm samples verified rollouts, computes GRPO advantages, amplifies incorrect-rollout penalties, and updates the clipped surrogate objective.
  • 4.3 Algorithm: ACE normalizes confidence shifts by sequence length to make penalty magnitudes comparable across rollouts of different lengths.

5 Experiments: ACE Expands the Reasoning Boundary

ACE is evaluated across three model families, two RL algorithms, and two math benchmarks using matched training recipes, and consistently improves large-k Pass@k while preserving comparable Pass@1. The strongest results show that ACE expands the reasoning boundary and composes with DAPO’s diversity-preserving strategy.

  • Experimental Setup: Experiments fine-tune Qwen2.5-Math-7B, Qwen3-8B-Base, and Llama-3.1-8B-Instruct on DAPO-Math-17K using GRPO and DAPO within VERL.Evaluations use MATH-500 and AIME 2025 with rule-based verification, while training recipes and budgets are matched across methods.
  • Experimental Setup: Pass@k is reported for k ∈{1, 2, 4, 8, 16, 32}; Pass@1 reflects exploitation, whereas large-k values reflect exploration and the reasoning boundary.Sampling uses temperature 0.7 and top-p = 0.95.
  • Qwen2.5-Math-7B: On MATH-500, ACE-GRPO raises Qwen2.5-Math-7B Pass@32 from 91.3% to 94.3% (+3.0pp), while ACE-DAPO reaches 96.1% (+1.5pp over DAPO’s 94.6%).ACE consistently improves larger-k metrics while maintaining comparable Pass@1.
  • Qwen3-8B-Base: On AIME 2025, ACE-GRPO raises Qwen3-8B-Base Pass@32 from 29.8% to 32.4% (+2.6pp), while ACE-DAPO reaches 34.4% (+1.3pp over DAPO’s 33.1%).The same improvement pattern holds on a different model family.
  • Llama-3.1-8B-Instruct: On MATH-500, ACE-GRPO raises Llama-3.1-8B-Instruct Pass@32 from 79.3% to 81.5% (+2.2pp), while ACE-DAPO reaches 82.1% (+1.7pp over DAPO’s 80.4%).On AIME 2025, ACE-GRPO improves Pass@32 from 7.0% to 8.2% (+1.2pp) in a low-accuracy regime.
  • Cross-Method Findings: ACE-DAPO achieves the strongest results across all k, and consistent gains across three model families indicate composability with DAPO and generality of ACE’s mechanism.ACE’s gain over DAPO is smaller than over GRPO because Clip-Higher and ACE partially address the same overconfident-error pathology, while ACE adds trajectory-level selectivity.

Main Result

ACE preserves Pass@1 while expanding the reasoning boundary at larger sampling budgets across model families and benchmarks. Diagnostics indicate that ACE suppresses overconfident errors, retains more entropy, and benefits from smooth confidence modulation.

  • Main results: ACE-GRPO improves Pass@32 by +2.2–3.0pp over GRPO across three model families and two benchmarks, while ACE-DAPO improves over DAPO by +1.2–1.7pp.ACE-DAPO reaches 96.1% on MATH-500 Pass@32 and achieves the best overall results.
  • Overconfident error dynamics: ACE-GRPO maintains lower overconfident error fraction and lower mean overconfidence magnitude than standard GRPO throughout training.The diagnostics measure the prevalence and severity of incorrect rollouts whose confidence exceeds that under the reference policy.
  • Main results: ACE consistently improves the full Pass@k spectrum, with larger gains at high k, and ACE-DAPO outperforms DAPO across sampling budgets, model families, and benchmarks.The comparison covers MATH-500 and AIME 2025 with Qwen2.5-Math-7B, Qwen3-8B-Base, and Llama-3.1-8B-Instruct.
  • Modulation-function ablation: Both ACE-ReLU and ACE-Softplus outperform standard GRPO for k > 1, while ACE-Softplus exceeds ACE-ReLU with a widening Pass@32 gap of +1.2pp.Softplus provides smooth gradients and non-zero modulation near c_i = 0, unlike ReLU.
  • Entropy dynamics: ACE-GRPO retains more entropy on Qwen2.5-Math-7B and maintains more stable entropy on Qwen3-8B-Base during the first 20 training steps.Standard GRPO exhibits rapid entropy collapse on Qwen2.5-Math-7B; Llama-3.1-8B-Instruct is excluded from this diagnostic because its lower baseline accuracy makes entropy less comparable.
  • Mechanistic interpretation: ACE’s selective KL term acts as entropy regularization restricted to overconfident errors, leaving exploratory errors with c_i ≤ 0 comparatively less affected.This mechanism addresses probability sinks formed by uniformly penalized overconfident errors.

6 Limitations and Future Work

ACE’s current scope is constrained by its reliance on a calibrated reference model, binary rewards, and sequence-length normalization that may not transfer cleanly to very long chains.

  • ACE’s confidence score may misidentify spurious patterns when the reference model is poorly calibrated.The paper suggests moving-average checkpoints as a possible alternative for future work.
  • ACE currently assumes binary rewards, so continuous or partial feedback requires redefining overconfident errors.The paper specifically mentions process reward models as a setting requiring this extension.
  • For outputs exceeding 10K tokens, confidence-shift dynamics may differ and the normalization c̄_i = c_i/T_i may need refinement.

7 Conclusion

The paper identifies overconfident errors as a previously overlooked RLVR pathology and proposes ACE to penalize them selectively while preserving exploratory errors.

  • ACE addresses overconfident errors by dynamically amplifying their negative advantages while leaving exploratory errors untouched.The method is presented as a simple modification to the advantage function.

Core Contributions

The paper’s core contributions are a per-rollout confidence diagnostic, a theoretical selective-regularization decomposition, and improvements across the full Pass@k spectrum without sacrificing Pass@1.

  • ACE formalizes per-rollout confidence shift as a diagnostic orthogonal to prompt difficulty and reports accumulation of overconfident errors during RLVR.The confidence shift is defined as c_i = log(πθ(y_i|x)/πref(y_i|x)).
  • ACE’s gradient decomposes into a selective regularizer on overconfident errors plus a tempering residual.
  • ACE improves the full Pass@k spectrum, especially at large k, without sacrificing Pass@1.The contribution statement also notes that ACE adds only a single Softplus computation per incorrect rollout.

A Proof of Theorem 1 (Selective Regularization Decomposition)

The proof shows that ACE captures the score-function component of a selective regularizer, while omitting differentiation through the confidence-dependent Softplus term creates an explicit residual.

  • ACE’s additional gradient differs from standard GRPO only in negative-advantage terms and treats Softplus(c_i) as a detached scalar reward modifier.The per-prompt negative advantage magnitude is constant across rollouts within a group.
  • The true selective-regularizer gradient contains a score-function term captured by ACE and a second residual term from differentiating Softplus(c).The product-rule decomposition separates Term I, captured by ACE, from Term II, the residual.
  • Summing over incorrect rollouts makes ACE’s gradient match the negative-advantage-weighted Term I exactly.The proof describes this as an exact identity after taking expectations.
  • The residual arises because ACE treats Softplus(c_i) as fixed and omits the gradient through c_i itself.
  • For c in [1, 3], the residual-to-Softplus ratio ranges from 31–56%, partially moderating the effective regularizer strength.The residual is therefore not negligible in this confidence range.
  • Stopping the gradient through Softplus(c_i) follows standard policy-gradient practice, while retaining the residual would make penalty magnitude an optimization target.The paper frames this as avoiding a feedback loop in which the objective also reduces the penalty magnitude itself.

B Gradient Quality Analysis

ACE increases gradient variance, but under supported alignment and high-variance conditions, its signal grows faster than its noise, improving gradient quality.

  • Alignment assumption: Overconfident errors tend to have score functions more aligned with the expected gradient direction, expressed as positive directional covariance.This alignment supplies the condition under which confidence-dependent weighting can improve quality.
  • Second-moment effects: ACE strictly increases the mean squared gradient norm of incorrect rollouts whenever errors are not all zero-gradient.This follows from the additive weighting factor (1 + αϕ_i) > 1.
  • Directional variance: ACE necessarily increases directional variance because its additive reweighting raises the gradient's second moment.The analysis characterizes this increase as an unavoidable cost of the weighting structure.
  • Quality improvement: When initial gradient noise exceeds signal, ACE strictly improves the gradient quality ratio for sufficiently small α.The condition is Var[u_i] > (E[u_i])^2, equivalently Qstd_d < 1.
  • Quality improvement: ACE improves gradient quality because its signal increase can outpace the accompanying variance increase along the optimization-relevant direction.The result is established under positive alignment covariance and a low initial quality ratio, typical of highly variable stochastic policy gradients.

C Implementation Details

ACE is implemented as a lightweight advantage modification for incorrect rollouts, using sequence-level policy–reference confidence and preserving standard advantages for correct rollouts.

  • Compute overhead: ACE adds one Softplus computation per incorrect rollout per training step, with reported overhead below 0.1% of wall-clock time.Rollout generation remains the stated training bottleneck.
  • Confidence computation: ACE uses sequence-level confidence c = log_probs_policy − log_probs_ref, normalized by sequence length.The main experiments use sequence-level aggregation to capture trajectory confidence.
  • Advantage computation: Incorrect-rollout advantages are multiplied by 1 + α · Softplus(c), while correct rollouts retain the standard advantage.The combined advantage switches between the standard and ACE-modified forms according to binary reward correctness.
  • Compatibility: ACE can be inserted into GRPO, PPO, or REINFORCE advantage computation without changing model architecture, rollout generation, or reward computation.The modification is confined to the advantage calculation in the RLVR loop.
  • Sensitivity to α: α = 1.0 achieves Pass@32 = 94.3% (+3.0pp over GRPO) on MATH-500, while performance remains stable for α ∈ [0.5, 2.0].Pass@1 decreases slightly only at α ≥ 2.0; α = 1.0 is used as the default.
  • Experimental controls: Experiments use matched training recipes and budgets within each model when comparing GRPO with ACE-GRPO and DAPO with ACE-DAPO.The matching covers data, verifier, rollout settings, optimization, clipping/KL coefficients, sequence lengths, and update counts.
Loading 2602.21420v1…