PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Probability-Based Reinforcement Learning From Posterior Concentration

Shiu-Hong Kao, Yubo Zhao, Zhenyu Tian, Pengzhan Sun, Yicong Li, Angela Yao

arXiv:2610.01458v1cs.AI

TL;DR

Probability-based rewards offer verifier-free supervision, but their reliability for long reasoning traces remains underexplored. The paper identifies posterior concentration and proposes RLCPR, which combines uncertainty-aware sampling and concentration-aware regularization. RLCPR outperforms RLPR on six of seven benchmarks while using fewer cumulative rollout tokens.

  • Problem

    The reliability of probability-based rewards for long reasoning traces is underexplored, although verifier-free supervision is useful where correctness is unavailable, ambiguous, or expensive to verify.

    Source: §1 · pp. 1–2

  • Method

    RLCPR combines uncertainty-aware data sampling before generation with regularization that penalizes unnecessarily long traces when posterior rewards concentrate.

    Source: §1 · p. 2, §5 · p. 7

  • Results

    RLCPR outperforms RLPR on six of seven benchmarks by up to 4.0 percentage points while requiring fewer cumulative rollout tokens.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Concentration-aware training can improve effectiveness and token efficiency in probability-based RL.

    Source: §7 · p. 10

    The PCP analysis relies on eventual non-increase of length-normalized entropy, a sufficient condition motivated by observed branching-factor declines.

    Source: §3.2 · p. 4, §4.2 · p. 5

Abstract

from arXiv · show

Verifier-free reinforcement learning with probability-based rewards offers a promising way to train LLMs on general reasoning tasks where external verifiers are unavailable. Yet the reliability of these rewards, especially in long-horizon reasoning, remains underexplored. This work identifies a length-dependent failure mode of probability rewards, which we call the Posterior Concentration Phenomenon (PCP). We show that the probability of a reference answer conditioned on a reasoning trace often collapses to a low-variance interval as the trace becomes lengthy. This phenomenon results in nearly indistinguishable rewards, which, under GRPO-based settings, makes probability-based policy optimization unstable and inefficient. Motivated by this, we propose Reinforcement Learning with Concentration-aware Posterior Rewards (RLCPR), a verifier-free RL framework to explicitly account for PCP for better optimization stability and token efficiency. It has two components: uncertainty-aware data sampling, which reduces concentration-prone rollouts before generation, and concentration-aware regularization, which penalizes unnecessarily long traces when posterior rewards collapse. Extensive experiments show that, alongside higher token efficiency, RLCPR outperforms the state-of-the-art verifier-free RL baseline by up to 4.0% on six of seven benchmarks, including general-domain and mathematical reasoning challenges.

1 INTRODUCTION

Probability-based rewards provide verifier-free supervision, but their reliability across long reasoning traces is uncertain. This work identifies posterior concentration and proposes RLCPR, which outperforms RLPR on six of seven benchmarks.

1 INTRODUCTION

  • Posterior probability-based rewards offer a verifier-free alternative by scoring a reasoning trace with the model’s probability of the ground-truth answer conditioned on that trace.They provide dense, self-contained supervision without requiring an external model during training.Source: §1 · pp. 1–2
  • The Posterior Concentration Phenomenon (PCP) makes rewards for correct and incorrect answers less distinguishable and contracts their cross-rollout variance as traces lengthen.The paper connects this empirical behavior to log-likelihood concentration and provides a theoretical formulation under a mild condition motivated by evidence on LLM branching factors.Source: §1 · p. 2
  • PCP can undermine GRPO because similar rewards across rollouts leave reward differences unable to indicate which reasoning traces are better.As rollouts tend to lengthen during training, training dynamics are pushed further into the concentrated regime.Source: §1 · p. 2
  • RLCPR outperforms RLPR by up to 4.0 percentage points on six of seven benchmarks, including general-domain and mathematical reasoning challenges.The framework combines uncertainty-aware data sampling with regularization that penalizes unnecessarily lengthy traces.Source: §1 · p. 2

2 RELATED WORK

Prior work studies confidence reliability in language models and develops verifier-free RL rewards from model probabilities. The reliability of these rewards for long reasoning traces remains underexplored.

2 RELATED WORK

  • Studies of confidence reliability report that answer-token probabilities can reflect consistency with generated reasoning rather than correctness, while chain-of-thought explanations may rationalize hidden biases or fail to determine predictions.Prior beliefs may also induce confirmation bias in rationale generation and answer prediction.Source: §2 · p. 3
  • Verifier-free RL methods derive rewards from model confidence, including reference-answer posterior probabilities, answer log-probabilities, token-level perplexities, and length-normalized or debiased rewards.The reliability of probability-based supervision for long reasoning traces remains underexplored.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries define posterior-based rewards and their use in group-relative RL, then introduce log-likelihood concentration as motivation for analyzing posterior reward collapse.

3.1 POSTERIOR PROBABILITY-BASED RL

  • A posterior reward represents the model’s probability of the reference answer conditioned on a generated reasoning sequence, with methods using different token-level aggregations of that signal.VeriFree uses sequence likelihood, while RLPR uses token-level probability mean.Source: §3.1 · p. 3
  • GRPO compares sampled candidate outputs within a prompt group, while RLPR filters low-variance reward groups only after generation and scoring, leaving their generation cost incurred.The filtering criterion uses the group reward standard deviation relative to an exponential moving average of within-group standard deviations.Source: §3.1 · pp. 3–4

3.2 ENTROPY AND LOG-LIKELIHOOD CONCENTRATION FOR LLMS

  • Log-likelihood concentration means sufficiently long sequences of the same length tend to have similar negative log-likelihoods despite different generation trajectories.The analysis uses eventual non-increase of length-normalized realized entropy as a sufficient condition, motivated by the observed decrease in branching factor with generation length.Source: §3.2 · p. 4

4 POSTERIOR CONCENTRATION PHENOMENON

The section asks whether posterior rewards distinguish reasoning quality and finds that their discriminability contracts as traces lengthen. It names this pattern PCP and characterizes conditions under which posterior probabilities for similar-length traces become more alike.

4.1 WARM-UP: EXPLORATION EXPERIMENTS

  • On MMLU-Pro and GPQA-Diamond, correct- and incorrect-answer posterior rewards converge with trace length while overall and within-group reward variances contract.The within-group contraction matters for the groups used in relative policy optimization.Source: §4.1 · p. 4, Figure 2 · p. 5

4.2 POSTERIOR CONCENTRATION PHENOMENON

  • The derivation separates unequal-length posterior differences into a deterministic length-induced entropy shift and trace-specific residual differences; the deterministic shift cancels for equal-length traces.The shift is W_n − W_{n+s}, while the residual term captures deviations from the log-likelihood approximation.Source: §4.2 · p. 5
  • Under eventual non-increase of average token entropy, large answer-window entropy shifts become rarer as the length range grows, supporting the PCP characterization when residual differences are close.The proposition frames posterior similarity probabilistically across sampled lengths; it does not imply lower model confidence.Source: §4.2 · p. 6
  • When posterior rewards lose discriminability, long rollouts may receive similar rewards despite reasoning-quality differences, weakening relative learning signals while retaining generation cost.The section motivates avoiding concentration-prone prompts and discouraging unnecessary length after posterior differences collapse.Source: §4.2 · p. 6

5 RLCPR

RLCPR combines uncertainty-aware sampling before generation with a length penalty for concentrated rollout groups. Its sampling uses direct-answer uncertainty as a proxy for concentration risk, while its regularizer targets long traces when posterior rewards collapse.

5 RLCPR

  • Uncertainty-aware data sampling uses direct-answer uncertainty as a proxy for concentration risk and filters questions before full reasoning-trace generation.The uncertainty estimates require only short direct-answer completions and are cached across training steps.Source: §5 · pp. 6–7, Figure 3 · p. 7
  • Concentration-aware regularization applies a length penalty only to groups detected as concentrated, where traces are long and posterior probabilities are similar.Runtime thresholds are set from batch quantiles, and the penalty weight is suggested to start low and increase gradually.Source: §5 · pp. 7–8
  • RLCPR beats same-backbone RLPR on six of seven benchmarks, improves average accuracy by 1.8%, and achieves higher training token efficiency.Table 1 reports overall performance on seven reasoning benchmarks.Source: Table 1 · p. 8

6 EXPERIMENTS

RLCPR improves average accuracy over RLPR on six of seven benchmarks while using fewer rollout tokens. Its components and training schedule are evaluated through ablations.

6.1 EXPERIMENTAL SETTINGS

  • Experiments use Qwen3-4B-Base, training on 77K non-mathematical reasoning prompts and evaluating on four general-domain and three mathematical benchmarks.RLCPR trains for 600 steps with an output-token budget comparable to RLPR.Source: §6.1 · p. 8

6.2 MAIN RESULTS

  • Average accuracy rises from 56.2% with RLPR to 58.0% with RLCPR, which outperforms RLPR on six of seven benchmarks.Despite training only on non-mathematical prompts, RLCPR also improves MATH-500, Minerva, and AIME24 by 2.2, 3.0, and 2.3 points, respectively.Source: §6.2 · pp. 8–9
  • At 320 steps, RLCPR uses 0.75 billion rollout tokens, 62.3% fewer than RLPR’s 1.99 billion, while achieving higher MMLU-Pro accuracy.After 600 steps, RLCPR uses 1.66 billion tokens.Source: §6.2 · p. 9, Figure 4 · p. 9

6.3 ABLATION STUDY

  • Combining UDS and CR achieves the best performance on all three ablation benchmarks, improving over the baseline by up to 6.7%.CR complements UDS by adding a length-aware penalty when posterior rewards concentrate; UDS alone slightly lowers MATH-500 and Minerva performance.Source: §6.3 · p. 9, Table 2 · p. 9
  • Removing the CR cold start lowers accuracy by 4.9, 8.6, and 7.9 percentage points on MMLU-Pro, TheoremQA, and WebInstruct, respectively.Fixing UDS thresholds instead of annealing them also lowers accuracy across all three benchmarks.Source: §6.3 · p. 9, Table 3 · p. 9

7 CONCLUSION

The paper identifies posterior concentration as a length-dependent loss of reward discriminability and proposes RLCPR to address it. Experiments report better benchmark performance with fewer rollout tokens than RLPR.

7 CONCLUSION

  • RLCPR accounts for posterior concentration through uncertainty-aware data sampling and selective length regularization.Experiments show it outperforms RLPR on six of seven reasoning benchmarks while requiring fewer cumulative rollout tokens.Source: §7 · pp. 9–10

AI USE STATEMENT

The authors used AI tools to polish language and presentation, check proofs, refine exposition, and assist with code and figure creation; they retain responsibility for the work.

AI USE STATEMENT

  • AI tools assisted with manuscript editing, proof checking, code implementation, and figure creation; the authors retain responsibility for correctness, originality, and integrity.Source: AI USE STATEMENT · p. 10

A PROOF OF LEM. 4.1

The proof establishes that lengths violating a stability condition have asymptotically zero density. A diagnostic also finds higher direct-answer uncertainty for questions whose rollout groups satisfy PCP.

A PROOF OF LEM. 4.1

  • Non-negative token-level entropy that is eventually non-increasing converges to a finite limit.The proof then shows that reasoning lengths where the shifted W values differ by more than ε form an asymptotically zero-density set.Source: A PROOF OF LEM. 4.1 · pp. 12–13

B PROOF OF PROP. 4.1

  • The proposition proof bounds the probability of a deviation using a uniform random reasoning length, the preceding lemma, a residual term, and Cesàro convergence.Source: B PROOF OF PROP. 4.1 · p. 14

C DIAGNOSTIC EXPERIMENT: RELATION BETWEEN UNCERTAINTY AND CONCENTRATION

D COMPLEXITY ANALYSIS

This material analyzes UDS generation-cost tradeoffs and records the protocols and implementation details for PCP diagnostics and RLCPR experiments.

D COMPLEXITY ANALYSIS

  • With r = 0.55, UDS filters about 45% of candidate questions before rollout generation; net generation savings require amortized estimation overhead below avoided rollout cost.Post-generation filtering generates rollouts for every candidate first; the analysis excludes reward scoring and policy updates from its cost comparison.Source: D COMPLEXITY ANALYSIS · p. 15

E.1 PCP’S EXPLORATION-STUDY SETUP

  • The PCP exploration study samples 128 MMLU-Pro and GPQA-Diamond questions each, generates eight responses per question, and caps responses at 1,536 tokens.It fits separate third-order regressions for correct and incorrect traces and examines posterior-reward variance across length bins, including within GRPO groups.Source: §E.1 · pp. 15–16

E.2 UNCERTAINTY–CONCENTRATION STUDY SETUP

  • The uncertainty–concentration diagnostic disables RLPR filtering and tracks direct-answer uncertainty in PCP and non-PCP subsets during 100 training steps.It uses batch size 320 and GRPO group size 8.Source: §E.2 · p. 16

E.3 RLCPR: IMPLEMENTATION DETAILS

  • RLCPR implementation generates eight responses per question, uses a 1e−6 learning rate, sets the KL coefficient to 0, and performs four policy updates per step.The batch size is 768.Source: §E.3 · p. 16

E.4 RLCPR TRAINING PROMPT

  • The Qwen3 prompt requires reasoning inside <think> tags followed by an answer inside <answer> tags, and removes the special think tokens from the vocabulary.The prompt includes an example reasoning-and-answer format before the user question.Source: §E.4 · p. 16

F ADDITIONAL ABLATION STUDY

The ablations favor selective concentration-aware regularization and uncertainty-directed sampling, while the PCP analysis retains an open theoretical condition.

F ADDITIONAL ABLATION STUDY

  • Removing the concentration gate lowers accuracy by 1.6 and 3.5 percentage points on MMLU-Pro and TheoremQA, respectively, while obtaining only comparable points on WebInstruct.The comparison is between concentration-gated regularization and applying the length penalty to all groups.Source: F ADDITIONAL ABLATION STUDY · p. 16, Table A1 · p. 17
  • Filtering more high-uncertainty than low-uncertainty data improves results over reversing the rates; widening the cutoff range reduces accuracy across all three mathematical benchmarks.The widened range places most data in the same uncertainty bin, leaving the data distribution largely unchanged.Source: F ADDITIONAL ABLATION STUDY · pp. 16–17, Table A2 · p. 17
  • A regularization weight around 0.5 is recommended; particularly high or low λ typically degrades performance, with percentile levels fixed at (0.8, 0.2) without tuning.The authors note that model- or benchmark-specific tuning may further improve performance.Source: F ADDITIONAL ABLATION STUDY · p. 17, Table A3 · p. 17

ABOUT PCP’S RESIDUAL CONDITION

  • The PCP theory assumes cross-trace residual-difference convergence as sufficient; when that condition holds remains open, and finite-length bounds are future work.The authors identify deriving sufficient conditions and finite-length bounds as directions beyond this work’s scope.Source: ABOUT PCP’S RESIDUAL CONDITION · p. 17
Loading 2610.01458v1…