PaperlayerView exact revision on arXiv

Source-linked AI summary

CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment

Guofu Xie, Yunsheng Shi, Hongtao Tian, Ting Yao, Xiao Zhang

arXiv:2508.02298v4cs.LGcs.AIcs.CL

TL;DR

RLVR assigns identical rewards to all tokens, while existing fine-grained approaches can be unreliable, costly, or inefficient. CAPO uses an off-the-shelf LLM-as-GenPRM to generate deterministic step-wise critiques in one pass, and experiments show consistent gains over supervised and RL-based methods across mathematical and general reasoning benchmarks. The analysis links these gains to learning correct reasoning pathways.

  • Problem

    RLVR’s response-level binary rewards provide coarse credit assignment, while existing fine-grained methods face reliability, reward-hacking, supervision, or online-efficiency limitations.

    Source: §1 · pp. 2–3, §2 · p. 4

  • Method

    CAPO uses an off-the-shelf general-purpose LLM-as-GenPRM to generate deterministic step-wise critiques in one pass, applies token-level credits, and improves robustness through voting.

    Source: §1 · pp. 2–3, §7 · p. 10

  • Results

    CAPO consistently outperforms supervised fine-tuning and reinforcement-learning methods across mathematical and general reasoning benchmarks.

    Source: §1 · p. 3, §7 · p. 10

  • Takeaways & Limitations

    CAPO fosters learning of correct reasoning pathways that lead to correct answers and promotes more effective exploration.

    Source: §1 · p. 3, §7 · p. 10

    CAPO’s reliability depends on using a GenPRM substantially more capable than the policy model and providing it with ground-truth information.

    Source: C DISCUSSION AND LIMITATIONS · p. 15

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has improved the reasoning abilities of Large Language Models (LLMs) by using rule-based binary feedback. However, current RLVR methods typically assign the same reward to every token. This coarse-grained feedback hampers precise credit assignment, making it hard for models to identify which reasoning steps lead to success or failure, and often results in suboptimal policies. Methods like PPO provide credit assignment by value estimation, but yield inaccurate and unverifiable signals due to limited sampling. On the other hand, methods using Process Reward Models can provide step-wise rewards but suffer from several key limitations: they require high-quality process supervision labels, the feedback is unreliable due to probabilistic reward modeling, and their application in online reinforcement learning (RL) is time-consuming. To overcome these limitations, we introduce a simple but efficient method-Credit Assignment Policy Optimization (CAPO). Instead of training auxiliary models, CAPO directly leverages an off-the-shelf, general-purpose LLM as a Generative Process Reward Model (LLM-as-GenPRM) to generate all step-wise critique by one pass only based on the correctness of the step itself, providing deterministic token-level credits to refine the tokens that were originally assigned identical rule-based rewards. To further enhance the accuracy and robustness, we employ voting mechanisms that scale with the number of generated critiques. Extensive experiments on various backbones like Llama and Qwen models show that CAPO consistently outperforms supervised learning-based and RL-based fine-tuning methods across four challenging mathematical benchmarks and three out-of-domain benchmarks. Further analysis shows that CAPO can help the model to foster the learning of correct reasoning pathways leading to correct answers.

1 INTRODUCTION

RLVR offers reliable binary feedback but assigns the same response-level reward to every token, limiting credit assignment. CAPO uses an off-the-shelf LLM to generate deterministic step-wise credits and consistently outperforms supervised and RL-based alternatives.

1 INTRODUCTION

  • RLVR assigns a binary reward for the entire response to all tokens, so it cannot distinguish correct reasoning steps from individual errors.Source: §1 · p. 2
  • Fine-grained alternatives can be unreliable, reward-hackable, costly to supervise, or inefficient online because they depend on estimation-based signals, auxiliary models, or repeated process-reward inference.Source: §1 · pp. 2–3
  • CAPO uses an off-the-shelf general-purpose LLM as a Generative Process Reward Model to judge step correctness in one pass and assign token-level credits without specialized reward models.Source: §1 · pp. 2–3
  • CAPO consistently outperforms supervised fine-tuning and other reinforcement-learning methods lacking fine-grained credit assignment across four mathematical and three general reasoning benchmarks.Source: §1 · p. 3
  • CAPO analyzes the conflict between outcome and process rewards and uses asymmetric reward shaping to prioritize correct outcomes while encouraging sound reasoning.Source: §1 · p. 3

2 PROBLEM ANALYSES

Outcome-based RLVR gives every token the same final-outcome reward, leaving the responsible reasoning actions unidentified. Fine-grained methods address this gap but introduce reliability, reward-hacking, annotation, and online-efficiency concerns.

2 PROBLEM ANALYSES

  • RLVR treats an entire response as one action and assigns identical final-outcome rewards to all tokens, making partially correct and flawed reasoning difficult to distinguish.Source: §2 · p. 4
  • Fine-grained credit methods using reward or value models can be exploited through inaccuracies or biases, allowing policies to maximize modeled rewards rather than true performance.Source: §2 · p. 4
  • Process reward models additionally require costly manual annotations and multiple online inference calls, while probabilistic or Monte Carlo estimation makes process rewards noisy and unreliable.Source: §2 · p. 4
  • The central research question is how to obtain effective fine-grained feedback while avoiding complex dependencies that lead to reward hacking.Source: §2 · p. 4

3 CAPO: THE PROPOSED APPROACH

CAPO generates deterministic process credits by having an LLM identify erroneous reasoning steps in one pass, aggregates multiple critiques, and applies token-level penalties. Its asymmetric reward shaping preserves outcome priority while enabling process-oriented learning.

3 CAPO: THE PROPOSED APPROACH

  • CAPO prompts an off-the-shelf LLM-as-GenPRM to identify erroneous steps in one pass, assigns asymmetric penalties to their tokens, and uses the resulting credits for RL training.Source: §3 · p. 5, §3.1 · p. 5

3.1 PROCESS CREDITS GENERATION VIA LLM-AS-GENPRM

  • Multiple critiques are aggregated with voting to reduce stochastic decoding noise; intersection voting favors precision, while majority voting balances precision and recall.Source: §3.1 · p. 5

3.2 FINE-GRAINED CREDIT ASSIGNMENT

  • CAPO localizes response-level rewards to tokens belonging to identified erroneous steps, suppressing those segments while appropriately rewarding correct ones.Source: §3.2 · pp. 5–6
  • When process penalties dominate, models may generate long, simple responses with higher per-token rewards but lower final accuracy, so outcome rewards must remain the corrective priority.Source: §3.2 · p. 6
  • Early process learning can slow training because correct intermediate steps may not yet produce correct answers, whereas later it distinguishes samples with better reasoning among equally successful responses.Source: §3.2 · p. 6
  • CAPO therefore sets W_whole > W_process, ensuring every correct-answer trajectory outranks every wrong-answer trajectory while still refining reasoning within that constraint.Source: §3.2 · p. 6

4 EXPERIMENTS

CAPO is evaluated against supervised fine-tuning and RL baselines across mathematical and out-of-distribution reasoning benchmarks. It consistently improves performance, with gains supported by critique voting and asymmetric outcome–process rewards.

4.1 EXPERIMENTS SETUPS

  • CAPO is evaluated on four mathematical benchmarks and three out-of-distribution benchmarks using Llama and Qwen backbones, with comparisons against SFT and GRPO baselines.Evaluation reports Pass@1 averaged over three runs, while the main CAPO experiments use four critiques with voting.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • +2.5 points overall across all benchmarks for Qwen2.5-7B, while CAPO also gains +3.5 points on math benchmarks over the baselines.For Llama-3-1B, CAPO gains +3.5 points on OOD tasks and +2.9 points overall; for Llama-3-3B, it leads the best baseline by +2.1 points in math and +1.8 points overall.Source: §4.2 · pp. 7–8, Table 2 · p. 7, Table 3 · p. 8
  • CAPO and its variants consistently outperform GRPO without explicit credit assignment across backbones, including gains on general reasoning tasks.The reported improvements are therefore not limited to mathematical reasoning.Source: §4.2 · p. 8
  • Using different off-the-shelf GenPRMs without fine-tuning demonstrates low sensitivity to the verifier choice and supports CAPO’s generality.Qwen2.5-72B-Instruct and Llama-3-70B-Instruct serve as GenPRMs for the Qwen and Llama experiments, respectively.Source: §4.1 · p. 7, §4.2 · p. 8
  • Performance scales with the number of generated critiques, reaching its best overall level at N=8 across mathematical and OOD reasoning tasks.Voting more critiques is described as reducing noisy judgments and producing more accurate credit assignment.Source: §4.2 · pp. 8–9, Figure 3 · p. 9
  • The asymmetric reward configuration C=2, P=1 outperforms alternatives because outcome rewards remain dominant while process rewards provide a meaningful secondary signal.Symmetric and extreme process-focused settings produce ambiguous or poorly calibrated rewards, while C=2, P=0.1 underperforms because its process penalty is weak.Source: Table 4 · p. 8, §4.2 · p. 9
  • CAPO produces longer responses than GRPO while average token-level rewards also increase, indicating more extensive reasoning is paired with higher reported quality.The analysis contrasts this pattern with verbosity that merely introduces additional errors.Source: §4.2 · p. 9

5 ADDITIONAL EXPERIMENTS

The paper places detailed limitations and additional experiments outside the main text, in Appendices C and E.

  • Appendix C contains the detailed discussion and limitations of CAPO, while Appendix E reports additional analyses and experiments.These include process-versus-outcome rewards, voting mechanisms, ground-truth solutions, GenORM and GenPRM comparisons, and rollout case studies.Source: §5 · p. 9

6 RELATED WORK

The paper relates CAPO to work on reinforcement learning for LLM reasoning, credit assignment, and generative reward modeling.

  • Related work covers LLM reasoning with reinforcement learning, credit assignment in reinforcement learning, and generative reward modeling.The paper directs readers to Appendix B for detailed discussion because of space limitations.Source: §6 · p. 9

7 CONCLUSION

CAPO provides fine-grained credit assignment by combining single-pass LLM-generated critiques, voting, and reward shaping. Experiments report stronger benchmark performance and learning of reasoning pathways associated with correct answers.

  • CAPO uses an off-the-shelf LLM to generate step-wise critiques in one pass, combines multiple critiques by voting, and balances outcome and process rewards.The method avoids complex auxiliary models while producing reliable credits for reinforcement learning.Source: §7 · pp. 9–10
  • CAPO outperforms existing RLVR methods without accurate credit assignment on challenging mathematical and general reasoning benchmarks.Further analysis reports more effective exploration through learning correct reasoning pathways.Source: §7 · p. 10

8 ETHICS STATEMENT

The paper focuses on improving mathematical and logical reasoning in LLMs through objective reasoning tasks and standard academic benchmarks, without directly addressing broader societal risks.

  • The research targets more reliable and correct internal reasoning processes in large language models.Source: §8 · p. 10

9 REPRODUCIBILITY STATEMENT

The paper describes reproducibility resources, limited supportive LLM use in writing, and related work on reinforcement learning and credit assignment. It situates CAPO among methods addressing inaccurate or non-verifiable credit assignment.

9 REPRODUCIBILITY STATEMENT

  • The implementation, prompts, experimental setup, hyperparameters, and design analyses are provided to support replication.The authors state that code and scripts are included in the supplementary material, with additional details in the appendices.Source: §9 · p. 10

A STATEMENT ON LLM USAGE

  • LLMs were used only for writing assistance and citation formatting, not for research design, data analysis, or interpretation.Source: A STATEMENT ON LLM USAGE · p. 14

B RELATED WORK

  • Recent RL algorithms improve optimization, while this work addresses imprecise or non-verifiable credit assignment.Source: B RELATED WORK · p. 14
  • PRM-based methods can be inaccurate and costly online, whereas reward-densification methods use inaccurate credit estimates and lack reliable process rewards.VinePPO replaces value-function learning with value-free Monte Carlo estimates but incurs substantial computational cost.Source: B RELATED WORK · p. 14
  • Generative reward modeling reframes reward estimation as language modeling and can use chain-of-thought reasoning and voting for more robust ranking.Source: B RELATED WORK · p. 14

C DISCUSSION AND LIMITATIONS

The discussion explains why CAPO’s GenPRM can provide grounded verification, while acknowledging computational overhead, segmentation choices, and prompt sensitivity as practical boundaries.

Reliability and the Capability of the LLM-as-a-GenPRM

  • Using a stronger GenPRM than the policy model and supplying the ground-truth answer supports error localization and more reliable verification.Source: C DISCUSSION AND LIMITATIONS · p. 15

Computational Considerations in Online RL

Step Segmentation

Prompt Sensitivity of the GenPRM

  • GenPRM performance may depend on prompt wording, although the selected prompt was reported effective across models and tasks.The authors state that the prompt was not heavily tuned and included instructions to assess each step independently.Source: C DISCUSSION AND LIMITATIONS · pp. 15–16

D IMPLEMENTATION DETAILS

The implementation uses Math-Verify, delimiter-based step segmentation, model-specific training configurations, GenPRM sampling settings, and evaluation tables, while Figure 4 tracks training dynamics.

  • Math-Verify supplies the automatic rule-based judgment used in the implementation.Source: D IMPLEMENTATION DETAILS · p. 16
  • Generated responses are split at double-newline delimiters and wrapped in step markers before constructing the GenPRM prompt.Source: D IMPLEMENTATION DETAILS · p. 16
  • RL and SFT hyperparameters are documented in Table 5, using widely accepted settings for the configurations.Source: D IMPLEMENTATION DETAILS · p. 16, Table 5 · p. 16
  • Table 6 reports sampling parameters for the LLM-as-GenPRM.Source: Table 6 · p. 16
  • Tables 7 and 8 compare Pass@1 across methods and voting mechanisms for Llama-3-1B-Instruct and Qwen2.5-7B.Both tables cover mathematical in-distribution and general out-of-distribution reasoning tasks, using N=4 critiques for CAPO with voting.Source: Table 7 · p. 17, Table 8 · p. 17
  • Figure 4 plots accuracy, response length, and mean per-token reward over training steps for CAPO on Qwen2.5-1.5B.Source: Figure 4 · p. 17

E ADDITIONAL EXPERIMENTS

Additional experiments show that CAPO’s process-oriented signal differentiates equally rewarded samples, while voting strictness and reward balance determine how reliably that signal supports learning. Across analyses, CAPO improves performance and robustness, but outcome-based alternatives and some auxiliary inputs provide weaker or potentially misleading benefits.

Process and outcome rewards

  • A small process signal can help later optimization, whereas emphasizing process rewards too early can encourage verbosity without improving outcomes.The reported best trade-off includes C=2 and P=0.1, while early process-focused training may slow optimization.Source: E ADDITIONAL EXPERIMENTS · p. 17
  • When outcome rewards plateau, process rewards differentiate samples with identical outcomes and prioritize trajectories with better reasoning.This improves sample efficiency and keeps training effective after many responses receive the same outcome reward.Source: E ADDITIONAL EXPERIMENTS · p. 18

Voting mechanisms

  • All CAPO variants consistently outperform the GRPO baseline in mean scores across mathematical and general reasoning tasks.The authors attribute these gains to reliable, fine-grained credit assignment across aggregation strategies.Source: E ADDITIONAL EXPERIMENTS · p. 18, Table 11 · p. 20, Table 12 · p. 20
  • For smaller models, Intersection voting gives the strongest performance by requiring all critiques to agree, favoring high-precision reasoning signals.The authors connect this strict filtering to stronger out-of-distribution performance for smaller Qwen2.5-1.5B and Llama-3-1B-Instruct models.Source: E ADDITIONAL EXPERIMENTS · pp. 18–19
  • For the larger Qwen2.5-7B model, majority and union voting perform best because looser consensus preserves exploratory reasoning despite occasional verifier disagreements.The authors frame voting as a precision–recall trade-off that varies with model capability.Source: E ADDITIONAL EXPERIMENTS · p. 19

Ground-truth solutions

  • Adding ground-truth solutions does not reliably improve CAPO and can slightly degrade general reasoning when their distribution differs from policy-generated solutions.The authors suggest distributional mismatch can introduce noise into GenPRM judgments.Source: Table 9 · p. 18, E ADDITIONAL EXPERIMENTS · p. 19

GenORM comparison

  • Increasing GenORM sample count produces unstable, relatively small GRPO gains compared with CAPO’s LLM-as-GenPRM approach.On Math benchmarks, AMC performance is unstable and Math Mean gains are marginal because outcome verification does not provide fine-grained credit.Source: Table 10 · p. 18, Figure 5 · p. 19, E ADDITIONAL EXPERIMENTS · p. 20

Case studies

  • CAPO’s case study shows that simpler integer-based reasoning avoids unnecessary fractions, reducing cognitive load and susceptibility to arithmetic or conceptual errors.The comparison illustrates why process quality matters even when binary rewards treat different correct or incorrect trajectories identically.Source: E ADDITIONAL EXPERIMENTS · p. 21

F PROMPT

This section presents prompts for LLM-as-GenPRM to assess solution correctness and identify erroneous steps, as well as prompts and examples for SFT/RL training. It also illustrates how a propagated sign error leads to an incorrect solution and contrasts this with a more efficient roots–coefficients approach.

Generative Reward Model Prompt

  • The generative reward prompt asks a math teacher to judge correctness against ground truth, independently verify each step, and list erroneous step indices in a fixed format.The prompt requires direct output with a Yes/No judgment and an <incorrect steps> list.Source: F PROMPT · p. 22

Training Prompt

  • The training prompt instructs models to reason step by step and place the final answer inside \boxed{} for SFT and RL training.It combines a system instruction with a user-provided question.Source: F PROMPT · p. 22

Prompt Demonstration

  • For the line-angle example, the demonstrated solution computes a zero dot product and concludes that the lines are perpendicular, with θ = 90°.The supplied worked response derives direction vectors, applies the dot-product formula, and reports the final angle.Source: F PROMPT · p. 23

Case Studies

  • The prompt examples distinguish responses with correct answers and reasoning from responses with correct answers but incorrect reasoning, wrong answers, or wholly wrong reasoning.The examples include the die-factor problem, where correct responses give 20, while erroneous responses give 19 or 50.Source: F PROMPT · pp. 24–27

LLM-as-GenPRM on Correct Responses

  • When evaluating a correct response, LLM-as-a-GenPRM returns a Yes judgment and an empty incorrect-step list because the derivation and final answer are correct.The example verifies a perfect-square trinomial solution whose final value is a = 9.Source: F PROMPT · pp. 28–29

LLM-as-GenPRM on Wrong Responses

  • A separate wrong-response case applies the same judgment-and-error-list format to a quadratic-root problem, with the supplied passage beginning the stepwise evaluation.The prompt includes the ground-truth solution and a candidate response for checking, but the excerpt ends during that evaluation.Source: F PROMPT · p. 29
  • The solution is judged incorrect, with steps 5–11 identified as erroneous.The critique specifically marks steps 5 through 11 as incorrect.Source: F PROMPT · p. 30
  • A missing negative sign in step 5 causes the error to propagate through the subsequent steps and produce an incorrect value for v.The term 21p´21´301q 10 should have a negative sign in the original expression.Source: F PROMPT · p. 30
  • The resulting v does not match ground truth because the provided method is unnecessarily complex and error-prone; the correct approach uses the quadratic roots–coefficients relationship.The critique states that this relationship finds v more efficiently.Source: F PROMPT · p. 30

Related papers

Loading 2508.02298v4…