Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning

Motivation

• GRPO broadcasts a single group-normalized advantage to every token, overlooking that only some reasoning tokens determine correctness.This coarse assignment can increase gradient variance and accelerate entropy collapse.Source: §1 · p. 1
• Entropy-based token weighting may misallocate credit because uncertainty does not necessarily reflect outcome-relevant importance.High entropy can occur for stylistic or lexically ambiguous tokens with little effect on correctness.Source: §1 · pp. 1–2

OAR

• OAR scores tokens by how much perturbing them shifts the model’s final-answer distribution, using that distribution as a practical outcome surrogate.The surrogate avoids directly attributing sparse, non-differentiable verifier rewards to individual tokens.Source: §1 · p. 2
• OAR-P uses counterfactual perturbations for high-fidelity attribution, whereas OAR-G uses an efficient gradient-based proxy for scalable online training.The two strategies trade attribution fidelity against computational efficiency.Source: §1 · p. 2

Advantage reshaping

• The Bi-Level mechanism suppresses low-impact tokens, boosts pivotal tokens, and renormalizes weights to preserve overall advantage mass.This design targets sharper learning signals without changing the effective total advantage scale.Source: §1 · p. 2

Results

• OAR consistently outperforms strong GRPO baselines, with OAR-P setting the performance upper bound and OAR-G retaining most gains at minimal computational cost.The reported results come from mathematical reasoning benchmarks.Source: §1 · p. 2

GRPO

• GRPO samples multiple completions, normalizes their rewards within the group, and uses the resulting advantage for policy optimization.The method avoids training a learned value function.Source: §2.1 · p. 2
• The same scalar advantage is applied to all token-level log-probability terms in each sampled response.This creates sequence-level rather than token-specific credit assignment.Source: §2.1 · p. 2
• GRPO then optimizes a PPO-style clipped surrogate objective using importance ratios.The clipped objective is the policy-optimization component described in the preliminaries.Source: §2.1 · p. 2

Intrinsic signals

• Entropy-based variants redistribute advantages with intrinsic token-uncertainty signals and entropy-dependent bonuses.The shaping rule includes gradient stopping for stability.Source: §2.2 · pp. 2–3
• Intrinsic entropy proxies may emphasize uncertain tokens that have little effect on correctness, while additive shaping can change sequence-dependent update scales.Renormalization or retuning is needed to address the altered advantage mass.Source: §2.2 · p. 3

OAR framework

• OAR equips GRPO with token-level credit assignment through OAR-P perturbation attribution and OAR-G gradient-based approximation.Source: §3 · p. 3

OAR-P

• OAR-P masks each reasoning token, recomputes the final prediction distribution, and measures the resulting KL divergence as token importance.Large divergence indicates an outcome-critical token, whereas near-zero divergence indicates syntactic filler; computing scores for length L requires L additional forward passes.Source: §3.1 · p. 3

OAR-G

• OAR-G injects Gaussian noise into token embeddings and uses Gradient × Input sensitivity to approximate outcome effects with a single backward pass.This replaces O(L) counterfactual forward passes and is computationally negligible compared with generation.Source: §3.2 · p. 4

Importance normalization

• Raw importance scores are log-transformed and min-max normalized within each sequence before advantage reshaping.Source: §3.2 · p. 4

Bi-Level advantage reshaping

• Bi-level reshaping suppresses low-importance tokens, boosts high-importance tokens, and sum-preserving renormalization keeps total advantage mass unchanged.The mechanism redistributes credit across tokens rather than globally rescaling advantages.Source: §3.3 · p. 4

Training objective

• By integrating outcome-grounded credit into the surrogate loss, OAR encourages refinement of critical reasoning steps while remaining robust to high-variance noise in long mathematical traces.Source: §3.3 · p. 4

Baselines

• The comparison includes vanilla GRPO, random credit, entropy-based weighting, and KTAE token-level advantage corrections.Source: §4.2 · pp. 5–6

Reasoning performance

• OAR-P improves Qwen2.5-7B AIME25 P@1 from 9.4 to 11.8 and the overall average from 51.3 to 53.7.Source: §4.3 · p. 6
• OAR-P raises the Qwen2.5-Math-7B overall average from 57.3 to 59.5.Source: §4.3 · p. 6

Training dynamics

• Across training runs, OAR improves optimization stability, reaches higher rewards with fewer steps, and avoids premature entropy collapse.Source: §4.3 · p. 6

Scaling to different sizes

• On Qwen2.5-Math-1.5B, both OAR-G and OAR-P consistently improve over baselines.Source: §4.3 · p. 6, Table 2 · p. 6

Causal-token identification

• The counterfactual Oracle labels a token as causal when replacing it flips final-answer correctness; otherwise it is non-causal noise.Source: §5.1 · p. 6

Alignment visualization

• OAR-P assigns high importance to outcome-critical symbols and downweights stylistic or redundant tokens, qualitatively matching the Oracle mask.Source: §5.1 · p. 6

Quantitative correlation

• Across top-K% percentiles, OAR-P and OAR-G achieve higher Oracle causal-token recall than entropy weighting, with OAR-P consistently best.Source: §5.1 · p. 7

Computational trade-off

• Normalized training cost is 1.0× for GRPO, 1.4× for OAR-G, and 4.2× for batched OAR-P; naive serial OAR-P costs 18.7×.Source: §5.2 · p. 7, Figure 7 · p. 7

Bi-level gating

• Larger gating thresholds generally produce sparser credit allocations than vanilla GRPO.ESS ratio measures weight uniformity, while top-10% advantage mass measures concentration in the highest-weighted tokens.Source: §6.1 · p. 7
• Moderate thresholding performs best: boost-only gating can collapse, suppress-only gating is stable but inefficient, and balanced gating achieves the best accuracy.Source: §6.1 · p. 7

Outcome definitions

• OAR uses an outcome probe to summarize final-answer prediction, and different probes can produce different token rankings and credit reallocations.Source: §6.2 · pp. 7–8
• The ablation compares last-token logits, answer-span mean logits, and answer-span joint likelihood as outcome operators.Source: §6.2 · p. 8

Outcome-definition results

• Answer-span mean logits achieve the highest Oracle top-K% recall and best downstream training performance, followed by last-token logits, while answer-span joint likelihood performs worst.Distribution-based operators can capture outcome-distribution shifts that scalar likelihood may miss.Source: §6.2 · p. 8

Reinforcement learning with verifiable rewards

• RLVR uses automatically checkable outcome signals, while GRPO stabilizes large-scale training by group-normalizing verifier rewards.Source: §7 · p. 8

Fine-grained credit assignment

• Prior fine-grained credit methods use attention dynamics, response-level confidence, or token-level certainty and correctness associations to redistribute advantages.Source: §7 · p. 8

• OAR improves credit assignment in GRPO for long-horizon reasoning by redistributing sequence-level advantages according to outcome contribution.Source: §8 · p. 8
• OAR-P uses counterfactual attribution, whereas OAR-G uses a lightweight gradient-based proxy to estimate token importance.Source: §8 · p. 8
• Experiments on mathematical reasoning benchmarks show consistent gains over strong GRPO baselines.Source: §8 · p. 8

• OAR’s answer-distribution proxy can be imperfect when distributional changes are weakly coupled with verifier acceptance.Source: Limitaions · p. 9
• OAR-P’s masking intervention may introduce distribution shift compared with in-distribution edits.Source: Limitaions · p. 9
• Experiments focus on mathematical reasoning with verifiable final answers, leaving open-ended and interactive settings for future extension.Source: Limitaions · p. 9
• Several closely related methods are not yet fully reproducible, limiting the breadth of current comparisons.Source: Limitaions · p. 9

• Additive proxy shaping can create sample-dependent update scales because its gradient magnitude depends on sequence length and high-proxy-token density.Source: §A.1.2 · p. 11
• The outcome probe is the model’s predictive distribution at an outcome-bearing position, and token influence is measured by the KL divergence between factual and masked distributions.Source: §A.1.3 · p. 11
• If token influence is approximately zero, masking it cannot substantially change the probability mass of any verifier-defined correct outcome set.Source: §A.1.3 · pp. 11–12
• Standard GRPO broadcasts one trajectory-level advantage to every token, injecting variance when nuisance tokens did not influence the reward.Source: §A.1.1 · p. 11, §A.1.4 · p. 12
• For a counterfactually non-influential token, OAR suppresses gradient-variance contribution relative to standard GRPO.Source: §A.1.4 · p. 12
• As ε →0, OAR’s gradient noise at a nuisance-token position vanishes, focusing optimization on tokens that causally affect the outcome.Source: §A.1.4 · p. 12
• Figure 8 compares OAR-G with an oracle causal mask and an entropy-based baseline on a representative reasoning trace.Source: §A.2 · p. 12, Figure 8 · p. 12
• The ESS ratio measures weight uniformity, while top-10% advantage mass measures how much absolute advantage is carried by the highest-weighted subset of valid tokens.Source: §A.3 · pp. 12–13