PaperlayerView exact revision on arXiv

Source-linked AI summary

Semifactual Credit-Augmented Policy Optimization

Junshu Pan, Zhizhang Fu, Shulin Huang, Yiran Ding, Zifan Cheng, Wenqi Shao, Qiaosheng Zhang, Yue Zhang

arXiv:2609.40360v1cs.LGcs.AIcs.CL

TL;DR

RLVR improves LLM reasoning, but models remain sensitive to task-irrelevant prompt features, and GRPO assigns the same outcome-derived advantage to every response token. The paper introduces SCAPO, using semifactual stability to adjust token-level credit for unstable tokens. Across two Qwen3 model scales, SCAPO improves AIME accuracy over GRPO and achieves the best results on most evaluated benchmarks.

  • Problem

    LLMs trained with RLVR remain sensitive to irrelevant information and input perturbations, while how training factors affect RLVR generalization remains underexplored.

    Source: §1 · p. 2

  • Method

    SCAPO measures fixed-response token probability drift under semifactual prompt interventions and uses normalized stability scores to reduce advantages for relatively unstable tokens during early GRPO training.

    Source: §1 · p. 2

  • Results

    SCAPO outperforms GRPO on all reported benchmark metrics and achieves the best results on most benchmarks among compared RLVR methods; AIME 2024–2026 accuracy gains are +5.63 and +4.17 points at 4B and 1.7B, respectively.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Semifactual stability can serve as a training signal that complements outcome rewards for finer-grained token-level credit assignment in RLVR.

    Source: §1 · p. 3, §6 · p. 9

    Training experiments cover mathematical reasoning with dense Qwen3 models of 1.7B and 4B parameters trained on DAPO-Math-17K; effectiveness at larger scales and across broader training domains remains unevaluated.

    Source: A LIMITATIONS · p. 13

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning capabilities of large language models (LLMs), yet their predictions remain sensitive to task-irrelevant prompt features. We investigate this sensitivity through semifactual prompt interventions that preserve the underlying problem and its answer. Our analysis reveals substantial variation in token-level sensitivity and shows that suppressing high-drift token candidates during decoding improves reasoning accuracy without updating model weights. These findings highlight a limitation of Group Relative Policy Optimization (GRPO), which assigns the same outcome-derived advantage to every response token and may reinforce potential spurious dependence alongside useful reasoning. Motivated by this observation, we introduce Semifactual Credit-Augmented Policy Optimization (SCAPO), a causally inspired variant of GRPO that incorporates semifactual stability into token-level credit assignment. SCAPO measures token probability drift for fixed responses under semifactual interventions and uses normalized stability scores to reduce advantages for relatively unstable tokens during early training, while granting no additional credit for stability alone. On Qwen3-4B-Base and Qwen3-1.7B-Base, SCAPO improves AIME 2024-2026 accuracy over GRPO by 5.63 and 4.17 percentage points, respectively. At both model scales, SCAPO achieves the best results on most evaluated mathematics benchmarks and all evaluated out-of-distribution benchmarks among the compared methods. These results suggest that semifactual stability provides an effective training signal for improving reasoning and generalization through finer-grained credit assignment in RLVR. The code is available at https://github.com/DtYXs/SCAPO.

1 INTRODUCTION

RLVR can improve reasoning while leaving models sensitive to task-irrelevant prompt features, and GRPO’s shared token advantage does not account for that sensitivity. The paper introduces SCAPO, which uses semifactual stability for token-level credit assignment and improves benchmark performance over GRPO.

1 INTRODUCTION

  • GRPO assigns the same outcome-derived advantage to every valid response token, potentially reinforcing prompt sensitivity alongside useful reasoning.The paper frames this as a limitation because token-level sensitivity can vary even when the problem and answer are preserved.Source: §1 · p. 2
  • Semifactual interventions preserve the mathematical problem and answer while altering task-irrelevant prompt features to probe token-level sensitivity.The paper presents this as a way to study potential spurious dependence in LLM inference.Source: §1 · p. 2
  • SCAPO incorporates semifactual stability into GRPO token-level credit, reducing advantages for relatively unstable tokens without granting extra credit for stability alone.It measures fixed-response token probability drift under semifactual prompt interventions, then applies the credit augmentation early in training before continuing with standard GRPO.Source: §1 · p. 2
  • SCAPO improves AIME 2024–2026 accuracy over GRPO by +5.63 and +4.17 points on Qwen3-4B-Base and Qwen3-1.7B-Base, respectively.It outperforms GRPO on all evaluated benchmarks and achieves the best results on most competition-level mathematics benchmarks and the highest accuracy on out-of-distribution benchmarks at both scales.Source: §1 · pp. 2–3

2 TOKEN-LEVEL SEMIFACTUAL SENSITIVITY

The section measures token probability drift under answer-preserving prompt changes and tests whether suppressing high-drift candidates improves decoding. It reports heterogeneous sensitivity across tokens and improved accuracy with model weights unchanged.

2 TOKEN-LEVEL SEMIFACTUAL SENSITIVITY

  • Semifactual perturbations alter incidental prompt features while preserving the underlying mathematical problem and final answer.The four types are paraphrase, minor typo noise, irrelevant scenario wrapping, and appended irrelevant context; GPT-5.5 generates them under instructions to preserve problem quantities, expressions, constraints, and requested quantity.Source: §2 · p. 3
  • The diagnostic samples one response per original prompt from frozen Qwen3-4B-Base on 1,000 questions selected from DAPO-Math-17K.Token drift compares probabilities assigned to the identical sampled token under the original and perturbed prompts; the measurement uses a bounded-symmetric distance and averages across four perturbations.Source: §2 · p. 3
  • Reflection markers and discourse connectives show mean drifts of 2.74× and 2.32× the overall mean, while mathematical symbols and numbers show 0.47× and 0.37×.Across token positions, 12.82% show no recorded change, while nonzero drift magnitudes span several orders of magnitude.Source: §2 · p. 4
  • 14.2 percentage points: suppressing high-drift candidates raised Qwen3-4B-Base accuracy on the 1,000-question panel from 15.8% to 30.0%, with weights unchanged.At each decoding step, the method masks a prefix of non-EOS candidates ranked by drift, keeping cumulative probability mass at or below 0.8 and retaining EOS.Source: Figure 3 · p. 4, §2 · p. 4

3 SEMIFACTUAL CREDIT-AUGMENTED POLICY OPTIMIZATION

SCAPO combines group-level outcome advantages with semifactual stability signals to assign token-level credit. It probes fixed responses under perturbed prompts, penalizes relatively unstable tokens, and applies the augmentation only during early training.

3.1 PRELIMINARIES: GROUP RELATIVE POLICY OPTIMIZATION

  • GRPO applies the same outcome-derived advantage to every valid response token, so it cannot distinguish dependence on task-irrelevant prompt features.The group advantage is standardized using the mean and standard deviation of group rewards.Source: §3.1 · p. 4

3.2 FIXED-RESPONSE SEMIFACTUAL PROBING

  • SCAPO teacher-forces each fixed sampled response under the original and semifactual prompts, measuring token-probability drift while keeping tokens and prefixes unchanged.The rollout policy remains fixed during each probe and is updated as training proceeds.Source: §3.2 · pp. 5–6

3.3 GROUP-RELATIVE STABILITY ESTIMATION

  • SCAPO standardizes drift within each prompt group and perturbation type, aggregates the scores, and retains only negative relative-stability values.Negative values identify relatively unstable tokens; stability alone earns no credit because it does not imply correctness.Source: §3.3 · p. 6

3.4 POLICY OPTIMIZATION WITH SEMIFACTUAL CREDIT

  • SCAPO adds a detached, scaled stability correction to each GRPO advantage, lowering credit for unstable tokens without treating stability as correctness.Augmentation is applied during an initial phase, after which optimization continues with standard GRPO.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments compare SCAPO with representative RLVR methods on two Qwen3 base models using matched training and evaluation settings. SCAPO improves mathematical and out-of-distribution results, while ablations and runtime measurements examine its correction design and cost.

4.1 SETUP

  • Training uses DAPO-Math-17K and Qwen3-4B-Base or Qwen3-1.7B-Base, with evaluation on competition-level mathematics and out-of-distribution benchmarks.Comparisons match training data, reward function, prompt template, optimizer, rollout settings, step budget, and evaluation protocol within each model scale.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • AIME accuracy increases by 5.63 and 4.17 percentage points over GRPO on Qwen3-4B-Base and Qwen3-1.7B-Base, respectively; SCAPO leads six of eight metrics at 4B and all eight at 1.7B.It also has the highest average accuracy at both scales.Source: §4.2 · p. 7, Table 1 · p. 8
  • SCAPO achieves the highest accuracy on all three out-of-distribution benchmarks at both scales, including GPQA-Diamond gains of 6.19 and 3.83 percentage points over GRPO.The gains extend to perturbed mathematical problems and scientific reasoning beyond the training domain.Source: §4.2 · p. 7, Table 1 · p. 8

4.3 ABLATION STUDIES

  • Negative-only corrections achieve the highest average accuracy across five competition-level benchmarks, supporting reduced credit for unstable tokens without rewarding stability alone.The ablation compares negative-only, all-sign, and positive-only corrections.Source: Table 2 · p. 8, §4.3 · p. 9

4.4 COMPUTATIONAL EFFICIENCY

  • Semifactual probing and credit construction account for 1.8% of measured training runtime in the 4B experiment.Probes reuse sampled responses through teacher forcing and run only during the initial credit-augmentation phase.Source: §4.4 · p. 9

5 RELATED WORK

Prior RLVR work advances reasoning through group-relative optimization and refinements to normalization, sampling, weighting, clipping, and exploration, but response-level advantages do not distinguish reasoning tokens. SCAPO adds token-level credit from semifactual stability alongside verifiable outcome rewards.

5 RELATED WORK

  • Response-level advantages in RLVR remain too coarse to distinguish individual reasoning tokens.Prior work also studies reasoning-path coverage, distinguishing sampling efficiency from expansion of a model’s reasoning repertoire.Source: §5 · p. 9
  • Prior fine-grained credit methods use process annotations, outcome labels, Monte Carlo rollouts, token statistics, or perturbation- and gradient-based attribution.These approaches guide token selection, weighting, or policy updates.Source: §5 · p. 9
  • SCAPO derives negative-only advantage corrections from fixed-response sensitivity to semifactual prompt interventions, using teacher-forced token probabilities rather than output-side attribution.It retains verifiable outcome rewards while refining credit at the token level.Source: §5 · p. 9

6 CONCLUSION

SCAPO uses semifactual sensitivity to reduce credit for relatively unstable tokens, and it outperforms GRPO across reported metrics at two Qwen3 scales. The work also reports reproducibility resources, while its experiments remain bounded to dense Qwen3 models trained on mathematical reasoning.

6 CONCLUSION

  • SCAPO outperforms GRPO on all reported benchmark metrics at both Qwen3 scales and leads on most benchmarks among compared RLVR methods.The conclusion frames semifactual interventions as diagnostic probes and training signals; future work will test larger models, broader architectures, and domains beyond mathematics.Source: §6 · p. 9

AI USE STATEMENT

  • The authors used generative AI for perturbation generation, writing edits, grammar correction, code debugging, and script development.They state that the authors take responsibility for the work’s final content.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The authors provide code, training and evaluation data, and detailed hyperparameters; the datasets and pretrained models are publicly available.The code is built on the open-source EasyR1 framework.Source: REPRODUCIBILITY STATEMENT · p. 10

A LIMITATIONS

  • Experiments cover mathematical reasoning with dense Qwen3 models of 1.7B and 4B parameters trained on DAPO-Math-17K; larger scales and broader domains remain unevaluated.GPQA-Diamond provides evidence of transfer to scientific reasoning, while other architectures, including MoE and hybrid models, remain future work.Source: A LIMITATIONS · p. 13

B SEMIFACTUAL PERTURBATIONS AND TOKEN-LEVEL ANALYSIS

The appendix describes how semifactual prompts are constructed and used to measure token sensitivity, then details diagnostic and decoding procedures. Case studies illustrate token-probability changes under answer-preserving perturbations, and the filtering procedure masks high-drift candidates under a probability-mass cap.

B.1 PERTURBATION CONSTRUCTION

  • Four perturbation types—paraphrase, typo noise, scenario wrapping, and irrelevant context—alter incidental prompt features while preserving the problem and answer.The prompt-generation instructions prohibit changing quantities, mathematical facts, hints, or solution-relevant context.Source: §B.1 · pp. 13–15, Table 3 · p. 14

B.2 DIAGNOSTIC SETUP

  • On a fixed 1,000-question DAPO-Math-17K subset, frozen Qwen3-4B-Base responses are teacher-forced under original and four perturbed prompts to measure token drift.One response is sampled per original prompt; the diagnostic yields 870,586 token drifts.Source: §B.2 · p. 15

B.3 TOKEN TAXONOMY AND STATISTICS

  • Response tokens are assigned to eight mutually exclusive categories using lexicons for reflection markers and discourse connectives, with reflection matches taking precedence.Table 4 reports category shares and mean drifts across 870,586 response-token positions, relative to an overall mean drift of 0.01640.Source: §B.3 · p. 15, Table 4 · p. 16

B.4 COMPLETE-RESPONSE CASE STUDIES

  • Figures 6 and 7 show correct-answer cases where individual response-token probabilities change under answer-preserving prompt perturbations.Darker shading represents greater mean probability drift across the four perturbations.Source: §B.4 · p. 16, Figure 6 · p. 17, Figure 7 · p. 18

C.1 DECODING SETUP

  • The decoding comparison uses frozen Qwen3-4B-Base on the same 1,000 questions, with temperature 1.0, top-p 1.0, and an 8,192-token limit.Each question receives one independently generated response per strategy; the cumulative clean-policy probability mass cap is 0.8.Source: §C.1 · p. 19

C.2 FILTERING PROCEDURE

  • At each step, candidates are ranked by decreasing mean drift and masked up to a 0.8 probability-mass cap, while EOS is retained.The cap limits removed probability mass, not the fraction of tokens masked; drift and the masked set are recomputed for each prefix.Source: §C.2 · p. 19

C.3 RESULTS

Filtering on the 1,000-question frozen-model evaluation produces more incorrect-to-correct than correct-to-incorrect answer transitions, for a reported gain of 14.20 percentage points.

C.3 RESULTS

  • 14.20 percentage points: filtering yields 202 incorrect-to-correct and 60 correct-to-incorrect answer transitions on 1,000 questions.Table 5’s lower block reports question counts for the four combinations of baseline and filtered answer correctness.Source: §C.3 · p. 19, Table 5 · p. 19

D OPTIMIZATION ANALYSIS

The optimization analysis bounds the effect of finite-duration credit augmentation under a stochastic-gradient model and shows its contribution decays after augmentation ends. The training setup uses rule-based rewards and specifies shared token-mean optimization and batch settings.

D OPTIMIZATION ANALYSIS

  • Replacing each advantage by its stability-corrected value changes the group gradient by at most λH under the stated gradient and importance-ratio bounds.Token-mean aggregation across groups preserves this bound.Source: D OPTIMIZATION ANALYSIS · pp. 19–20
  • The finite-duration augmentation theorem considers any T ≥ 1 with 0 < η ≤ 1/L and schedule Eq. 11, defining ΔF = Fsup − E F(θ1) and mT = min(T, N0).It assumes F is L-smooth and bounded above by Fsup.Source: D OPTIMIZATION ANALYSIS · p. 20
  • With fixed λ0 and N0, the augmentation contribution to the average stationarity bound decays as N0/T after augmentation ends.T is an arbitrary observation horizon, not a prescribed training budget.Source: D OPTIMIZATION ANALYSIS · p. 20

E.1 TRAINING HYPERPARAMETERS

  • A rule-based verifier assigns reward 1 to a correct boxed answer and 0 otherwise, with no separate format reward.All models use the stated mathematical-reasoning prompt template.Source: §E.1 · p. 20
  • Each rollout batch contains 128 prompt groups, optimized once in mini-batches of 64 groups, for two policy-optimization steps per rollout step.All methods use token-mean loss reduction for the mathematical reasoning tasks.Source: §E.1 · pp. 20–21

E.2 EVALUATION PROTOCOL AND TRAINING CURVES

The evaluation protocol specifies the datasets, sampling settings, and scoring procedure used for final checkpoints, alongside periodic rule-based validation. The detailed results section reports yearly accuracy underlying the main table.

E.2 EVALUATION PROTOCOL AND TRAINING CURVES

  • Final checkpoints were evaluated with identical prompts and sample counts across methods, temperature 0.7, top-p 0.9, and a maximum response length of 16,384 tokens.Mathematical benchmark responses were scored with Math-Verify.Source: §E.2 · p. 22
  • Omni-Math evaluation uses its 2,821-problem subset designed for rule-based evaluation.Table 7 summarizes evaluation datasets and sample counts; its caption notes all 24 answer-option permutations for GPQA-Diamond.Source: §E.2 · pp. 21–22, Table 7 · p. 22
  • Rule-based validation is performed before training and every 40 policy-optimization steps, with Figure 1 presenting the corresponding accuracy curves.Source: §E.2 · p. 22

F.1 DETAILED BENCHMARK RESULTS

  • Table 8 reports yearly accuracy percentages underlying Table 1, marking best results in bold and second-best results with underlining.Source: §F.1 · p. 22, Table 8 · p. 22

F.2 PASS@k RESULTS

SCAPO achieves the highest Pass@128 on AIME and AMC at both model scales. The reported Pass@k curves estimate the probability of at least one correct answer within k attempts.

F.2 PASS@k RESULTS

  • SCAPO achieves the highest Pass@128 on AIME and AMC at both model scales.Source: §F.2 · p. 23, Figure 8 · p. 23
  • Pass@k measures the probability of at least one correct answer within k attempts, estimated from 128 responses per problem.The evaluation uses the same sampling and scoring settings as the main evaluation.Source: §F.2 · p. 23
Loading 2609.40360v1…