Source-linked AI summary
Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning
Qiangqiang He, Zhongheng Wu, ZiJian Wang
TL;DR
Long-CoT RLVR methods lack reliable token-level credit signals, motivating CSCR, which reallocates credit away from counterfactually sensitive tokens while preserving verifier direction and total credit. CSCR consistently outperforms GRPO across long-CoT mathematical reasoning models and benchmarks, with ablations favoring moderate downweighting.
Problem
GRPO broadcasts sequence-level advantages uniformly across tokens, while OPSD assumes privileged likelihood shifts provide reliable answer-aligned supervision despite unequal token contributions.
Method
CSCR counterfactually measures token sensitivity, attenuates credit for highly sensitive tokens, and renormalizes advantages to preserve verifier direction and the original credit budget.
Results
CSCR consistently outperforms GRPO across models and long-CoT mathematical reasoning benchmarks, while moderate downweighting is most effective and stronger modulation destabilizes optimization.
Takeaways & Limitations
Privileged likelihood shifts primarily reflect counterfactual sensitivity rather than token-level learning value, so credit should be reduced for highly sensitive tokens.
Takeaways & Limitations
OPSD relies on the assumption that privileged conditioning produces an answer-aligned target distribution.
Abstract
from arXiv · showhide
Reinforcement learning with verifiable rewards (RLVR) is central to improving long-CoT reasoning in large language models. Critic-free methods such as GRPO convert response-level rewards into advantages and uniformly broadcast them across tokens, overlooking their unequal contributions to the final outcome. On-policy self-distillation (OPSD) instead provides dense distributional supervision by minimizing the forward KL divergence between an unprivileged policy and a privileged self-teacher, implicitly assuming that the resulting likelihood shifts encode reliable answer-aligned information. We test this premise by fixing each sampled trajectory and re-scoring it under two opposing outcome conditions, one asserting correctness and the other incorrectness. Most affected tokens shift in the same direction under both conditions, with few sign reversals and substantial overlap in the induced optimization signals. Large shifts also concentrate on highly substitutable surface-form tokens, whereas tokens carrying problem-specific reasoning content are less sensitive. These findings show that privileged shifts fail to provide reliable answer-aligned directions, while their magnitudes primarily reflect counterfactual sensitivity rather than token-level learning value. Based on these observations, we propose Counterfactual Sensitivity Credit Reallocation (CSCR), a simple extension of GRPO that reduces credit for highly sensitive tokens and renormalizes token-level advantages to preserve both the original credit budget and verifier-determined direction. On long-CoT mathematical reasoning benchmarks, CSCR consistently outperforms GRPO baseline with the same number of policy updates. Targeted ablations further corroborate our diagnosis: privilege-induced directions are unreliable, moderate downweighting is most effective, and stronger modulation destabilizes optimization.
1 Introduction
The introduction shows that privileged likelihood shifts are sparse, commonly directed across opposing outcome conditions, and concentrated on surface-form tokens rather than problem-specific reasoning. It proposes CSCR, which reallocates credit away from sensitive tokens while preserving verifier-derived direction and the total credit budget.
- Introduction: RLVR improves long-CoT reasoning, but critic-free GRPO uniformly broadcasts group-level advantages across tokens despite their unequal contributions.OPD and OPSD provide dense supervision, yet teacher-derived likelihood shifts may not align with verifier correctness.
- Introduction: At λ=0.05, 82.4%–84.7% of tokens are insignificant under both privileged conditions, while only 8.2%–10.4% shift significantly under both.The analysis re-scores each fixed on-policy trajectory under opposing positive and negative privileged conditions.
- Introduction: Despite opposing outcome semantics, 91.4%–95.8% of jointly significant tokens have same-sign shifts, contradicting the polarity reversal expected from positive versus negative credit.This indicates privileged likelihood shifts do not reliably preserve verifier-derived answer polarity.
- Introduction: Approximately 58% of full-vocabulary forward-KL optimization magnitude overlaps across opposing conditions, while large shifts favor substitutable surface tokens over problem-specific reasoning tokens.Examples of sensitive surface-form tokens include Wait, Therefore, and But.
- Introduction: CSCR treats counterfactual shifts as token sensitivity, exponentially downweights highly sensitive tokens, and renormalizes advantages to preserve the credit budget and verifier-determined direction.Experiments report consistent gains over GRPO, with moderate downweighting effective and stronger modulation destabilizing optimization.
2 Diagnosing Privileged Likelihood Shifts
The analysis finds that opposing privileged conditions produce largely same-direction, overlapping likelihood shifts rather than reliable answer-aligned supervision. Large shifts primarily affect substitutable surface-form tokens, while problem-specific reasoning tokens are comparatively stable, indicating that shift magnitude reflects counterfactual sensitivity.
- OPSD assumption: OPSD treats privileged full-vocabulary probability differences as direct token-level optimization signals and assumes they are answer-aligned.The signal increases logits where privileged conditioning raises probability and decreases them where it lowers probability.
- Shift direction: 77.2%–79.1% of significant vocabulary shifts are same-sign across opposing conditions, versus 13.2%–15.3% opposite-sign shifts.The same-sign proportion remains stable as the threshold increases, while opposite-sign shifts decrease.
- Signal overlap: CPC ranges from 0.486 to 0.671, with mean and median both 0.583, showing substantial overlap in opposing optimization signals.Approximately 58% of combined signal magnitude shares vocabulary coordinates and directions under both conditions, unlike opposite GRPO advantages, whose CPC is 0.
- Token sensitivity: Large privileged shifts concentrate on highly substitutable surface-form tokens and discourse markers rather than problem-specific reasoning content.Frequently affected examples include But, So, Let, Wait, and Therefore, while near-zero shifts include digits, equality signs, addition signs, and parentheses.
- Token sensitivity: The positive- and negative-shift Top-100 lists share 80 tokens under the positive condition and 72 under the negative condition, so shift sign does not identify distinct answer-aligned token classes.Shift magnitude primarily identifies tokens responsive to counterfactual privileged conditioning.
- Robustness: Three additional counterfactual prompt pairs reproduce the same conclusions, supporting the robustness of the diagnosis across prompt formulations.The pairs compare correct versus incorrect solutions, affirmation versus denial of the gold answer, and concise outcome-only conditions.
3 Method
CSCR extends GRPO by reallocating credit away from highly counterfactually sensitive tokens while retaining the verifier-derived sequence advantage as the optimization direction. It attenuates sensitive positions and renormalizes their weights to preserve the trajectory-level credit budget.
- Method: CSCR assigns smaller token-level weights to tokens with higher counterfactual sensitivity while keeping the verifier-derived sequence advantage unchanged.The method reallocates, rather than reverses, the verifier-determined optimization direction.
- Method: GRPO uniformly broadcasts each trajectory-level advantage across all tokens in the response.This uniform token credit assignment is the baseline that CSCR modifies.
- Method: CSCR maps sensitivity scores to attenuating weights controlled by an onset threshold, decay rate, and maximum attenuation strength.Tokens below the threshold retain unit weight, while more sensitive tokens receive progressively smaller weights approaching 1 − γ.
- Method: The method normalizes weights within each response so that credit is redistributed across token positions without reducing the total trajectory-level credit budget.Normalization shifts credit from highly sensitive positions toward less sensitive ones.
4 Experiments
Experiments on long-CoT mathematical reasoning show that CSCR consistently improves over GRPO and self-distillation baselines across model scales and benchmarks. Training dynamics and ablations indicate that mild attenuation of sensitive tokens improves optimization, whereas excessive attenuation destabilizes it.
- Datasets: CSCR is trained on DAPO-17K and evaluated on five held-out competition-level mathematical reasoning benchmarks spanning varying difficulty.The benchmarks are AMC23, AIME24, AIME25, AIME26, and SMT-2025.
- Experimental setup: Experiments use Qwen3-1.7B and Qwen3-4B with full-parameter fine-tuning for 300 steps, sampling 256 trajectories per step.All methods use learning rate 2 × 10^-6; CSCR defaults are λ=0.05, α=10, and γ=0.2.
- Overall Performance: CSCR achieves the best result across all ten model–benchmark combinations and improves over each benchmark’s strongest competitor by 3.9 points on Qwen3-1.7B and 1.7 points on Qwen3-4B.The gains are consistent across both model scales and all five benchmarks.
- Training Dynamics: CSCR separates from GRPO early and maintains higher smoothed reward, while response length rises from approximately 10,000 to 11,500 tokens without length explosion or reward collapse.GRPO remains near 9,500–10,000 tokens, and CSCR stays below the 20,480-token generation limit.
- Ablation: Mild attenuation of sensitive tokens improves stability, whereas downweighting with γ∈{0.5, 1.0} collapses reward, entropy, and response length.Response length falls to approximately 6,000 and 2,000 tokens for γ=0.5 and γ=1.0, respectively.
5 Conclusion
The study evaluates whether privileged likelihood shifts provide reliable token-level supervision for long-CoT RLVR. Across 400 on-policy trajectories, counterfactual re-scoring finds predominantly same-direction shifts under opposing outcome conditions and substantially overlapping forward-KL signals.
- Conclusion: 400 on-policy trajectories were analyzed by separating privileged shift direction from magnitude at sampled-token and full-vocabulary levels under fixed prefixes.The analysis directly tests whether privileged likelihood shifts provide reliable token-level supervision for long-CoT RLVR.
- Conclusion: Opposing outcome conditions induce predominantly same-direction shifts and substantially overlapping forward-KL signals under counterfactual re-scoring.This pattern challenges the reliability of privileged likelihood shifts as answer-aligned token-level supervision.
- Conclusion: Large privileged shifts concentrate on substitutable surface-form tokens rather than necessarily indicating reliable token-level supervision.The supplied conclusion passage identifies this concentration as a central finding of the counterfactual analysis.
A Proof of Zero Counterfactual Perturbation Consistency for GRPO
At the on-policy reference point, opposite-sign advantages assigned to the same trajectory produce antiparallel GRPO logit-space optimization signals and zero Counterfactual Perturbation Consistency. This zero-overlap result also holds wherever PPO clipping is inactive, providing a reference for evaluating privileged forward-KL signals.
- Proof assumptions: The derivation analyzes only the clipped policy-gradient term and omits the KL regularizer.Advantages are treated as fixed with respect to the policy parameters, consistent with the standard GRPO policy-gradient objective.
- Antiparallel signals: Opposite-sign advantages make the two GRPO gradient-descent signals antiparallel at every token position, regardless of whether their magnitudes are equal.The comparison fixes the problem, trajectory, prefix, sampled token, probability ratio, and policy distribution.
- Zero CPC: The resulting signals have no same-coordinate, same-direction overlap, so Counterfactual Perturbation Consistency equals zero.This conclusion applies at the on-policy reference point and more generally in regions where PPO clipping is inactive.
B Top-50 Tokens under Counterfactual Privileged Prompts
Counterfactual privileged prompts most strongly perturb highly substitutable surface-form and discourse tokens, while problem-specific mathematical tokens are less sensitive. Sensitive-token identities and classes substantially overlap across positive and negative conditions, limiting semantic separation by shift sign.
- Token sensitivity patterns: Large-shift lists are dominated by discourse markers, transition cues, and substitutable surface-form tokens such as But, So, Let, Wait, Therefore, and Maybe.Near-zero lists are enriched with digits, variables, arithmetic operators, and mathematical notation.
- Token sensitivity patterns: Privileged conditioning primarily perturbs stylistic realization and discourse organization, while tokens encoding problem-specific calculations are substantially less sensitive.This contrast remains stable under both 𝑐pos and 𝑐neg, although neither category is semantically exclusive.
- Cross-prompt overlap: 47 of the Top-50 positive-shift tokens appear under both prompts, yielding a Jaccard similarity of 47/53 = 88.7%.The Top-50 negative-shift lists share the same 47 tokens and also have a Jaccard similarity of 88.7%.
- Cross-prompt overlap: Under 𝑐pos, Top-50 positive- and negative-shift lists account for 71,445 and 72,021 sampled-token occurrences, respectively, and share 36 token identities.Relative frequencies differ more under 𝑐neg, but substantial identity overlap remains stable across both conditions.
- Cross-prompt overlap: Shift sign provides limited semantic separation, whereas shift magnitude more consistently identifies general sensitivity to privileged conditioning.Positive and negative shifts affect broadly similar token classes within each privileged condition.
C Robustness across Alternative Counterfactual Privileged Prompts
The appendix tests whether the main findings depend on polarized privileged prompts by introducing three alternative prompt pairs while fixing the problem, trajectory, and token prefixes. These constructions isolate answer-level, outcome-only, and full-reasoning privileged information to assess whether likelihood-shift patterns persist across different conditioning forms and strengths.
- C.1 Alternative Counterfactual Privileged Prompt Constructions: Three additional counterfactual privileged-prompt pairs test robustness beyond the deliberately polarized prompts used in the main text.The constructions are summarized in Table C.1.
- C.1 Alternative Counterfactual Privileged Prompt Constructions: Each construction fixes the problem, sampled trajectory, and every token prefix, changing only the privileged context used for likelihood re-scoring.Within each pair, positive and negative conditions use the same prompt template and differ only in the privileged context.
- C.1 Alternative Counterfactual Privileged Prompt Constructions: The answer-level construction holds the ground-truth answer constant while describing it counterfactually as correct or incorrect.This isolates asserted answer-level correctness while controlling for answer content.
- C.1 Alternative Counterfactual Privileged Prompt Constructions: The mild outcome-only construction states whether the sampled trajectory reaches the correct final answer without revealing the answer or providing additional solution information.It does not reveal the answer, provide a reference solution, identify a specific error, or explicitly evaluate the question.
- C.1 Alternative Counterfactual Privileged Prompt Constructions: The three constructions cover privileged information at the answer, trajectory-outcome, and full-reasoning levels.They therefore span substantially different forms and strengths of counterfactual conditioning.
- C.1 Alternative Counterfactual Privileged Prompt Constructions: These alternatives determine whether observed likelihood-shift patterns persist across different counterfactual conditioning schemes rather than arising from the main experiments’ polarized wording.The comparison is designed to separate prompt-wording effects from broader robustness of the findings.
C.2 Diagnostic Results Across Alternative Prompt Constructions
Across three alternative counterfactual prompt constructions, opposing privileged conditions predominantly preserve same-sign likelihood shifts and overlapping optimization signals rather than producing answer-aligned reversals. Large shifts consistently concentrate on stylistic, locally substitutable tokens, while problem-specific mathematical tokens remain near-zero.
- Full-Vocabulary Shift Composition: Same-sign shifts dominate across alternative prompts, while opposite-sign shifts remain a minority even when complete correct and incorrect reasoning processes are supplied.For reference-solution conditioning, same-sign changes account for 63.7%–65.3% of the reference set versus 19.1%–21.2% for opposite-sign changes.
- Overlap in Counterfactual Optimization Signals: Answer-level feedback produces a mean CPC of 0.636, showing substantial same-coordinate, same-direction overlap between opposing privileged optimization signals.Nearly two thirds of the combined full-vocabulary signal magnitude overlaps despite reversing the asserted correctness judgment.
- Overlap in Counterfactual Optimization Signals: Reference-solution conditioning yields mean M = 0.138 and D = 0.064, yet mean and median CPC remain above 0.53.Supplying semantically distinct correct and incorrect reasoning processes increases perturbation magnitude without producing predominantly opposing optimization signals.
- Sensitive Token Types under Alternative Prompts: Large-shift lists are dominated by sentence starters, discourse markers, function words, and other locally substitutable surface-form tokens across all three prompt constructions.Examples include The, But, So, Therefore, Wait, Alternatively, Let, Maybe, and Check.
- Sensitive Token Types under Alternative Prompts: Near-zero lists instead contain digits, arithmetic operators, delimiters, variables, and other tokens directly participating in problem-specific mathematical expressions.Examples include 0–9, +, −, =, /, parentheses, square-root and exponent symbols, and variables such as 𝑋, 𝑌, 𝑁, and 𝐾.
E Representative Failure Cases of Shift-Directed GRPO
All three SD-GRPO variants collapsed through repetitive generation, with representative AIME24 cases showing plausible reasoning transition into indefinitely repeated spans beginning with “The.” The collapse reflects preferential positive reinforcement of this common surface-form token rather than a threshold-specific selection problem.
- Representative Failure Cases: All three SD-GRPO variants with λ∈{0.05, 0.1, 0.2} exhibit the same qualitative failure mode.The variants modify different fractions of token-level advantages, yet their failure behavior is qualitatively identical.
- Representative Failure Cases: Across all 30 AIME24 validation responses, collapsed runs eventually enter repetitive loops whose recurring span begins with “The.”Table E.1 likewise shows initially plausible reasoning transitioning into indefinitely repeated spans beginning with “The.”
- Representative Failure Cases: 8,328 occurrences of “The” satisfy z_pos > 0.1, versus 4,652 satisfying z_pos < −0.1 under the positive counterfactual condition.These privileged-shift statistics indicate that the token is more often associated with positive than negative selected-position advantages.
- Representative Failure Cases: Replacing selected verifier-derived advantages with sign(z_i,t)|A_i| more frequently assigns positive than negative advantages to “The,” reinforcing its generation.Autoregressive feedback then amplifies this surface-form preference into persistent repetition.
- Representative Failure Cases: The identical collapse across all three thresholds shows that SD-GRPO failure is not specific to selecting too many tokens.Even a small selected subset can preferentially reinforce a common surface-form token and trigger repetitive degeneration.