Source-linked AI summary
Unlocking Token Rewards via Training-Free Reward Attribution
Sitong Wu, Haoru Tan, Bin Xia, Xichen Zhang, Jingyao Li, Shaofeng Zhang, Xiaojuan Qi, Bei Yu, Jiaya Jia
TL;DR
Outcome and process rewards remain coarse, while existing token-level supervision faces a trade-off between costly, uncertain reward models and weakly aligned proxies. P2T uses a training-free gradient approximation to turn an existing process reward into token-level rewards. It consistently outperforms outcome reward across model families, reasoning paradigms, and modalities, including a reported +11.5% on AIME24 for Qwen2.5-Math-7B.
Problem
Outcome and process rewards provide coarse supervision, while existing token-level methods trade costly, uncertain learned rewards against cheap proxies weakly aligned with token quality.
Source: §1 · pp. 1–2Method
P2T estimates each token’s influence on an existing process reward through null-token replacement and a first-order gradient approximation, without training an additional reward model.
Source: §1 · p. 2Results
P2T consistently outperforms outcome reward across model families, reasoning paradigms, and modalities, including +11.5% on AIME24 for Qwen2.5-Math-7B.
Source: §1 · p. 2, §5 · p. 11Takeaways & Limitations
P2T’s token-level rewards can be integrated into standard RL algorithms such as GRPO without training an additional reward model.
Source: §5 · p. 11P2T’s effectiveness depends on PRM quality, and an unreliable PRM on the target domain may limit its benefits.
Source: §5 · p. 11
Abstract
from arXiv · showhide
In this paper, we propose an extremely efficient, training-free method to extract token-level reward signals directly from an existing deep reward model. Our core idea is to attribute the overall process reward to individual tokens by estimating each token's influence. This influence is defined as the change in the final macroscopic reward (e.g., the process reward) when a token is replaced with a semantically null token. Naively calculating this influence is computationally infeasible, requiring N forward passes through the PRM for an N-token sequence. We overcome this bottleneck by proposing a highly efficient gradient-based estimator. Specifically, we use a first-order Taylor approximation, which simplifies the influence calculation to the inner product of the difference between the token embedding and the null token embedding, and the gradient of the reward with respect to the token embedding. This requires only a single forward and backward pass. The resulting token-level rewards enable standard RL algorithms to perform precise credit assignment without requiring additional reward model training. Experiments on challenging reasoning benchmarks demonstrate that our method substantially improves policy optimization efficiency and enhances the generalization of LLM reasoning capabilities. Our P2T outperforms the outcome reward by +4.9% on MathVista for Qwen2.5-VL-7B-Instruct, and +11.5% on AIME24 for Qwen2.5-Math-7B. The code will be released via https://github.com/stonewst/P2T.
1 Introduction
Existing outcome and process rewards provide coarse supervision, while current token-level approaches trade training cost and uncertain labels against cheap but weakly aligned proxies. P2T attributes a coarse reward to tokens using a training-free gradient approximation, and improves reasoning results over outcome reward on reported benchmarks.
1 Introduction
- Outcome rewards and process rewards both remain coarse-grained, since even a PRM’s step-level scores are ambiguous across the many tokens in a step.The paper frames cost-effective, principled, and robust token-level reward estimation as an open question.Source: §1 · pp. 1–2
- Existing token-level methods either train reward models from noisy, low-fidelity synthetic labels or use cheap proxy metrics that are weakly aligned with token quality.Large-scale human annotation for a truly fine-grained token reward model is described as prohibitively expensive.Source: §1 · p. 1
- P2T estimates each token’s influence on the overall process reward by comparing the original token with a null token, using a first-order approximation computed with one forward and backward pass.It normalizes the influences and uses them to redistribute the coarse process reward into token-level rewards.Source: §1 · p. 2
- +4.9% on MathVista for Qwen2.5-VL-7B-Instruct and +11.5% on AIME24 for Qwen2.5-Math-7B versus outcome reward; P2T also reaches the outcome-reward counterpart’s best test accuracy with 3–4× fewer training steps.The paper also reports +14.2% on MinervaMath for DeepSeek-R1-Distill-Qwen-1.5B.Source: §1 · p. 2
- P2T converts process-level rewards into token-level rewards without training a dedicated token reward model, providing fine-grained supervision for reinforcement learning.The paper reports improved reasoning performance against sparse and other fine-grained reward methods.Source: §1 · p. 2
- The approximation reduces attribution complexity from O(N) to O(1) for an N-token sequence, supporting practical use in large-scale RL training.The reported experiments also show improved policy optimization stability and final reasoning accuracy across benchmarks.Source: §1 · p. 2
2 Method
P2T attributes a process reward to tokens by estimating each token’s influence, then uses normalized influence scores to distribute the step reward. A gradient-based approximation makes attribution efficient, and the resulting token rewards can be integrated into RL algorithms.
2.1 Definition of Token Attribution Score
- P2T estimates each token’s marginal influence by comparing the reward with and without that token’s embedding, replaced by a null embedding.A positive score marks a token whose removal lowers reward; a negative score marks one whose removal raises it.Source: §2.1 · pp. 2–4
2 Method
- P2T converts process-level PRM rewards into token-level rewards for standard RL algorithms to support precise credit assignment.Normalized token influences allocate more credit to influential tokens in correct steps and heavier penalties to influential tokens in erroneous steps.Source: Figure 1 · p. 3
2.2 Efficient Approximation
- A first-order gradient approximation replaces N reward-model passes with a single backward pass for all token embeddings.The gradient of reward with respect to each token embedding estimates its influence when that embedding is replaced by the null embedding.Source: §2.2 · p. 4
2.3.1 P2T Token Reward Calculation
- For each reasoning step, P2T computes token influence using the causal PRM reward for the prefix ending at that step.The score for a token in step s is calculated from R(E≤s).Source: §2.3.1 · p. 4
- P2T normalizes token influences within each step and redistributes its process reward by influence magnitude, retaining the process reward as a baseline.The baseline helps limit disturbance from noisy or imprecise attribution scores; a signed variant underperforms the magnitude-based reward.Source: §2.3.1 · pp. 4–5
2.3.2 GRPO with P2T Token Reward
- In GRPO, P2T token rewards are combined with the response-level outcome advantage, with α controlling the token guidance’s contribution.The default α is 1.0; when the PRM is less reliable, a smaller α or a gradual increase during training can help stabilize optimization.Source: §2.3.2 · p. 5
3 Experiments
Experiments compare P2T with outcome and dense token-level rewards across text-only and multimodal models, then examine design choices through ablations. Across matched settings, P2T improves benchmark performance and training efficiency.
3.1 Settings
- Experiments span Qwen and LLaMA models from 1.5B to 7B, use VisualPRM for multimodal models and ReasonFlux-PRM for text-only models, and report average Pass@1.Evaluation covers five multimodal benchmarks and seven math benchmarks for text-only models; training settings and data vary by model.Source: §3.1 · pp. 5–6
3.2.1 Comparison with Outcome Reward RL
- Across 40 model–benchmark pairs, GRPO+P2T outperforms GRPO in every case, with average gains ranging from +2.5% to +9.4%.The comparisons span model families, sizes, reasoning paradigms, and text-only and multimodal inputs.Source: Figure 2 · p. 6, Table 1 · p. 6, §3.2.1 · p. 7
- GRPO+P2T gains are larger on harder competition benchmarks (+4.0% to +12.2%) than on MATH-500 (+0.6% to +5.8%), and are strongest where outcome-based learning stalls.For DeepSeek-R1-Distill-Qwen-1.5B, average accuracy gains are +1.6% with GRPO and +8.4% with GRPO+P2T; for Qwen2.5-VL-7B-Instruct on MathVista, they are +2.0% and +6.9%, respectively.Source: §3.2.1 · p. 7
- Continued P2T training improves both outcome-RL-converged models on their original data and reaches GRPO’s best AIME25 Pass@1 with 3–4× fewer steps.GRPO+P2T also remains above GRPO and ends with a clear margin in the Qwen2.5-Math-1.5B training comparison.Source: §3.2.1 · p. 7, Figure 3 · p. 8
3.2.2 Comparison with Existing Token-level Reward
- On six benchmarks, P2T improves average accuracy over GRPO by +7.2%, PRIME by +4.7%, and SPRO by +2.3%.The comparison uses Eurus-2-7B-SFT and Eurus-2-RL-Data.Source: §3.2.2 · pp. 7–8, Table 2 · p. 8
3.3 Ablation Study
- Within-step redistribution raises accuracy from 50.6 to 53.8; +3.2% of P2T’s +5.8% overall gain comes from assigning credit among tokens rather than using step scores alone.The default result is 53.8, and the ablation varies one design factor at a time.Source: §3.3 · p. 8, Table 3 · p. 9
- Accuracy varies by at most 0.6% as η ranges from 0.25 to 1.0, while the signed variant scores 52.6 versus 53.8 for magnitude-based redistribution.All η settings remain above the process-reward baseline of 50.6; the authors leave better use of sign information for future work.Source: §3.3 · p. 9
- P2T improves over outcome reward with every tested PRM, but text-only PRM results span 50.0–53.8; among null tokens, <|endoftext|> scores 53.8 and <|im_end|> 50.3.The authors link the lower score for <|im_end|> to its response-completion signal and recommend checking PRM competence if training is unstable.Source: §3.3 · p. 9
4 Related Work
Related work describes reward signals used in reinforcement fine-tuning, from sparse outcome rewards to process and token-level rewards. It reviews how these approaches assess outputs or assign credit at different levels of granularity.
Reinforcement fine-tuning
- Reinforcement fine-tuning uses rewards to guide policy updates that refine model parameters toward expected performance under the reward criteria.The reward function or model assesses output quality and supplies scalar feedback for learning.Source: §4 · pp. 9–10
Outcome Reward
- Outcome rewards judge final-answer correctness, but sparse feedback can reinforce incorrect early steps in otherwise correct trajectories.LLMs can also serve as reward models for grammar, truth-relative correctness, and preference.Source: §4 · p. 10
Process Reward
- Process rewards score reasoning steps more finely than outcome rewards, but each step still groups many tokens; PRM training can require extensive fine-grained annotation.The cited work includes human-annotated and automated step-wise reward data, as well as multimodal PRMs.Source: §4 · p. 10
Token Reward
- Token-reward methods assign credit using reward-model or policy log probabilities, learned token scores, step scores, or expected incremental contributions.Some approaches train token-level reward models, while others derive rewards without a dedicated token-level model.Source: §4 · p. 10
5 Conclusion
P2T converts an existing process reward model’s coarse reward into token-level rewards through efficient attribution, and experiments show consistent gains over outcome reward across models, reasoning paradigms, and modalities. Its effectiveness depends on the quality of the PRM, particularly in the target domain.
5 Conclusion
- P2T consistently outperforms outcome reward across model families, reasoning paradigms, and modalities, with particularly large gains on harder benchmarks and where outcome-based RL stalls.It also improves models already converged under outcome-based RL and outperforms existing token-level reward methods.Source: §5 · p. 11
- P2T converts an existing process reward model’s coarse-grained reward into token-level rewards by estimating each token’s influence with a first-order Taylor approximation.The approximation requires one forward and backward pass, rather than separate inference for each token.Source: §5 · p. 11
- P2T’s token-level rewards integrate with standard RL algorithms without additional reward-model training, and reach outcome-based RL’s best accuracy with 3–4× fewer training steps.Source: §5 · p. 11
- P2T’s effectiveness depends on PRM quality, so an unreliable PRM in the target domain may limit its benefits.Future work will explore using attribution-score signs to provide more informative token-level rewards.Source: §5 · p. 11