PaperlayerView exact revision on arXiv

Source-linked AI summary

DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards

Kaiyi Zhang, Wei Wu, Yankai Lin

arXiv:2605.21467v1cs.LGcs.CL

TL;DR

RLVR’s response-level rewards leave token-level credit assignment poorly understood, while standard aggregation can dilute directions that distinguish higher- from lower-reward responses. DelTA uses discriminative token reweighting to reshape the update, outperforming same-scale baselines across seven mathematical benchmarks and achieving the best result on every benchmark.

  • Problem

    RLVR’s sequence-level advantages do not explain which token-level updates drive sparse distributional changes, and shared high-frequency directions can weaken reward-side discrimination.

  • Method

    DelTA estimates discriminative token coefficients and uses them to reweight a self-normalized RLVR surrogate, making positive and negative update directions more contrastive.

  • Results

    DelTA achieves the best result on every benchmark and the highest average score at both model scales, raising the 8B average from 25.14 to 28.40.

  • Takeaways & Limitations

    The discriminator view serves as an analysis and design principle for shaping token-level policy-update directions in sequence-level RLVR.

  • Takeaways & Limitations

    The local discriminator characterization is used as a design principle rather than an exact description of the full nonlinear clipped RLVR training trajectory.

Abstract

from arXiv · show

Reinforcement learning from verifiable rewards (RLVR) has emerged as a central technique for improving the reasoning capabilities of large language models. Despite its effectiveness, how response-level rewards translate into token-level probability changes remains poorly understood. We introduce a discriminator view of RLVR updates, showing that the policy-gradient update direction implicitly acts as a linear discriminator over token-gradient vectors and thereby determines which token probabilities are increased or decreased during learning. Under standard sequence-level RLVR, this discriminator is constructed from positive- and negative-side centroids formed by advantage-weighted averaging of token-gradient vectors. However, such centroid construction can be dominated by shared high-frequency patterns, such as formatting tokens, diluting sparse yet discriminative directions that better distinguish high-reward responses from low-reward ones. To address this limitation, we propose $\textbf{DelTA}$, a discriminative token credit assignment method that estimates token coefficients to amplify side-specific token-gradient directions and downweight shared or weakly discriminative ones. These coefficients reweight a self-normalized RLVR surrogate, making the effective side-wise centroids more contrastive and thereby reshaping the RLVR update direction. On seven mathematical benchmarks, DelTA outperforms the strongest same-scale baselines by 3.26 and 2.62 average points on Qwen3-8B-Base and Qwen3-14B-Base, respectively. Additional results on code generation, a different backbone, and out-of-domain evaluations further demonstrate the generalization ability of DelTA.

1 INTRODUCTION

The introduction presents RLVR as an effective response-level reward paradigm whose token-level effects can be understood through an implicit discriminator. It identifies centroid dilution as a limitation of standard updates and proposes DelTA to emphasize discriminative token-gradient directions.

  • Motivation: RLVR improves LLM reasoning using response-level verifiable rewards without dense process-level annotations.Reported gains span mathematics, code generation, and formal problem solving.
  • Discriminator View: Policy-gradient updates implicitly form a linear discriminator over token-gradient vectors that determines whether candidate-token probabilities increase or decrease.The discriminator contrasts aggregates from positive- and negative-advantage responses.
  • Limitation: Standard sequence-level RLVR averages positive- and negative-advantage token-gradient vectors into centroids that can emphasize shared high-frequency structure rather than discriminative directions.Formatting tokens and other common response structure can dilute sparse signals distinguishing higher- from lower-reward responses.
  • DelTA: DelTA reweights token-gradient terms using coefficients that upweight side-specific directions and downweight shared or weakly discriminative ones.This reshapes the effective aggregates and the induced RLVR discriminator.

2 PRELIMINARIES

This section establishes critic-free group-relative RLVR using DAPO as the concrete framework, with sequence-level rewards and group-normalized advantages driving token-level surrogate updates. The analysis focuses on the surrogate-induced update rule, while dynamic sampling is disabled experimentally.

  • Framework: Critic-free group-relative RLVR samples a response group for each prompt, assigns sequence-level rewards, and normalizes advantages within the group.The group-normalized advantage uses the reward mean, reward standard deviation, and a small numerical constant.
  • Token-Level Analysis: Although the response-level advantage is shared across tokens in a response, token-level importance ratios determine how surrogate contributions accumulate for token-gradient analysis.The token-level ratio is the basic object used to analyze per-token updates.
  • DAPO-style Surrogate: DAPO optimizes a clipped surrogate with asymmetric clipping and token-level normalization across all response tokens.DAPO is used as the main concrete example of the critic-free group-relative RLVR framework.
  • DelTA Overview: DelTA estimates token coefficients from contrasts between positive- and negative-advantage token-gradient aggregates to reweight the sequence-level RLVR objective.This overview motivates the token-credit-assignment analysis developed after the preliminaries.
  • Experimental Scope: Dynamic sampling is disabled for all experiments because it affects rollout filtering rather than the per-token surrogate’s form.The analysis therefore isolates the update rule induced by the surrogate objective.

3 METHOD

Section 3 explains sequence-level RLVR as an implicit linear discriminator over token-gradient vectors: token probabilities rise or fall according to alignment with positive- and negative-side reference directions. DelTA reshapes these centroids by emphasizing discriminative token directions and downweighting shared patterns, thereby modifying token-level credit assignment.

  • Motivation: Sequence-level RLVR can produce sparse, targeted token-distribution changes even though all tokens in a response share one scalar advantage.This motivates analyzing how the update selects candidate tokens rather than attributing selection directly to explicit token-level rewards.
  • Discriminator view: A candidate token’s local log-probability change is determined by the inner product between its token-gradient vector and the policy-update direction.The result follows from a first-order Taylor characterization around the old policy parameters.
  • Discriminator view: The RLVR update acts as an implicit linear discriminator, increasing candidate-token probability when its positive-side score exceeds its negative-side score.The positive and negative scores arise from advantage-weighted token-gradient aggregates separated by the sign of response-level advantage.
  • Centroid limitation: Standard RLVR uses advantage-weighted positive- and negative-side centroids, but shared frequent patterns can dominate both and dilute rarer, more discriminative directions.This creates a mismatch between centroids that summarize each side within itself and centroids that effectively discriminate between higher- and lower-reward responses.
  • DelTA: DelTA reweights token terms in the RLVR surrogate to reshape the effective side-wise centroids, induced discriminator, and local update direction.It initializes from original advantage-weighted centroids and iteratively assigns larger scores to token-gradient directions closer to their own side than the opposing side.
  • Implementation: In practice, DelTA computes stop-gradient token coefficients with a layer-restricted LM-head gradient proxy because full-parameter gradients for all rollout tokens are computationally prohibitive.The proxy affects coefficient computation, while the analysis remains formulated using full-parameter token-gradient vectors.

4 EXPERIMENTS

Experiments show that DelTA consistently improves over same-scale RL baselines across seven mathematical benchmarks and both Qwen3 backbone sizes. Additional evaluations support its effectiveness in training dynamics, code generation, and a different backbone.

  • Experimental setup: DelTA is evaluated against DAPO, DAPO w/ Forking Tokens, SAPO, and FIPO using Qwen3-8B-Base and Qwen3-14B-Base with matched hyperparameters.Training uses DeepMath-103K and disables dynamic sampling to isolate method effects.
  • Evaluation: Seven mathematical benchmarks are evaluated with maximum generation length of 30,000 tokens and 16 sampled responses per problem.Reported performance is averaged over all samples.
  • Main results: 3.26 points is the average-score improvement from 25.14 to 28.40 on the Qwen3-8B-Base backbone over the strongest same-scale baseline.DelTA achieves the best result on every benchmark at this scale.
  • Generalization: DelTA also improves DAPO on code-generation benchmarks and remains effective on Olmo3-7B-Base, indicating benefits beyond the main mathematical setting.The paper discusses computational overhead separately in Appendix L.1.
  • Training dynamics: DelTA continues improving after DAPO plateaus and slightly degrades, reaching a higher final reward while maintaining longer responses with lower entropy.The response-length and entropy trends suggest the divergence is not merely due to shorter answers.

5 ANALYSIS

The analysis evaluates DelTA on representative mathematical benchmarks through diagnostic studies of its contrastive token assignment, token-level learning signals, design components, hyperparameter robustness, and out-of-domain generalization. Results show that DelTA’s coefficients identify valuable token-gradient directions, while several design choices are examined through ablations.

  • 5 ANALYSIS: The analysis primarily evaluates DelTA on Qwen3-8B-Base across AIME25, AIME26, HMMT25-Nov, and HMMT26-Feb.These four representative mathematical benchmarks are abbreviated as HMMT25 and HMMT26.
  • 5 ANALYSIS: Five diagnostic questions examine opposite-side comparison, token-level learning signals, design components, hyperparameter sensitivity, and out-of-domain generalization.The studies test whether own-side centrality explains gains, whether λ_i,t is useful for token selection, whether DelTA components are necessary, and whether performance generalizes beyond the analyzed benchmarks.
  • Q2: Token-level learning signals: Top-λ training on 50% of tokens consistently outperforms full-token DAPO and random 50% selection, while bottom-λ training quickly collapses.Evaluation shows the same pattern: top-λ selection improves accuracy across benchmarks, whereas random selection stays close to DAPO.
  • Q2: Token-level learning signals: The results indicate that λ_i,t separates token-gradient directions with high effective learning value from shared or misleading directions that weaken policy improvement.This conclusion follows from the sharp separation in training dynamics and evaluation accuracy under the token-selection strategies.
  • Q3: Design components: DelTA’s ablation study removes adaptive γ, entropy regularization, coefficient normalization, range mapping, or centroid refinement while keeping other training settings unchanged.The variants respectively alter assignment temperatures, soft assignments, coefficient-mass normalization, token-weight mapping, or coefficient estimation.

6 RELATED WORK

Prior RLVR work improves reasoning with PPO-style and critic-free objectives, while addressing the mismatch between response-level rewards and token-level updates through reweighting, value-based, and token-selection methods. DelTA is complementary, deriving token coefficients from the update’s own positive-negative discriminator rather than external dense rewards, value estimates, or auxiliary selection rules.

  • RLVR foundations: RL has improved LLM reasoning in verifiable-feedback domains through PPO-style optimization and critic-free objectives including GRPO and DAPO.Related work also studies RLVR mechanisms, training stability, and efficiency.
  • Token credit assignment: Prior methods address response-level rewards versus token-level updates with token- or step-level reweighting, process reward models, learned value functions, and token-selection signals.Token-selection signals include entropy and future influence.
  • DelTA’s distinction: DelTA is complementary: it reweights the RLVR surrogate with token coefficients derived from the update’s induced positive-negative discriminator.This avoids relying on external dense rewards, value estimates, or auxiliary token-selection rules.

7 CONCLUSION

The paper studies token-level learning in RLVR through a local discriminative lens, showing that sequence-level policy-gradient updates implicitly discriminate over token-gradient vectors. This discriminator can be dominated by shared, weakly discriminative directions rather than reward-distinguishing ones.

  • Conclusion: RLVR token learning is analyzed from a local discriminative perspective using DAPO as a representative sequence-level objective.The analysis focuses on how sequence-level policy-gradient updates affect token-level learning.
  • Conclusion: DAPO’s policy-gradient update induces an implicit linear discriminator over candidate token-gradient vectors.The discriminator is defined by positive- and negative-side centroids.
  • Conclusion: The positive- and negative-side centroids can be dominated by shared, weakly discriminative directions instead of directions distinguishing higher- from lower-reward responses.This identifies a limitation in the centroid-based view of sequence-level RLVR updates.

A LIMITATIONS · B BROADER IMPACTS

DelTA shows consistent gains but retains limitations in proxy-based coefficient estimation, evaluation breadth, and computational overhead. Its potential benefits for verifiable-reward training coexist with misuse risks from stronger reasoning models.

  • A LIMITATIONS: DelTA offers a lightweight approach to discriminative token-level credit assignment within sequence-level RLVR, while leaving directions for improvement and broader validation.
  • A LIMITATIONS: DelTA estimates token coefficients with layer-restricted token-gradient proxies because full-parameter gradients for all sampled tokens are computationally expensive at RLVR scale.The proxy is used only for stop-gradient coefficient estimation, while the weighted RLVR objective updates full policy parameters.
  • A LIMITATIONS: DelTA’s proxy ablations show robustness to different layer-restricted proxy choices, while richer and more efficient proxy methods remain open for exploration.
  • A LIMITATIONS: DelTA’s empirical evaluation focuses primarily on mathematical reasoning, with additional validation on code generation, different backbones, and out-of-domain benchmarks.Future evaluations could cover multi-turn interaction, tool-use tasks, and domains with more diverse verifiable signals.
  • A LIMITATIONS: DelTA introduces additional computation for coefficient estimation, although the measured overhead is modest in the reported setting.More efficient caching or lower-cost proxy computation could further reduce this cost.
  • B BROADER IMPACTS: DelTA may improve the effectiveness and efficiency of training reasoning-capable language models in domains where correctness can be automatically verified, including mathematics and code generation.The work may also help researchers diagnose and improve how sequence-level RLVR updates allocate token-level credit.
  • B BROADER IMPACTS: Stronger reasoning models may be misused to generate misleading content, harmful code, or support other dual-use applications.DelTA adds no new user-facing deployment mechanisms, sensitive-information datasets, or privacy-invasive capabilities, but may improve RLVR-trained model capabilities.

C DERIVATION OF THE LOCAL DAPO UPDATE · D MEAN DIRECTIONS AS WEIGHTED LEAST-SQUARES CENTROIDS · E REPRESENTATIVE RLVR VARIANTS AS TOKEN-WEIGHTED CENTROID ESTIMATORS

The appendices derive the local DAPO update and its induced next-token log-probability change, then interpret side-wise mean directions as weighted least-squares centroids. They further show that RLVR variants modify these centroids through different effective token-weighting schemes, including response length, entropy, and future influence.

  • C DERIVATION OF THE LOCAL DAPO UPDATE: Around the old policy, DAPO clipping is locally inactive because every sampled-token ratio equals one, so its clipped and unclipped surrogate gradients agree.The derivation analyzes a first-order update around θold with dynamic sampling disabled.
  • C DERIVATION OF THE LOCAL DAPO UPDATE: The local DAPO policy-gradient direction is formed by advantage-weighted token-gradient vectors and decomposes into positive- and negative-advantage contributions.This is the raw surrogate-gradient direction before optimizer preconditioning, weight decay, or gradient clipping.
  • C DERIVATION OF THE LOCAL DAPO UPDATE: The induced first-order change in a candidate next-token log-probability is obtained by substituting the local DAPO update into ∆log π(x | c).The resulting expression is the next-token log-probability change analyzed in Section 3.1.
  • D MEAN DIRECTIONS AS WEIGHTED LEAST-SQUARES CENTROIDS: The positive- and negative-side mean directions are exactly the unique weighted least-squares centroids of their respective token-gradient vectors when both side masses are nonzero.The positive-side objective is positive definite for M+ > 0, and the negative side follows analogously.
  • D MEAN DIRECTIONS AS WEIGHTED LEAST-SQUARES CENTROIDS: If one advantage side has zero total mass, its centroid is undefined and the group contributes no positive-negative centroid contrast.Such degenerate groups are skipped in the centroid construction.
  • E REPRESENTATIVE RLVR VARIANTS AS TOKEN-WEIGHTED CENTROID ESTIMATORS: GRPO uses response-balanced token-gradient averages, giving longer responses smaller per-token weights and shorter responses larger per-token weights.Each response contributes one averaged direction before group-relative advantages aggregate those directions.
  • E REPRESENTATIVE RLVR VARIANTS AS TOKEN-WEIGHTED CENTROID ESTIMATORS: DAPO gives equal weight to every valid token, making response contributions proportional to length and producing token-balanced side-wise centroid estimates.The global token-count normalizer is omitted because it does not affect the update direction.
  • E REPRESENTATIVE RLVR VARIANTS AS TOKEN-WEIGHTED CENTROID ESTIMATORS: Forking-token DAPO filters centroids to high-entropy tokens, whereas FIPO weights tokens by clipped future-influence signals rather than simply selecting uncertain tokens.Both methods alter the effective token support or mass used to estimate positive and negative directions.

F LAST-LAYER TOKEN-GRADIENT PROXY

DelTA estimates relative token coefficients with a scalable, layer-restricted last-layer proxy under a frozen-representation approximation rather than reconstructing full-parameter gradients. Proxy ablations show robustness to proxy choice, with top-K hidden-gradient features outperforming the default and random coefficients substantially worse.

  • Proxy construction: DelTA uses a layer-restricted token-gradient proxy derived from the LM head under a frozen-representation approximation.The proxy treats the final hidden state as a scalable representation for estimating relative token coefficients.
  • Proxy construction: The proxy is not an exact full-parameter gradient reconstruction, but a pragmatic reduction for scalable token-level analysis and reweighting.This approximation follows the use of first-order and layer-restricted computations for scalability.
  • Proxy ablations: 24.29 average performance from the top-K hidden-gradient proxy improves over 23.27 with the default proxy.The top-K proxy incorporates competition among high-probability vocabulary candidates and provides the strongest coefficient-estimation signal in the ablation.
  • Proxy ablations: 18.34 average score for the random-λ baseline substantially underperforms both DelTA variants, indicating that arbitrary token reweighting does not explain DelTA’s benefit.The baseline uses randomly sampled coefficients from the same bounded range while keeping normalization, objective, and other training settings unchanged.
  • Proxy ablations: Meaningful discriminative structure is required: tokens receive larger weights when their gradients better represent their own advantage side than the opposite side.The main experiments retain the conservative output-row proxy because it is the minimal and most direct layer-restricted implementation.

G DERIVATION OF THE DELTA SOFT ASSIGNMENT SCORE … L.1 COMPUTATIONAL OVERHEAD

DelTA derives token assignment scores from side-specific squared-distance margins and implements them through lagged centroid refinement, bounded coefficient normalization, and weighted DAPO training. Its main significance claim is statistically supported, while its practical overhead is modest: one refinement iteration adds K + 2 passes and 37 seconds, or approximately 10.2% of first-step time.

  • G DERIVATION OF THE DELTA SOFT ASSIGNMENT SCORE: DelTA assigns higher positive-side scores when token-gradient vectors are closer to the positive centroid than the negative centroid, with the negative-side rule obtained by swapping centroids.The corresponding squared-distance margin is positive exactly when the vector is closer to its own-side centroid.
  • G DERIVATION OF THE DELTA SOFT ASSIGNMENT SCORE: The soft assignment solution is the unique maximizer because the entropy-regularized one-dimensional objective is strictly concave, and the negative-side objective is symmetric.The resulting closed-form score matches Eq. (6).
  • H DELTA IMPLEMENTATION DETAILS: In each rollout batch, DelTA initializes side-wise centroids, performs K stop-gradient alternating refinement iterations, and computes final raw scores using refined centroids and cached temperatures.The raw scores are then mapped to bounded coefficients and normalized for training.
  • H DELTA IMPLEMENTATION DETAILS: The normalized coefficients are used as stop-gradient per-token multipliers under the standard DAPO token average, preserving relative token reweighting while changing only global coefficient scale.Coefficients are computed once per rollout batch and held fixed across repeated optimization passes.
  • I DETAILED SETTINGS: The main training uses Qwen3-8B-Base for 220 steps and Qwen3-14B-Base for 300 steps, selecting checkpoints by the highest AIME25 avg@8 score under a fixed rule.All methods and backbone sizes use the same training budget and checkpoint-selection rule.
  • J BASELINE DETAILS: The experiments compare DelTA against DAPO, forking-token DAPO, SAPO, and FIPO, which respectively provide strong baseline, entropy-token filtering, smooth gating, and Future-KL token reweighting approaches.DAPO is the main strong baseline; the forking-token variant retains only the top 20% high-entropy tokens.
  • K SIGNIFICANCE TEST DETAILS: DelTA significantly outperforms the strongest same-scale baseline at both model scales: SAPO for the 8B backbone and FIPO for the 14B backbone.The claim uses evaluation-run-level significance testing rather than repeated full training runs.
  • L.1 COMPUTATIONAL OVERHEAD: K + 2 additional actor forward passes are required for coefficient estimation; with K = 1, DelTA’s first training step is 37 seconds slower than DAPO, approximately 10.2% of total first-step time.The extra passes compute stop-gradient coefficients, while optimization otherwise follows the baseline weighted DAPO procedure.

L.2 OTHER MODEL ARCHITECTURES … L.6 TOKEN WEIGHT ANALYSIS

Across model families, coding tasks, out-of-domain benchmarks, and hyperparameter settings, DelTA consistently improves over DAPO and remains relatively robust. Token-weight analysis indicates that DelTA emphasizes discriminative reasoning-related tokens while downweighting shared or less informative background tokens.

  • L.2 OTHER MODEL ARCHITECTURES: DelTA improves the average score from 19.01 to 22.80 on all seven mathematical reasoning benchmarks with Olmo3-7B-Base, outperforming DAPO without architecture-specific tuning.The gain is 3.79 points, with particularly large improvements on AIME24, AIME26, HMMT25-Nov, and Brumo25.
  • L.3 CODE GENERATION: DelTA improves the weighted average from 47.7 to 49.5 across HumanEval+, MBPP+, and LiveCodeBench, outperforming DAPO on all three code-generation benchmarks.Models are trained for two epochs under the same recipe, with five sampled rollouts per problem.
  • L.4 Q4: IS DELTA SENSITIVE TO ITS HYPERPARAMETERS?: Changing the coefficient range causes only small average-score variations, while λmin = 0.5 achieves a comparable average score to the base setting.The studied coefficient ranges include [0.5, 1.2], [0.8, 1.5], and [0.5, 1.5].
  • L.4 Q4: IS DELTA SENSITIVE TO ITS HYPERPARAMETERS?: Increasing centroid refinement depth to K = 2 or K = 3 consistently reduces performance, indicating that a single refinement step is sufficient.The results suggest that excessive refinement may overfit the batch-level token-gradient geometry, while the base configuration performs best.
  • L.5 Q5: DOES DELTA GENERALIZE TO OUT-OF-DOMAIN EVALUATION?: On out-of-domain benchmarks, DelTA raises the weighted average from 58.87 to 62.38 on Qwen3-8B-Base and from 66.77 to 68.40 on Qwen3-14B-Base.DelTA improves over DAPO on both GPQA-Diamond and MMLU-Pro for both backbone sizes.
  • L.6 TOKEN WEIGHT ANALYSIS: The token-cloud visualization qualitatively compares high- and low-weight tokens using average DelTA coefficients over selected generated tokens collected at roughly 108 tokens during training.Larger tokens in the high-weight cloud have higher average coefficients, while larger tokens in the low-weight cloud have lower average coefficients.
  • L.6 TOKEN WEIGHT ANALYSIS: DelTA assigns larger coefficients to reasoning-related tokens whose gradient vectors are more representative of their own advantage side, and smaller coefficients to shared or less informative background tokens.The coefficients reflect discriminative alignment with the token’s own advantage side rather than semantic importance alone.
Loading 2605.21467v1…