Rewards as Labels: Revisiting RLVR from a Classification Perspective

Problem

• GRPO-style objectives under-update low-probability positive tokens and allow high-probability negative tokens to dominate updates.These mismatches inefficiently allocate credit and can contribute to suboptimal policy updates.Source: §1 · pp. 1–2

REAL framework

• REAL reformulates policy optimization as classification by treating verifiable rewards as categorical labels rather than scalar weights.It increases relative log-probabilities for positive rollouts and decreases them for negative rollouts.Source: §1 · p. 2

Mechanism

• REAL’s monotonic and bounded gradient weighting mitigates Gradient Misassignment in Positives and Gradient Domination in Negatives.The framework also introduces anchor logits to enhance policy learning and improve gradient allocation across rollouts.Source: §1 · p. 2

Empirical validation

• REAL improves training stability and consistently outperforms GRPO and its variants across mathematical reasoning benchmarks.The experiments cover AIME 2024/2025, MATH 500, AMC 2023, Minerva, and Olympiad Bench.Source: §1 · p. 2
• Using a vanilla binary cross-entropy loss, REAL remains stable without an explicit KL penalty and outperforms DAPO by 4.5% on average.REAL also achieves competitive performance against strong recent baselines.Source: §1 · p. 2
• Even with a vanilla binary cross-entropy loss, REAL remains stable and exceeds DAPO by 4.5% on average.This result is reported alongside the broader stability and performance gains.Source: §1 · p. 2

2.1. Group Relative Policy Optimization

• GRPO samples multiple rollouts from the previous policy and uses their rewards for advantage estimation.The method optimizes the current policy relative to the previous policy using an importance-sampling ratio.Source: §2.1 · p. 2
• GRPO’s loss uses importance sampling, KL regularization, advantages, and clipping controlled by ε.The cited formulation identifies β as the KL-divergence weight and ε as the clipping hyperparameter.Source: §2.1 · p. 2
• Table 1 reports the percentage and average gradient magnitudes for tokens across clipping ranges, with zero gradients where clipping is active.The reported clip ratio is ε = 0.2.Source: Table 1 · p. 3

2.2. Unified Version of Softmax Loss

• The unified softmax cross-entropy formulation represents positive and negative samples with separate logit sets, Zp and Zn.The objective explicitly contrasts positive logits with negative logits.Source: §2.2 · p. 3
• Minimizing the classification loss raises positive scores and suppresses negative scores, increasing the margin between the two sets.This provides the classification perspective underlying the paper’s policy-learning reformulation.Source: §2.2 · p. 3

GRPO gradient analysis

• GRPO gradient magnitude grows exponentially with relative log-probability, producing undesirable optimization behavior.The relative log-probability is denoted s_t in the analysis.Source: §3 · p. 3

Gradient Misassignment in Positives

• For positive rollouts, GRPO weakens gradients for low-probability tokens, diminishing correction for challenging but correct outputs.This is identified as Gradient Misassignment in Positives.Source: §3 · p. 3

Gradient Domination in Negatives

• For negative rollouts, unbounded gradients let high-probability tokens dominate group updates and suppress other informative samples.This outlier sensitivity reduces training stability.Source: §3 · p. 3

Empirical evidence

• 0.6943 is the average gradient magnitude for hard positives, while hard negatives reach 1.3553, corroborating the predicted imbalance.Additionally, 4.83% of positive tokens fall in [1−ε, 1) with magnitude 0.9788.Source: §3 · p. 3

Implications for REAL

• REAL uses bounded and monotonic gradient weighting to handle rollouts and tokens without clipping, supporting more stable optimization.This contrasts with GRPO-style weighting, whose problematic behavior motivates REAL.Source: §3 · p. 3

4.1. Revisiting RLVR as Classification

• REAL partitions rollouts into positive and negative sets using verifiable rewards, then reformulates policy optimization as classification within each prompt-conditioned group.Relative log-probability scores serve as logits for distinguishing desirable from undesirable rollouts.Source: §4.1 · p. 4
• The training procedure samples rollout groups, assigns binary verification labels, computes relative log-probability logits, and updates the policy with the REAL objective.The algorithm iterates over prompt batches and mini-batches before returning the fine-tuned policy.Source: §4.1 · p. 4

4.2. REAL Objective

• REAL combines positive and negative softmax losses while supporting standard classification losses such as binary cross-entropy.For positives, lower scores increase the loss; for negatives, higher scores increase the loss, directing updates toward the desired separation.Source: §4.2 · p. 5
• Anchor logits fix the decision threshold at zero, pushing positive rollout scores above zero and negative rollout scores below zero.The anchor resolves ambiguous updates that could otherwise increase the positive-negative gap while both scores decrease.Source: §4.2 · p. 5

4.3. Gradient analysis of REAL

• REAL assigns monotonic, bounded gradient weights: positive gradients decrease with relative log-probability, whereas negative gradients strengthen as relative log-probability increases.The gradient magnitude is upper-bounded by 1/τ, providing controlled adaptive modulation across rollouts.Source: §4.3 · p. 5
• REAL avoids overly large updates without explicit KL regularization, and the paper reports comparable performance without a KL penalty.The authors attribute this stability to the adaptive gradient clipping induced by bounded weighting.Source: §4.3 · p. 5
• REAL consistently outperforms RLVR baselines across diverse benchmarks on the 1.5B backbone, with the comparison summarized in Table 2.The supplied table description states that REAL outperforms all RLVR baselines across diverse benchmarks.Source: Table 2 · p. 6
• REAL is evaluated at 7B scale and on DAPO-Math-17K to assess scaling and generalizability.Table 3 covers the 7B backbone, while Table 4 covers the DAPO-Math-17K distribution.Source: Table 3 · p. 6, Table 4 · p. 6

5.2. Main Results

• REAL improves average Pass@1 over GRPO by 9.5 points and DAPO by 6.7 points on the 1.5B model, while surpassing GSPO by 1.7 points at 7B.At 7B, REAL reaches 63.2% average Pass@1 and maintains its advantage across six mathematical reasoning benchmarks.Source: §5.2 · p. 7
• REAL retains its performance edge on the DAPO-Math-17K dataset despite the changed data distribution.The evaluation tests robustness beyond the primary training setup.Source: §5.2 · p. 7
• REAL maintains stable entropy and steadily improves rewards and validation Pass@1, unlike GRPO’s entropy collapse and DAPO’s entropy explosion.The training-dynamics comparison covers entropy, AIME 2024 validation Pass@1, and rewards for the 1.5B model.Source: §5.2 · p. 7, Figure 3 · p. 7

5.3. Further Analysis

• The BCE variant reaches 50.4% Pass@1 versus 52.6% for REAL, consistent with the benefits attributed to group-wise competition and anchor logits.The softmax objective encourages competition among rollouts, while anchor logits provide a reference for relative log-probabilities.Source: §5.3 · pp. 7–8
• Removing explicit KL divergence produces no significant performance difference, suggesting REAL supplies implicit and adaptive gradient clipping.The comparison includes no KL, current-policy-to-reference KL, and current-policy-to-old-policy KL settings.Source: §5.3 · p. 8
• Anchor logits slightly improve over their removal, while τ = 0.5 and τ = 1.0 remain stable but τ = 0.1 causes early collapse.REAL’s gradient weighting is bounded by 1/τ, so lower temperatures increase the upper bound and can destabilize training.Source: §5.3 · p. 8

6.1. Large Reasoning Models and RLVR

• RLVR uses rule-based verification to provide rewards without human preference judgments or learned reward models, reducing reward-model overhead.The discussion covers mathematical and program reasoning and positions RLVR as a post-training paradigm for reasoning models.Source: §6.1 · p. 8
• Most RLVR variants build on GRPO, while modifying algorithms, reward design, or sampling strategies.The cited examples include methods spanning these three directions.Source: §6.1 · p. 8
• The paper identifies Gradient Misassignment in Positives and Gradient Domination in Negatives as GRPO-style issues that motivate REAL.These mismatches are described as causing inefficient credit assignment and suboptimal policy updates.Source: §6.1 · p. 8

6.2. Classification Objectives

• REAL relies on softmax cross-entropy as a classification objective without extra enhancements.The related-work discussion presents softmax cross-entropy as simple and effective, while REAL applies it to categorical reward supervision.Source: §6.2 · p. 8

7. Conclusions

• REAL reframes verifiable rewards as categorical labels, formulating policy optimization as classification to address inefficient and suboptimal GRPO-style updates.The targeted issues are Gradient Misassignment in Positives and Gradient Domination in Negatives.Source: §7 · pp. 8–9
• The classification reformulation yields balanced and bounded gradient allocation across rollouts, improving training stability and policy performance.The conclusion presents classification reformulation as a principled path for RLVR policy learning.Source: §7 · p. 9

Impact Statement

• REAL targets inefficient training dynamics and unpredictable behavior by reformulating verifiable rewards as categorical labels.The impact statement links this design to a more principled and stable optimization paradigm.Source: Impact Statement · p. 9
• The paper associates stable optimization with developing more transparent, robust, and trustworthy AI systems.This is presented as the broader intended impact of the proposed RLVR framework.Source: Impact Statement · p. 9

• GRPO’s parameter gradient is given formally with respect to policy parameters θ.Source: §A.1 · p. 11
• Iclip represents the indicator function for the clipping constraint, while st measures relative log-probability between πθ(ot|q) and πold(ot|q).Source: §A.1 · p. 11
• The t-th token’s absolute gradient magnitude is rewritten explicitly for subsequent analysis.Source: §A.1 · p. 11

• REAL’s objective gradient is analyzed formally with respect to policy parameters θ.Source: §A.2 · p. 11
• The normalized relative log-probability score is defined for each rollout ok.Source: §A.2 · p. 11

Positive samples

• For positive rollouts k ∈ O+, the derivation isolates each sample’s contribution and obtains its absolute gradient magnitude.Source: §A.2 · pp. 11–12
• Separating one rollout from the remaining positive samples yields the weighting term WREAL after substitution.Source: §A.2 · p. 12

• The negative-sample derivation is analogous, and combining positive and negative samples summarizes REAL’s total absolute gradient magnitude.Source: §A.2 · p. 12
• C+ ≥1 and C−≥1 are context-dependent constants determined by other rollouts in the group.Source: §A.2 · p. 12