PaperlayerView exact revision on arXiv

Source-linked AI summary

Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards

Leqi Zheng, Jinbo Su, Fang Niu, Chaokun Wang, Weiping Wang, Jiajun Zhang, Shannan Yan, Jie Wu, Zhaolu Kang, Rong Fu, Hang Zhang

arXiv:2609.03342v1cs.LG

TL;DR

RLVR’s binary reward treats all correct trajectories alike, while existing dense alternatives either ignore available expert CoT or require costly offline annotation. GAR uses expert-anchor gradient alignment to provide online process rewards, and it consistently improves benchmark performance with positive transfer beyond math; its current scope excludes industrial production deployment.

  • Problem

    Binary RLVR rewards cannot distinguish among correct reasoning trajectories, while existing dense alternatives ignore available expert solutions or require expensive offline annotation.

  • Method

    GAR extracts truncated output-layer gradient vectors and uses cosine similarity with gradients from expert chain-of-thought anchors as a dense process reward.

  • Results

    GAR consistently improves pass@k over GRPO and other baselines on competition-level math benchmarks and transfers to GPQA Diamond and MMLU-Pro without domain-specific training data.

  • Takeaways & Limitations

    GAR reuses expert CoT already present in training corpora for online reasoning-aware supervision with less than 9% wall-clock overhead.

  • Takeaways & Limitations

    GAR has not yet been deployed in an industrial production environment.

Abstract

from arXiv · show

Reinforcement learning from verifiable rewards (RLVR) drives chain-of-thought reasoning in large language models, yet its binary outcome reward cannot distinguish among correct trajectories. Existing dense reward alternatives, from surface heuristics to process reward models, either ignore the expert solutions already present in training corpora or require expensive offline annotation. We propose Gradient-Aligned Reward (GAR), which operates in the policy's own gradient space: truncated backpropagation through the output projection layer extracts a compact gradient vector for each rollout, and cosine similarity with an expert-anchor gradient yields a dense, reasoning-aware reward with less than 9% wall-clock overhead. We prove that this cosine admits a multiplicative decomposition into prediction-error and activation-pattern factors, providing a concrete characterization of what the alignment signal measures. On Qwen3-4B and Qwen3-8B, GAR consistently improves over GRPO and other baselines on competition-level math benchmarks and transfers to GPQA Diamond and MMLU-Pro without domain-specific data. Code and data are available at https://github.com/LQgdwind/GAR.

1 Introduction

RLVR’s binary correctness reward cannot distinguish among correct reasoning trajectories, motivating GAR, which converts expert chain-of-thought into dense gradient-space supervision. GAR is lightweight and improves math performance while transferring to general reasoning benchmarks.

  • Motivation: Binary outcome rewards assign identical value to correct responses, collapsing their group-relative advantage and preventing preferential reinforcement of higher-quality reasoning.This creates a flat reward over the correct subset.
  • Motivation: Existing alternatives either use surface heuristics that miss reasoning structure or require expensive expert annotations and offline process supervision.These approaches also fail to exploit expert solutions already included in training corpora.
  • GAR: GAR uses truncated backpropagation through the output projection layer to compare rollout gradients with expert-anchor gradients by cosine similarity.The method turns available expert CoT solutions into dense, per-rollout process rewards.
  • GAR: The gradient cosine decomposes into prediction-error and activation-pattern factors, characterizing two complementary axes for distinguishing correct trajectories.This provides a formal account of what the alignment signal measures.
  • GAR: Less than 9% wall-clock overhead accompanies GAR’s lightweight online computation, which adds truncated output-layer alignment to standard training.The alignment cost is reduced to O(V × d).
  • Empirical Validation: GAR yields consistent pass@k gains on four competition-level math benchmarks and transfers to GPQA Diamond and MMLU-Pro on Qwen3-4B and Qwen3-8B.The transfer uses no domain-specific training data.

2 Method

GAR addresses flat rewards among correct rollouts by aligning their truncated gradient-activation signals with cached expert-anchor vectors. A verifier gate preserves outcome correctness while cosine-based, group-centered bonuses provide dense process supervision.

  • 2.1 The Gradient Alignment Hypothesis: GAR introduces intra-group variation among correct responses by rewarding gradient-space alignment with an expert anchor.This addresses the flat outcome-reward pathology.
  • 2.1 The Gradient Alignment Hypothesis: The alignment hypothesis treats similar reasoning processes as producing similarly directed gradients, while qualitatively different correct pathways produce weak alignment.The proposed signal is intended to distinguish reasoning pathways despite textual variation.
  • 2.2 Truncated Gradient Signal: GAR freezes the Transformer body and backpropagates only through the output projection layer, producing a compact vector of dimension d instead of |θ|.The truncated surrogate is designed to preserve trajectory ordering under mild regularity conditions.
  • 2.2 Truncated Gradient Signal: The gradient-activation signal multiplies output-layer loss sensitivity by hidden-state activation, then averages across response positions and L2-normalizes the result.This emphasizes dimensions where sensitivity and activation jointly concentrate and keeps cosine comparisons commensurate across lengths.
  • 2.3 Reward Formulation: Only verifier-passing rollouts receive alignment computation and bonuses; incorrect responses receive no alignment bonus or gradient computation.The hard gate confines GAR’s extra cost to verifier-passing rollouts.
  • 2.3 Reward Formulation: GAR maps each dataset-provided expert CoT to a cached anchor vector and scores each correct rollout by cosine similarity with that anchor.The anchor reuses existing annotations without additional annotation cost.
  • 2.3 Reward Formulation: The final reward adds a nonnegative, group-centered alignment bonus to the base correctness reward, with a small format penalty applied as specified.The clipping keeps every correct trajectory weakly above the outcome-only baseline before format penalties.

3 Theoretical Analysis

The analysis connects GAR’s gradient-activation cosine to output-layer NTK similarity and decomposes alignment into prediction-error and activation-pattern components. It also establishes variance amplification, safe shaping, and monotonic alignment improvement under stated assumptions.

  • 3.1 NTK Interpretation and Multiplicative Decomposition: The output-layer gradient is the parameter-gradient object used to relate a response’s loss behavior to expert-anchor alignment.The analysis defines go(x, y) as the gradient of the loss with respect to Wo.
  • 3.1 NTK Interpretation and Multiplicative Decomposition: The gradient-activation signal aggregates per-token products that capture the linearized contribution of hidden dimensions to log-likelihood.This gives the signal a first-order interpretation in the output-layer representation.
  • 3.1 NTK Interpretation and Multiplicative Decomposition: Under bounded error and hidden states, normalized output-layer NTK similarity is bounded above and below by gradient-activation cosine up to error-dispersion terms.Thus high-cosine trajectories correspond to high NTK similarity up to bounded distortion.
  • 3.1 NTK Interpretation and Multiplicative Decomposition: The alignment signal requires agreement in both prediction-error profile and activation pattern, so different computational pathways receive lower cosine scores.The decomposition characterizes these as complementary factors.
  • 3.1 NTK Interpretation and Multiplicative Decomposition: 3.5× higher cosine scores occur for same-method than alternative correct rollouts, with mean ¯bsame = 0.45 versus ¯bdiff = 0.13.This result is reported across six training-set problems with different correct solution methods.
  • 3.2 Safe Reward Shaping: The safe-shaping theorem guarantees that correct responses retain at least the positive base reward, while incorrect responses remain below them under the verifier gate.The clipped bonus prevents degradation relative to the outcome-only baseline.
  • 3.3 Variance and Unbiasedness: GAR amplifies variance among correct-trajectory advantages when alignment bonuses have non-zero within-subset variance, while prompt-group normalization keeps their mean bonus at zero.These properties address the variance collapse of flat outcome rewards.
  • 3.4 Monotonic Alignment Improvement: GAR-shaped updates satisfy ¯Ct+1 ≥ ¯Ct under a sufficiently small KL penalty, establishing monotonic expected cosine alignment.The stated mechanism assigns positive advantages exclusively to above-mean-cosine trajectories.

4 Experimental Setup

The experiments train Qwen3 base models with GAR integrated into two policy-optimization backbones and compare reward functions under matched data, compute, and rollout conditions.

  • Training Configuration: Qwen3-4B-Base and Qwen3-8B-Base are trained for 400 steps from base checkpoints without SFT warmup, using approximately 10k NuminaMath-CoT problems and K=16 rollouts per prompt.Each optimization step samples 2,048 trajectories before verifier gating.
  • Baselines: GAR-GRPO and GAR-REINFORCE++ isolate GAR’s reward contribution across two distinct policy-optimization backbones.The comparison separates reward-function effects from optimizer-side design choices.
  • Controlled Comparison: All methods use the same base checkpoint, training data, compute budget, and rollout infrastructure, differing only in reward function.This design avoids conflating reward shaping with additional data, supervised initialization, or a larger rollout budget.
  • Evaluation: Table 2 reports pass@k accuracy averaged over 10 independent runs, with paired significance tests under matched sampling conditions.GAR improvements over corresponding base optimizers are statistically significant at p < 0.05 by paired t-test across runs.
  • Evaluation: Performance is evaluated on four competition-level math benchmarks and two general reasoning benchmarks using exact answer match after normalization.Pass@1 measures single-attempt concentration on correct solutions, while pass@16 measures coverage across repeated samples.

5 Results

GAR improves the corresponding optimizer across competition-level math benchmarks, with the largest benefits at pass@1 and gains that persist across model scales and optimizer backbones.

  • Main Results: Up to 52.4% relative gain at pass@1 is reported for GAR on HMMT 2025 at the 4B scale.The improvement is among results across four competition-level benchmarks.
  • Main Results: GAR-GRPO raises 4B pass@1 from 2.42 to 3.18 on HMMT 2026 and from 3.83 to 5.00 on HMMT 2025.These correspond to relative gains of 31.4% and 30.5%, respectively.
  • Main Results: GAR-GRPO improves pass@1 at 8B from 4.24 to 5.15 on HMMT 2026 and from 4.50 to 6.00 on HMMT 2025.The reported comparisons show that the benefit persists as model capacity grows.
  • Main Results: Improvements remain positive at pass@16 across every dataset and optimizer, while the largest reported relative gain is GAR-REINFORCE++ on HMMT 2025 at 4B, from 3.17 to 4.83.GAR therefore improves both optimizer variants without reducing overall coverage of correct reasoning paths.

6 Empirical Analysis

GAR transfers from mathematical training to general reasoning, while ablations identify activation weighting and verifier gating as important design elements and measurements show alignment growth at modest computational cost.

  • 6.1 General Reasoning Transfer: At 4B, GAR-GRPO gains 2.57 GPQA pass@1 points and 2.41 MMLU-Pro points over GRPO.The same passage reports gains of 1.79 points on GPQA pass@4 and 1.94 points on majority voting.
  • 6.2 Ablation Study: Removing the outcome reward degrades performance below baseline, showing that gradient alignment refines correctness rather than replacing the binary verifier signal.Raw gradients are optimizer-sensitive, whereas activation weighting produces reliable gains for both GRPO and REINFORCE++.
  • 6.2 Ablation Study: The mean cosine alignment among correct rollouts increases from 0.10 to 0.42 while variance narrows between training steps 0 and 400.Because only verifier-passing rollouts are included, the increase reflects redistribution within the correct subset toward expert-aligned trajectories.
  • 6.4 Computational Overhead: 8.3% overhead is added over outcome-only GRPO by truncating backpropagation to the lm_head layer.This is lower than the reported overheads for G2RL (+37.1%) and Grad2Reward (+43.7%).

7 Related Work

Prior work shows that outcome rewards can elicit reasoning, while process supervision and gradient-based methods provide denser or alternative signals with differing requirements and assumptions.

  • RL for LLM Reasoning and Reward Design: Critic-free policy-gradient methods and direct reinforcement learning from base models have shown chain-of-thought reasoning emerging from outcome rewards alone.The related work situates GAR within reinforcement learning approaches for LLM reasoning.
  • RL for LLM Reasoning and Reward Design: Process reward models offer denser supervision but require step-level annotations or a separately trained verifier.Classical potential-based shaping establishes policy-preservation conditions while leaving the potential function unspecified.
  • Gradient Signals in LLM Training: Gradient information has been used for data attribution, curriculum learning, and representation engineering, with NTK theory providing formal grounding.Recent methods also apply gradient signals to reward design.
  • Gradient Signals in LLM Training: Recent gradient-based reward methods include Grad2Reward, G2RL, and GradAlign.These methods form the immediate gradient-signal context for GAR’s contribution.

8 Conclusion

GAR converts expert chain-of-thought solutions into gradient-space anchors for dense process supervision during standard RLVR training. Across math and general reasoning benchmarks, it improves pass@k while transferring beyond mathematics.

  • GAR converts expert chain-of-thought solutions into gradient-space reference vectors and scores rollouts by cosine similarity with those anchors.
  • GAR provides dense process supervision within standard RLVR training with less than 9% wall-clock overhead.
  • GAR consistently improves pass@k over GRPO, REINFORCE++, and gradient-based competitors on four competition-level math benchmarks.
  • GAR transfers positively to GPQA Diamond and MMLU-Pro using Qwen3-4B and Qwen3-8B base models.

Limitations

GAR demonstrates consistent improvements across two model scales and multiple benchmarks, but it has not yet been deployed in an industrial production environment.

  • The reported improvements span two model scales and multiple benchmarks.
  • GAR has not yet been deployed in an industrial production environment.

Ethics Statement

The paper reports no human subjects, private data, or dual-use applications, while its theoretical analysis characterizes GAR’s alignment signal and reward-shaping behavior.

  • GAR is evaluated exclusively on publicly available mathematical reasoning benchmarks with open licenses, without human subjects, private data, or dual-use applications.
  • Gradient alignment is near zero when trajectories concentrate their signals on disjoint hidden-dimension subsets, even if both answers are correct.This connects the alignment signal to differences in computational pathways rather than final-answer correctness alone.
  • Theorem 3 states that verifier gating preserves correctness rewards while clipped alignment bonuses differentiate among correct trajectories.Incorrect responses receive non-positive shaped rewards, while correct responses receive at least the base reward.
  • Proposition 4 establishes that GAR training cannot decrease the expected gradient-cosine alignment of correct rollouts with the expert reference.The improvement is expressed through a non-negative covariance term that vanishes only when correct trajectories are equally aligned.

J Computational Overhead Analysis

GAR’s overhead is reduced through truncated output-layer backpropagation, verifier gating, and anchor caching, with measured cost agreeing with a predicted 5%–12% range.

  • Anchor caching reduces the amortized anchor-gradient cost from O(KN_ydV) to O(N_ydV) for each prompt group.The anchor vector is computed once and reused across K rollouts.
  • For Qwen3-8B, V/(Ld) ≈ 1.03, making each truncated backward pass comparable in cost to one full forward pass.The analysis uses d=4096, V=152064, and L=36.
  • Verifier gating avoids gradient computation for responses that fail verification, reducing effective early-training overhead.The reported early phase has 40–70% of rollouts failing verification.
  • The predicted end-to-end GAR overhead is approximately 5% to 12%, matching the measured wall-clock overhead.
  • Pass@1 peaks at default GAR hyperparameters, whereas pass@16 favors slightly more aggressive settings and degrades more gracefully.The sweep covers four hyperparameters on AIME 2026.

L General Reasoning Transfer: 8B Results

At 8B, GAR-GRPO achieves the highest scores across all four general-reasoning transfer metrics, including a +2.28 absolute margin over GRPO on MMLU-Pro.

  • GAR-GRPO achieves the highest scores across all four Qwen3-8B general-reasoning transfer metrics.All methods are trained exclusively on mathematical data and evaluated zero-shot.
  • +2.28 absolute on MMLU-Pro is GAR-GRPO’s largest margin over GRPO at the 8B scale.
  • Same-method rollouts average 0.45 cosine alignment versus 0.13 for different-method rollouts, a 3.5× gap.The comparison covers six training-set problems with alternative correct solution methods.
  • The alignment signal distinguishes underlying computational pathways rather than only response length, formatting, or template phrasing.The decomposition links this distinction to prediction-error and activation-pattern factors.
  • GAR’s pass@1 peaks at defaults, while pass@16 favors broader gradient-signal settings and degrades more gracefully.
  • Expert solutions predominantly use concise, theorem-driven approaches, such as complex-number rotation and modular arithmetic.The passage connects these strategies with reported accuracy gains across benchmarks.
Loading 2609.03342v1…