Source-linked AI summary
Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning
Zhenwen Liang, Sidi Lu, Wenhao Yu, Kishan Panaganti, Yujun Zhou, Haitao Mi, Dong Yu
TL;DR
Existing LLM reinforcement-learning exploration relies on signals that may not reflect the policy’s update geometry. G2RL guides exploration using groupwise gradient features, and the paper reports improved accuracy and coverage across reasoning benchmarks.
Problem
Entropy bonuses and external semantic comparators can encourage variation without ensuring trajectories differ in policy-relevant update directions.
Source: §1 · p. 1Method
G2RL compares groupwise sequence features from final-layer sensitivity and applies bounded reward scaling to favor novel gradient directions.
Source: §1 · p. 2, §2.4 · p. 5Results
Across math and general reasoning benchmarks, G2RL consistently improves pass@1, maj@16, and pass@k over baselines.
Source: §1 · p. 2, §6 · p. 10Takeaways & Limitations
The approach aligns exploration with optimization geometry and integrates into GRPO without requiring extra backward passes.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Reinforcement learning has become essential for strengthening the reasoning abilities of large language models, yet current exploration mechanisms remain fundamentally misaligned with how these models actually learn. Entropy bonuses and external semantic comparators encourage surface level variation but offer no guarantee that sampled trajectories differ in the update directions that shape optimization. We propose G2RL, a gradient guided reinforcement learning framework in which exploration is driven not by external heuristics but by the model own first order update geometry. For each response, G2RL constructs a sequence level feature from the model final layer sensitivity, obtainable at negligible cost from a standard forward pass, and measures how each trajectory would reshape the policy by comparing these features within a sampled group. Trajectories that introduce novel gradient directions receive a bounded multiplicative reward scaler, while redundant or off manifold updates are deemphasized, yielding a self referential exploration signal that is naturally aligned with PPO style stability and KL control. Across math and general reasoning benchmarks (MATH500, AMC, AIME24, AIME25, GPQA, MMLUpro) on Qwen3 base 1.7B and 4B models, G2RL consistently improves pass@1, maj@16, and pass@k over entropy based GRPO and external embedding methods. Analyzing the induced geometry, we find that G2RL expands exploration into substantially more orthogonal and often opposing gradient directions while maintaining semantic coherence, revealing that a policy own update space provides a far more faithful and effective basis for guiding exploration in large language model reinforcement learning.
1 Introduction
The introduction identifies a mismatch between external exploration signals and the update directions that shape policy learning. G2RL instead guides exploration using the model’s own update geometry and reports improvements across reasoning tasks.
1 Introduction
- Existing exploration signals can reward semantic variation without distinguishing trajectories by their policy-relevant update directions.The paper motivates preferring trajectories that expand the policy’s update directions rather than adding redundant or uninformative gradients.Source: §1 · pp. 1–2
- G2RL compares sequence-level features from final-layer sensitivity and uses bounded reward scaling to emphasize trajectories introducing novel gradient directions.The features are obtained at negligible cost from a standard forward pass.Source: §1 · p. 2
- Grounding exploration in optimization geometry avoids semantic–optimization mismatch and integrates into GRPO without extra backward passes.Source: §1 · p. 2
- Across math and general reasoning tasks, G2RL improves single-sample accuracy and multi-sample coverage while producing more meaningful gradient dispersion.Source: §1 · p. 2
2 Method
G2RL guides exploration using sequence-level features derived from the policy’s first-order update geometry, then incorporates their groupwise comparisons into reward shaping. The method retains PPO-style clipping and KL control while changing how advantages are constructed.
2.1 Preliminaries and Group-Relative Policy Optimization
- G2RL modifies group-relative advantages while retaining PPO-style clipping and KL control; GRPO standardizes rewards across candidate responses without a learned critic.Each prompt has a group of responses sampled from a fixed behavior policy, with a scalar base reward for each response.Source: §2.1 · p. 3
2.2 Gradient Features: Policy-Referential Sensitivity
- The sequence feature aggregates final-layer token sensitivities, summarizing how a response would steer the output distribution and serving as a cheap proxy for update geometry.It is available from the forward pass without extra backpropagation.Source: §2.2 · pp. 3–4
2.3 Groupwise Gradient-Guided Exploration Score
- The groupwise score assesses a response’s direction against reward-weighted peer directions, treating near-orthogonality as evidence of new information for update geometry.Higher-reward responses receive greater weight as reference directions; the score is bounded and invariant to common feature rescaling.Source: §2.3 · p. 4
2.4 Gradient-Guided Reward Shaping
- For correct responses, high exploration scores boost rewards; for incorrect ones, they amplify penalties, while incorrect responses aligned with correct peers receive milder penalties.The method interprets the latter as near-misses whose update directions remain informative, and highly distinct incorrect directions as off-manifold behavior.Source: §2.4 · p. 5
2.5 Practical Reward Rescaling
- With binary base rewards, practical rescaling keeps the effective reward magnitude at |r̃(i)| ≤ 3 while allowing candidate weights to change within a group.The shaped reward is clipped to a fixed interval to keep the advantage scale stable.Source: §2.5 · p. 5
2.6 Computation and Implementation
- The token features require only forward-pass quantities and add negligible overhead; G2RL is a drop-in GRPO modification that leaves the rest of the RL pipeline unchanged.Feature aggregation uses a column gather and matrix–vector product with the token probabilities.Source: §2.6 · p. 5
3 Experiments
G2RL improves reasoning results across model sizes and benchmarks, while its training dynamics and exploration geometry indicate more useful diversity than entropy- or embedding-based signals. The experiments cover math and general reasoning tasks, with analyses of training behavior and gradient-space structure.
3.1 Benchmarks and Metrics
- Experiments use Qwen3 base models at 1.7B and 4B, with MATH training problems and a rule-based verifier; reported benchmark metrics include pass@1, maj@16, and pass@16.maj@16 verifies the most frequent final answer among 16 samples, with ties broken uniformly.Source: §3 · p. 5, §3.1 · p. 6
3.2 Main Results
- At 4B, AIME25 pass@1 rises from 17.5 to 20.1 and maj@16 from 23.9 to 29.0; G2RL leads all three reported AMC metrics.MATH500 gains are smaller but consistent across metrics, reaching pass@16 = 93.6.Source: §3.2 · p. 6, Table 1 · p. 6
- Across both model sizes, G2RL improves pass@1 on every dataset, with gains in multi-sample coverage and one reported 1.7B exception on AMC pass@16.On AMC pass@16, the baseline scores 68.5 versus G2RL’s 68.1.Source: §3.2 · p. 6, Table 1 · p. 6
3.3 Analysis on General Reasoning Tasks
- On GPQA, G2RL reaches pass@1 = 38.7, maj@16 = 44.0, and pass@16 = 89.2; on MMLUpro, its pass@1 micro-average is 58.47.At GPQA pass@32, EVOL-RL is marginally higher: 93.7 versus 93.5.Source: §3.3 · p. 6, Table 2 · p. 7
3.4 Training dynamics
- On AIME25, G2RL has the steepest, most stable mean@8 accuracy gains and fastest response-length increase, while entropy rises moderately alongside them.Entropy Bonus raises entropy most, but its growth is weakly coupled to accuracy; EVOL-RL improves below G2RL.Source: §3.4 · pp. 7–8, Figure 2 · p. 7
3.5 Analysis of Exploration Geometry
- On AIME25, G2RL shifts pairwise gradient similarity toward zero and raises negative-similarity pairs to 28.09%, versus 5.91% for vanilla GRPO.G2RL maintains higher semantic similarity than GRPO (0.769 versus 0.738), despite greater gradient-space diversity.Source: §3.5 · pp. 8–9, Figure 3 · p. 8
4 Discussion: The Geometry of Efficient Exploration
G2RL targets redundancy in update space rather than merely varying outputs: it promotes complementary gradient directions while preserving semantic consistency. Its reward weighting also distinguishes trajectories within the same correctness class.
The Geometry of Efficient Exploration
- Vanilla GRPO’s correct responses can be gradient-collinear despite surface diversity; G2RL yields a 5× increase in negative-similarity pairs, suggesting more complementary update directions.The discussion links complementary directions to counterbalancing dominant modes and continued stable exploration.Source: §4 · p. 9
- G2RL maintains higher semantic consistency than GRPO while achieving lower gradient similarity, showing that semantic variation and optimization-relevant variation can diverge.The reported semantic similarities are 0.77 for G2RL and 0.74 for GRPO.Source: §4 · p. 9
- Within binary-reward correctness classes, G2RL down-weights repetitive correct paths and amplifies rarer trajectories that open new update directions.This reweighting is intended to preserve learning signal after standard policy gradients would saturate on one mode.Source: §4 · p. 9
5 Related Work
Prior exploration methods use entropy, behavioral diversity, outcome rarity, or learned novelty signals. G2RL instead defines exploration through similarity in the policy’s own gradient feature space.
Exploration and diversity in reinforcement learning
- Earlier exploration approaches frame diversity through entropy, behavior repertoires, latent skills, rare outcomes, or learned novelty and consistency signals.These methods include quality–diversity approaches, DIAYN, outcome-based exploration, DARLING, EVOL-RL, and RESTRAIN.Source: §5 · pp. 9–10
Exploration-aware RL formulations for LLMs
- G2RL differs by measuring diversity in the model’s gradient feature space, avoiding auxiliary encoders and aligning its exploration signal with policy updates.The related-work discussion characterizes this as policy-referential similarity rather than entropy or external embedding-space similarity.Source: §5 · p. 10
6 Conclusion
The conclusion presents G2RL as a policy-intrinsic exploration method that reweights trajectories according to their update geometry. Across two Qwen3-base scales and math and general reasoning benchmarks, it reports better accuracy and more orthogonal gradient pairs while retaining coherent outputs.
6 Conclusion
- G2RL derives sequence-level features from last-layer gradient sensitivity and uses bounded reward-coupled weighting within GRPO to favor correct trajectories with new update directions.The method suppresses incoherent trajectories and is reported to break gradient collinearity without compromising training stability.Source: §6 · p. 10
- Across math and general reasoning benchmarks and two Qwen3-base scales, G2RL consistently improves pass@1, maj@16, and pass@k.The conclusion also reports healthier training dynamics and a five-fold increase in orthogonal gradient pairs while maintaining coherent, on-topic outputs.Source: §6 · p. 10
- The conclusion argues that efficient LLM-RL exploration is better guided by optimization-landscape geometry than by entropy inflation or superficial semantic dispersion.Source: §6 · p. 10
A.1.1 First-Order Sensitivity at the Last Layer
The section frames token t’s contribution to the log-likelihood gradient in last-layer space for a generated trajectory (x, y).
A.1.1 First-Order Sensitivity at the Last Layer
- For a generated trajectory (x, y), the section introduces the contribution of token t to the log-likelihood gradient in last-layer space.Source: §A.1.1 · p. 12
- The quantity is scoped to a generated trajectory (x, y).Source: §A.1.1 · p. 12
- The gradient contribution is considered in last-layer space.Source: §A.1.1 · p. 12
A.1.2 Layerwise Factorization of Backpropagation
The section relates trajectory-specific update geometry to a sequence feature Φ: variation in that feature and in layerwise operators are the stated sources of update diversity.
A.1.2 Layerwise Factorization of Backpropagation
- Update diversity across responses can arise from variation in Φ_t, reflecting last-layer token sensitivities, or variation in L_k(x, y), reflecting intermediate activations and Jacobians.Source: §A.1.2 · p. 12
- Angular dispersion among Φ(x, y) encourages updates to explore a broader parameter subspace without assumptions on the Jacobians.Φ is the shared factor through which upstream gradients pass.Source: §A.1.2 · p. 13
A.2 Training Configuration.
Training uses GRPO with Qwen3-1.7B-Base and Qwen3-4B-Base, different maximum response lengths, and a step-by-step system prompt.
A.2 Training Configuration.
- Experiments use Qwen3-1.7B-Base and Qwen3-4B-Base, with training implemented using GRPO.Source: §A.2 · p. 13
- Maximum generation lengths are 8k tokens for the 1.7B model and 12k tokens for the 4B model.Source: §A.2 · p. 13
- The system prompt asks the model to reason step by step and put its final answer within \boxed{}.Source: §A.2 · p. 13