PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic-Space Exploration and Exploitation in RLVR for LLM Reasoning

Fanding Huang, Guanbo Huang, Xiao Fan, Yi He, Xiao Liang, Xiao Chen, Qinting Jiang, Faisal Nadeem Khan, Jingyan Jiang, Zhi Wang

arXiv:2509.23808v5cs.LGcs.CL

TL;DR

Token-level proxies measure next-token uncertainty rather than semantic progress through reasoning, leaving unclear whether exploration and exploitation truly trade off. The paper measures semantic trajectories in hidden states and uses VERL to shape RLVR advantages. VERL consistently improves performance across evaluation settings, including a reported 21.4% absolute accuracy gain on Gaokao 2024.

  • Problem

    Token-level entropy and confidence reflect next-token uncertainty rather than how reasoning progresses across semantic structures, raising the question of whether the apparent exploration–exploitation trade-off is intrinsic.

    Source: §1 · pp. 1–2

  • Method

    The paper measures semantic breadth with ER and refinement dynamics with ERV and ERA, then uses ER/ERV-derived advantage shaping with ERA as an adaptive meta-control signal in VERL.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    VERL yields consistent improvements across models, RL algorithms, and reasoning benchmarks, including up to 21.4% absolute accuracy gain on Gaokao 2024.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ER and ERV are nearly uncorrelated in semantic space, supporting the conclusion that exploration and exploitation can be decoupled and improved simultaneously.

    Source: §1 · p. 2

    The study has not systematically examined training-data cleaning or curation, extension beyond math-focused verifiable-answer corpora remains open, and scaling costs for larger, more diverse datasets are unaddressed.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) for LLM reasoning is often framed as balancing exploration and exploitation in action space, typically operationalized with token-level proxies (e.g., output entropy or confidence). We argue that this apparent trade-off is largely a measurement artifact: token-level statistics reflect next-token uncertainty rather than how reasoning progresses over multi-token semantic structures. We therefore study exploration and exploitation in the hidden-state space of response trajectories. We use Effective Rank (ER) to quantify representational exploration and introduce its temporal derivatives, Effective Rank Velocity (ERV) and Effective Rank Acceleration (ERA), to characterize exploitative refinement dynamics. Empirically and theoretically, ER and ERV exhibit near-zero correlation in semantic space, suggesting the two capacities can be improved simultaneously. Motivated by this, we propose Velocity-Exploiting Rank Learning (VERL), which shapes the RLVR advantage with an auxiliary signal derived from ER/ERV and uses the more stable ERA as a meta-control variable to adaptively balance the incentives. Across multiple base models, RLVR algorithms, and reasoning benchmarks, VERL yields consistent improvements, including large gains on challenging tasks (e.g., 21.4\% in Gaokao 2024). The code is available at https://github.com/hf618/VERL.

1 Introduction

The introduction questions whether reasoning’s exploration–exploitation trade-off is intrinsic or reflects token-level measurement. It proposes hidden-state semantic measures and VERL, reporting consistent benchmark gains.

1 Introduction

  • Token-level entropy and confidence measure next-token uncertainty, which can misrepresent semantic progress across multi-token reasoning trajectories.Higher entropy may increase lexical variation without reaching new semantic directions, while lower entropy may collapse a trajectory into a familiar region.Source: §1 · pp. 1–2
  • VERL uses ER- and ERV-derived incentives, adaptively balanced by ERA, and improves results across models, RL algorithms, and benchmarks, with up to 21.4% absolute accuracy gain on Gaokao 2024.ERA tracks whether semantic refinement is accelerating or saturating and is described as more stable under short-term fluctuations.Source: §1 · p. 2
  • The paper shifts analysis to hidden-state semantic trajectories, introducing ER, ERV, and ERA to measure semantic breadth and its temporal refinement.ER represents how broadly hidden-state trajectories span semantic directions; ERV and ERA capture refinement dynamics.Source: §1 · p. 2
  • ER and ERV are nearly uncorrelated in semantic space, suggesting exploration and exploitation can be jointly improved rather than traded off.The authors argue the commonly perceived trade-off is largely an artifact of token-level measurement.Source: §1 · p. 2

2 Preliminaries

The preliminaries formulate reward-maximizing RLVR, introduce PPO and GRPO baselines, and define response and dataset hidden-state representations.

2.1 Problem Formulation and Notations

  • RLVR learns policy parameters by maximizing expected reward for a completed response trajectory evaluated against its prompt.The formulation represents a response as a token trajectory and uses a scalar reward to evaluate its quality.Source: §2.1 · pp. 2–3

2.2 Reinforcement Learning Baselines

  • PPO optimizes a clipped surrogate objective, while GRPO derives response-level advantages from sampled group rewards and applies them uniformly across output tokens.PPO clipping is intended to prevent excessively large, destabilizing policy updates.Source: §2.2 · p. 3

2.3 Hidden State Representations

  • The paper stacks final token hidden states into a response-level semantic trajectory and summarizes responses across prompts as a dataset-level semantic distribution.The response representation is a T-by-D matrix; dataset summaries are stacked across N prompts.Source: §2.3 · p. 3

3 A Hidden-State Perspective on Representational Dynamics

The paper treats hidden states as a semantic-space representation of reasoning, measuring breadth with Effective Rank (ER) and refinement dynamics with ERV and ERA. It also gives a scaling account of these metrics in terms of effective semantic directions.

3 A Hidden-State Perspective on Representational Dynamics

  • The analysis assumes semantic factors are approximately linearly decodable from hidden states, grounding its representational account in that condition.The theoretical analysis is based on this assumption about representation.Source: §3 · p. 3

3.1 Semantic Exploration: Effective Rank (ER)

  • Effective Rank (ER) measures semantic diversity through how evenly hidden states occupy non-redundant directions, and is a continuous measure bounded above by conventional rank.Unlike discrete rank, ER can rise as the singular spectrum flattens.Source: §3.1 · p. 3

3.2 Semantic Exploitation: Effective Rank Velocity (ERV) and Acceleration (ERA)

  • ERV tracks semantic breadth against historical-prefix averages rather than one-step changes, while ERA tracks whether refinement dynamics strengthen or weaken over time.Sustained positive ERV reflects representational refinement; positive ERA indicates strengthening refinement, while negative ERA is consistent with stabilization or saturation.Source: §3.2 · pp. 4–5

3.3 Scaling Properties of Semantic-Space Dynamics

  • Under an effective-support assumption of k approximately orthogonal directions, ER and ERV scale linearly with k, whereas ERA does not scale with k.At dataset level, k represents distinct semantic modes across responses; at response level, it represents reasoning steps adding non-redundant directions.Source: §3.3 · p. 5

4 Empirical Evidence for Decoupling in Semantic Space

During GRPO training, ER varies in direction across the tested base models while ERV rises in both, and correct and incorrect trajectories differ in their dynamics. At dataset level, validation performance co-evolves with ER dynamics, and ER can reveal changes that conventional rank misses.

4.1 Analysis of Response-Level Metrics

  • Across Qwen and Llama, ER rises for Qwen but falls for Llama, while ERV trends upward for both during GRPO post-training.The experiments use 8k hard MATH problems at levels 3–5 with verifiable reference answers.Source: §4 · p. 5, §4.1 · p. 6
  • Incorrect trajectories often have higher ER and ERV than correct ones, whereas ERA is consistently higher for correct trajectories and is described as a more reliable reasoning indicator.The reported pattern associates correct reasoning with refinement that strengthens over time, not simply greater breadth or faster growth relative to the past.Source: §4.1 · p. 6

4.2 Analysis of Dataset-level Metrics

  • Across training, validation performance co-evolves with dataset-level ER dynamics, while ERV and ERA trends indicate progressive refinement and expansion of semantic space.The reported performance measures include validation accuracy and score.Source: §4.2 · p. 6
  • Late in training, ER can continue rising after conventional rank plateaus, reflecting a more balanced singular spectrum and more uniform use of existing semantic directions.This captures refinement within the established space beyond what discrete rank captures.Source: §4.2 · p. 6

5 Velocity-Exploiting Rank-Learning

VERL adds hidden-state semantic signals to RL objectives, combining ER-based breadth and ERV-based refinement with ERA-controlled adaptive weighting. Its auxiliary signal is normalized against recent trends and shapes the original advantage while bounding the added incentive.

5 Velocity-Exploiting Rank-Learning

  • VERL augments standard RL objectives with ER, ERV, and ERA signals to encourage broader semantic coverage and productive refinement together.It is presented as an advantage-shaping approach for regulating how reasoning trajectories evolve in representation space.Source: §5 · pp. 6–7

5.1 Normalized Deviations in Semantic Space

  • VERL normalizes trajectory-level ER, ERV, and ERA by their recent EMA trends so each signal represents deviation from recent semantic-space behavior.The method defines relative deviation using the difference from the EMA divided by its absolute value.Source: §5.1 · p. 7

5.2 Adaptive Balancing via ERA as a Meta-Control Signal

  • VERL combines ER and ERV incentives, using ERA as a meta-control signal to adapt their relative emphasis; ERA acceleration may indicate over-collapse risk, while deceleration or saturation motivates stronger exploitation.The auxiliary signal rewards breadth or refinement above recent trends, with tanh bounding its magnitude for stability.Source: §5.2 · pp. 7–8

5.3 Advantage Shaping with Auxiliary Semantic-Space Signals

  • The method adds a clipped, non-negative auxiliary bonus to the original advantage, preserving its sign while bounding the shaping magnitude for stability.For GRPO, the auxiliary signal is applied once per trajectory; for PPO+GAE, it is broadcast to all tokens in that trajectory.Source: §5.3 · p. 8

6 Experiments

VERL improves performance across mathematical reasoning benchmarks, RL algorithms, and base models, with the strongest reported gains on challenging multi-step tasks. Ablations support adaptive ER/ERV shaping with ERA and show robustness across tested temporal strides and clipping values.

6.1 Experiment Settings

  • Training uses a rule-based reward for mathematical correctness and boxed-answer formatting, with 4 rollouts per prompt for GRPO and 1 for PPO.The maximum generation length is 1536 tokens.Source: §6.1 · p. 8

6.2 Main Results

  • VERL improves Pass@1 over RL baselines across 15 math benchmarks, including +21.4 on Gaokao 2024_I with Qwen2.5-7B and +10.0 on AIME24 with Llama-3.2-3B-Instruct.The reported gains are most pronounced on multi-step reasoning benchmarks.Source: §6.2 · p. 8
  • VERL improves average performance for both Llama and Qwen models under GRPO and PPO, indicating transfer across the tested algorithms and model families.The authors describe VERL as plug-and-play across these settings.Source: §6.2 · p. 8
  • VERL yields larger gains on Pass@k than Pass@1, especially on harder benchmarks, suggesting improved coverage of successful reasoning trajectories under larger sampling budgets.Pass@1 is used as a proxy for single-shot performance, whereas Pass@k reflects success with a larger search budget.Source: §6.2 · pp. 8–9
  • Benchmark-level drops are localized: despite some declines under GRPO, average Pass@1 improves by +2.0 for Llama and +2.1 for Qwen, and Llama’s Gaokao 2024_I change is optimizer-dependent.The passage reports GRPO drops on Llama Gaokao 2024_I (-7.1) and Qwen AMC23/CMATH (-5.0/-1.0), while Llama improves by +7.2 on Gaokao 2024_I under PPO.Source: §6.2 · p. 9

6.3 Ablations and Analysis

  • Adaptive mixing using ERA outperforms fixed mixing, supporting ERA as a signal for adjusting the relative emphasis on ER breadth and ERV refinement.The adaptive setting is β = σ(d2), compared with fixed β = 0.5.Source: §6.3 · p. 9
  • Improvements remain robust across tested temporal strides and clipping values, with s = 40 selected for best validation reward and κ = 2 giving the best performance.The stride is used to compute ERV and ERA; clipping bounds the shaping bonus relative to the original advantage.Source: §6.3 · p. 9
  • Combining exploration and exploitation shaping yields the most stable training and best final performance; either term alone has a distinct drawback.Exploration-only shaping improves coverage but underuses high-reward trajectories, while exploitation-only shaping gains faster initially but plateaus from insufficient breadth.Source: §6.3 · p. 9

7 Conclusion

The conclusion frames the apparent RLVR exploration–exploitation trade-off as largely a token-level measurement artifact and shifts analysis to hidden-state semantic space. It reports that VERL uses ER, ERV, and ERA for semantic-space advantage shaping and consistently improves performance and out-of-domain generalization on challenging reasoning benchmarks.

7 Conclusion

  • The paper argues that the apparent exploration–exploitation trade-off in RLVR is largely a measurement artifact of token-level action statistics.Its conclusion contrasts token-level statistics with hidden-state semantic-space analysis.Source: §7 · p. 9
  • ER, ERV, and ERA characterize semantic breadth and refinement dynamics, and the paper reports near-zero correlation between ER and ERV in semantic space.The conclusion presents this as a basis for considering exploration and exploitation jointly.Source: §7 · p. 9
  • Using ERA as a meta-control signal, VERL applies semantic-space advantage shaping and consistently improves in-domain performance and out-of-domain generalization on challenging reasoning benchmarks.Source: §7 · p. 9

Limitations

The paper identifies unresolved questions about VERL’s data curation, transfer beyond math-focused training data, and computational scaling. Its related-work discussion situates the approach among research on LLM reasoning, RLVR, token-level exploration proxies, and representation dynamics.

Limitations

  • The authors have not systematically studied how training-data cleaning or curation affects VERL, and extending results beyond math-focused data with verifiable answers remains open.They also have not addressed computational costs when scaling to larger, more diverse training datasets.Source: Limitations · p. 9

C.1 Reasoning with LLMs

  • Prior LLM reasoning work spans specialized and multimodal tasks, tool use, and methods for improving reasoning capability or reliability; this paper instead studies reasoning dynamics during RL training.Source: §C.1 · p. 18

C.2 Reinforcement Learning with Verifiable Rewards

  • Related RLVR work reports benefits including deeper reasoning and tool use, while some studies argue that reward maximization may suppress exploration by reinforcing high-reward trajectories.Source: §C.2 · p. 18

C.3 Exploration and Exploitation in RLVR for LLMs

  • Prior RLVR studies commonly treat next-token entropy as exploration and confidence as exploitation, but these token-level measures remain tied to uncertainty under a shared measurement framework.The paper’s approach instead studies exploration and exploitation at the response level in semantic space.Source: §C.3 · p. 18

C.4 Representation Dynamics in Deep Reinforcement Learning

  • Deep RL research links representation geometry and feature collapse to performance, sample efficiency, generalization, or transfer, motivating analysis of semantic-space dynamics in RLVR.Source: §C.4 · p. 19

D.1 Proof of Theorem 3.1

The appendix bounds effective rank by conventional matrix rank and relates its scaling to the number of semantic directions represented. It also provides theoretical support for treating ER as exploration and ERV as exploitation in hidden-state space, unlike coupled token-level metrics.

D.1 Proof of Theorem 3.1

  • For a nonzero hidden-state matrix, effective rank lies between 1 and conventional rank, which is at most min(T, D).Effective rank equals conventional rank when the nonzero singular values are all equal.Source: §D.1 · p. 19

Assume a hidden-state matrix has effective support

  • For k approximately orthogonal directions, ER and its corresponding ERV are upper-bounded linearly by k; in contrast, corresponding ERA does not scale with k.Source: Assume a hidden-state matrix has… · p. 19
  • Under maximal dataset-level semantic support, effective rank scales approximately with the number of prompts, while its first-order difference grows linearly and its second-order difference is constant.The analysis assumes a stride of 1 for deriving the difference-metric scaling orders.Source: Assume a hidden-state matrix has… · p. 20

E Additional Theoretical Support for Exploration and Exploitation Metrics

  • The theoretical analysis describes token-level average log probability and response entropy as algebraically coupled, while hidden-state ER and ERV are not; ERV is strongly correlated with greedy value improvement under the studied PPO-style architecture.It motivates ER as semantic exploration and ERV as representation-level exploitation.Source: E Additional Theoretical Support for… · p. 21

E.1 Action vs. Semantic Exploration–Exploitation Metrics

This section contrasts token-level action metrics with semantic-space metrics for exploration and exploitation, presenting terminal Effective Rank and Effective Rank Velocity as structurally independent measures. It also characterizes ERV as a lower bound on semantic exploitation strength under a positive-rate regime, growing linearly with segment length.

E.1 Action vs. Semantic Exploration–Exploitation Metrics

  • Under on-policy sampling, average log probability and response entropy are algebraically coupled, while terminal Effective Rank and Effective Rank Velocity are not functionally dependent.Thus the token-level metrics cannot vary independently, whereas the hidden-state metrics capture distinct aspects of the Effective Rank trajectory.Source: §E.1 · pp. 21–23

E.2 Effective Rank as Semantic Exploration

  • Effective Rank equals the number of equally energized orthogonal semantic directions and increases as the hidden-state spectrum spreads across more directions, making it a basis-invariant proxy for semantic exploration and uncertainty.This interpretation relies on semantic features being linearly decodable from hidden states.Source: §E.2 · pp. 23–24

E.3 Effective Rank Velocity as Semantic Exploitation

  • Effective Rank Velocity is a positive recency-weighted sum of ER increments; with fixed semantic support, continued spectrum uniformization produces positive ER drift as the model refines its chosen subspace.The exploitation regime requires the active semantic directions to remain fixed and each step to become more uniform on that support, yielding an ER increment of at least µ.Source: §E.3 · pp. 24–25
  • ERV quantitatively lower-bounds the persistent positive drift in effective rank along a trajectory, making it a measure of semantic exploitation strength.The proposition identifies ERV as a lower bound on the semantic exploitation rate.Source: §E.3 · p. 25
  • With hidden states satisfying Assumptions E.3 and E.4 and an exploitation regime of rate μ > 0, ERV is strictly positive and grows linearly with segment length K.Source: §E.3 · p. 25
  • When a model has discovered a useful semantic subspace and consistently enriches it, ERV lower-bounds the rate of semantic-complexity exploitation within that subspace.In this idealized regime, the lower bound is strictly positive and grows linearly.Source: §E.3 · p. 25

F.1 Training and Evaluation Details

The appendix details the training and evaluation setup, an incremental method for computing temporal metrics, and analyses of overhead and performance. It reports that VERL improves Pass@k exploration, particularly on harder benchmarks.

F.1 Training and Evaluation Details

  • Training uses 4 A800 GPUs, batch size 48, rollout length 1536, and 4 GRPO or 1 PPO rollout per prompt; evaluation uses temperature 0.6 and up to 2048 tokens.The evaluation reports Pass@1 for most benchmarks and Pass@256 accuracy as well as Pass@1 for AIME 2024.Source: §F.1 · p. 26

F.2 Efficient Incremental Computation of Higher-Order Metrics

  • Incremental Gram/covariance updates reduce matrix-construction cost from O(DT^3/s) to O(DT^2), making temporal dynamics feasible to compute.When T exceeds D and D and stride s are constants, the respective costs scale as O(T^2) and O(T).Source: §F.2 · pp. 26–27

F.3 Time Overhead of VERL Training

  • Across base models, computing hidden-state metrics adds negligible training time, even when ER, ERV, and ERA are computed on CPU.The authors describe this CPU evaluation as a worst-case stress test.Source: §F.3 · p. 27, Figure 6 · p. 27

G.1 Detailed Analysis of Response-Level Hidden States

  • Response-level hidden-state analyses across additional LLMs confirm that the earlier findings hold across base models and RL paradigms.The response-level figures show metrics for overall batches and for correct and incorrect samples.Source: §G.1 · p. 27, Figure 7 · p. 28, Figure 8 · p. 29

G.2 Detailed Analysis of Dataset-Level Hidden States

  • Dataset-level hidden-state analyses likewise confirm the earlier findings across additional LLMs and RL paradigms.The figures compare traditional and proposed metrics alongside validation score and sample correctness.Source: §G.2 · p. 27, Figure 9 · p. 30, Figure 10 · p. 31

G.3 Detailed Analysis of Pass@1 Performance

  • Pass@1 is treated as a measure of single-attempt answer generation and is evaluated after integrating VERL-based advantages into GRPO and PPO.Table 4 reports Pass@1 comparisons for instruction-tuned models.Source: §G.3 · pp. 27, 32, Table 4 · p. 32

G.4 Detailed Analysis of Pass@k Performance

  • VERL improves exploration under larger decoding budgets, especially on harder benchmarks.Table 5 reports performance across diverse decoding settings.Source: §G.4 · p. 32

G.5 Robustness Across Random Seeds

The appendix reports seed robustness and out-of-domain gains for VERL, compares it with token-level regularizers, and examines the hidden layer used for its metrics. Case studies illustrate improved numerical reasoning and constraint use.

G.5 Robustness Across Random Seeds

  • Across five seeds for LLAMA-3.2-3B-INSTRUCT, adding VERL to GRPO raises average Pass@1 from 39.24±0.72 to 45.18±0.38 and Pass@k from 59.87±1.37 to 62.78±0.63.Two-sided Welch’s t-tests at α = 0.05 assess each benchmark; most improvements are significant. For tasks without significance marks, means still improve, but seed variance is slightly larger.Source: §G.5 · p. 32

G.6 Detailed Analysis of OOD Performance

  • Mathstral-7B GRPO OOD average rises from 39.31 to 41.68 (+2.37); VERL also improves averages for PPO and Qwen2.5-7B.The gains are reported across heterogeneous domains for both RL algorithms and both models.Source: Table 10 · p. 34, §G.6 · p. 34

G.7 Comparison with Regularization Method in Action Space

  • VERL gives the strongest overall Pass@1 and Pass@k results, while combining token-level confidence and entropy does not reliably beat the better single proxy.VERL has the best or second-best Pass@1 scores on most datasets and leads on key high-budget benchmarks.Source: §G.7 · p. 35, Table 11 · p. 35, Table 12 · p. 35

G.8 Ablation on the Choice of Hidden Layer

  • Compared with layer 14, GRPO + VERL using the final hidden layer raises average Pass@1 from 41.69% to 43.44% and average Pass@k from 61.52% to 64.40%.Source: §G.8 · p. 36

H.1 Case Study For Pass@1 Setting

  • In case studies, auxiliary shaping corrects negative-number comparisons and helps the model respect structural and solution-form constraints that vanilla GRPO overlooks.Examples include inferring 20 houses per street and producing a purely exponential solution when required.Source: §H.1 · p. 36

H.2 Case Study For Pass@16 Setting

In two Pass@16 case studies, auxiliary shaping yields correct solutions where vanilla GRPO does not, and the geometry example shows broader strategy use.

H.2 Case Study For Pass@16 Setting

  • On the elementary integer problem, vanilla GRPO always outputs 3, while auxiliary shaping produces the correct answer 1 in 5 of 16 rollouts.The condition is x ≥ 1 and x ≠ 2.Source: §H.2 · p. 36, Figure 14 · p. 40
  • On the geometry problem, auxiliary shaping succeeds in 5 of 16 rollouts where vanilla GRPO succeeds in none, with more varied solution strategies.The shaped model uses the law of sines 9 times, the law of cosines 4 times, and other properties 3 times; vanilla GRPO uses the law of sines 11 times.Source: §H.2 · pp. 36–37, Figure 15 · p. 45
Loading 2509.23808v5…