The Invisible Leash: Why RLVR May or May Not Escape Its Origin

Fang Wu; Weihao Xuan; Ximing Lu; Mingjie Liu; Yi Dong; Zaid Harchaoui; Yejin Choi

Source-linked AI summary

The Invisible Leash: Why RLVR May or May Not Escape Its Origin

Fang Wu, Weihao Xuan, Ximing Lu, Mingjie Liu, Yi Dong, Zaid Harchaoui, Yejin Choi

arXiv:2507.14843v4Jul 20, 2025cs.LGcs.AIcs.CL

TL;DR

The paper asks whether RLVR expands LLM reasoning or mainly sharpens solutions already accessible to the base model, and investigates this through empirical-support analysis. It finds improved precision alongside reduced solution diversity: support shrinkage generally outweighs expansion, while token-level uncertainty can rise as answer-level diversity falls. The authors conclude that extending beyond the base model’s reach may require explicit exploration or off-policy mechanisms.

Problem
It remains unclear whether current RLVR expands a model’s reasoning boundary or mainly amplifies high-reward outputs already present in the base model’s distribution.
Method
The paper empirically analyzes RLVR through empirical support, defined by correct solutions realistically discoverable under finite sampling.
Results
RLVR increases precision while support shrinkage generally outweighs support expansion, and it can increase token-level entropy while reducing answer-level entropy.
Takeaways & Limitations
Extending reasoning beyond the base model’s effective reach may require explicit exploration or off-policy mechanisms that allocate probability mass to underrepresented solution regions.
Takeaways & Limitations
Across 1.5B–14B models, NDR never exceeds 0.04 and NSCR remains non-positive, limiting evidence for genuine support expansion.

Abstract

from arXiv · show

Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising method for enhancing LLMs' capabilities. However, it remains unclear whether the current practice of RLVR truly expands a model's reasoning boundary or mainly amplifies high-reward outputs that the base model already knows, thereby improving precision. This study presents an empirical investigation that provides fresh insights into the limits of RLVR. We examine how RLVR can operate as a support-constrained optimization mechanism that may restrict the discovery of entirely original solutions, remaining constrained by the base model's initial distribution. We also identify an entropy-reward trade-off: while RLVR reliably enhances precision, it may progressively narrow exploration and potentially overlook correct yet underrepresented solutions. Extensive empirical experiments validate that while RLVR consistently improves \texttt{pass@1}, \textit{the shrinkage of empirical support generally outweighs the expansion of empirical support under larger sampling budgets}, failing to recover correct answers that were previously accessible to the base model. Interestingly, while RLVR sometimes increases token-level entropy, it results in greater uncertainty at each generation step and declining answer-level entropy. This indicates that these seemingly more uncertain paths ultimately converge onto a smaller set of distinct answers. Taken together, we reveal potential limits of RLVR in extending reasoning horizons. Breaking this invisible leash requires future innovations that seed probability mass into underrepresented solution regions.

1. Introduction

The paper investigates whether RLVR expands reasoning beyond a base model’s accessible solutions or mainly sharpens existing high-reward behavior. Across domains, it finds improved precision alongside reduced solution diversity and support coverage.

Motivation: RLVR improves single-attempt accuracy but can underperform the base model when multiple attempts are allowed.This pattern motivates distinguishing precision gains from broader reasoning improvements.
Approach: The paper defines empirical support as correct solutions a model can realistically discover under finite sampling.This framework examines solution accessibility rather than only before-and-after accuracy.
Findings: Across reasoning benchmarks, RLVR loses access to more correct solutions than it gains while improving single-sample accuracy.Training progressively narrows the accessible solution space.
Findings: The precision–diversity trade-off appears across mathematics, logical reasoning, factual QA, and code generation.The reported pattern is not limited to one specialized domain.
Findings: RLVR can increase token-level entropy while reducing answer-level entropy, producing more uncertain steps that converge on fewer final solutions.The paper presents this divergence as evidence that local stochasticity does not ensure global diversity.
Implications: The authors describe an “invisible leash”: RLVR remains constrained by the base model’s effective reach and may require explicit exploration to reach underrepresented regions.Proposed directions include exploration or hybrid strategies that seed probability mass into those regions.

2. Related Works

Related work positions RLVR as objective-reward optimization while examining evaluation limits, verifier reliability, self-verification, efficiency, and exploration. These studies motivate more faithful reasoning metrics and methods that address sparse or imperfect feedback.

RLVR foundations: RLVR replaces subjective human-preference reward models with objective, algorithmically computable signals tied to verifiable success criteria.It has supported gains in applications including mathematics and medical multiple-choice QA.
Evaluation: Pass@K may overstate RLVR gains by ignoring intermediate reasoning correctness, motivating metrics that assess reasoning chains and final answers together.CoT-Pass@K is presented as one such alternative.
Algorithmic variants: RLVR variants address reward sparsity and verifier imperfections through asymmetric-noise modeling, self-verification, and denser feedback.These approaches integrate robustness or critique into the optimization loop.
Exploration: Other work studies efficiency-oriented objectives, supervised surrogates, and structural incentives intended to decouple exploration from exploitation.These directions target stabilization under sparse verifiable signals and richer hidden-state dynamics.

3. Numerical Metrics

The paper formalizes empirical support as practically reachable correct completions under finite sampling and introduces metrics that distinguish preservation, discovery, expansion, and shrinkage. These measures characterize whether RLVR broadens or redistributes accessible solutions.

Support formalization: The framework models RLVR as learning πθ(y | x) from a base distribution q(y | x) under verifiable reward and optional divergence regularization.The regularization strength is controlled by β > 0.
Support formalization: Correct-completion support contains outputs y with R(x, y) = 1 and positive probability under a distribution.Answer-level empirical support treats traces yielding the same verified outcome as one equivalence class.
Support formalization: Empirical support excludes completions below a cutoff ϵ because finite-sample training rarely observes extremely low-probability outputs.The threshold represents practically observable likelihood and is derived using sampling-confidence bounds.
Support dynamics: Support expansion occurs when RLVR raises a correct completion from q(y* | x) ≤ ϵ to πθ(y* | x) > ϵ.Such completions had negligible empirical probability under the base model but become practically reachable after training.
Support dynamics: Support shrinkage occurs when a completion moves from q(y* | x) > ϵ to πθ(y* | x) ≤ ϵ.This formalizes exclusion of correct solutions previously accessible under the base model.
Metrics: High SRR with low NDR indicates support-constrained optimization, whereas high SRR with high NDR indicates genuine capability expansion.The metric combinations distinguish preservation, discovery, inefficient redistribution, and aggressive exploration.

4. Evidence of Hidden-Support Dynamics

Across models and domains, RLVR mostly preserves existing correct-solution coverage while concentrating probability mass and producing limited, often transient expansion. Its support dynamics favor precision over discovery, with shrinkage generally exceeding expansion and reducing compatibility with diverse reasoning modes.

RLVR predominantly preserves base-model solution coverage across diverse models and domains, acting as a support-constrained optimization mechanism.Table 1 categorizes completions by preservation, shrinkage, expansion, and out-of-support status.
SRR remains high at approximately 0.93–0.99, while NDR stays at or below 0.04, indicating strong retention but rare discovery.Nemotron-7B and Nemotron-14B retain nearly all base-model solutions, while ProRL-1.5B-v2 reaches SRR = 0.93 and NDR = 0.02.
48 new completions across 11 benchmarks were discovered by ProRL-1.5B-v2, while larger models added fewer than 10 per domain.Expansion was relatively higher in non-math tasks and virtually stagnant in math tasks, with isolated Reasoning Gym exceptions.
175 completions were lost versus 48 gained by ProRL-1.5B-v2, producing a shrinkage-to-expansion ratio of approximately 3.6:1.Other models show similar ratios of approximately 2:1–3:1, and overall NSCR remains between −0.01 and −0.06.
On AIME2024, perplexity on Claude Sonnet 4 traces increased from 8.76 for Base to 14.91 for ProRL, indicating reduced probability assignment to diverse external reasoning styles.Format differences contribute, but the reported dominant effect is reduced compatibility with broader reasoning modes.
During training, high SRR persists while NSCR gradually decreases, revealing steady contraction of accessible solutions after transient exploration.The resulting concentration on stable trajectories helps explain early gains and later degradation on more diverse mathematics tasks.
RLVR improves precision mainly by concentrating mass on known high-reward solutions, rather than by expanding reasoning capacity.Across 1.5B–14B models and math, non-math, and multimodal domains, SRR stays near one, NDR near zero, and NSCR is uniformly negative.
Observed expansion recomposes existing subskills or corrects prompt-format misalignment, rather than introducing solutions fundamentally absent from the base distribution.Expansion cases involve modular task structure, weakly represented component combinations, or format variants already accessible to the base model.

5. Entropy Reduction and pass@k Trade-off

RLVR improves precision while reshaping uncertainty in different ways: answer-level entropy declines consistently, whereas token-level entropy varies across models. This creates a pass@k trade-off in which sharper output distributions can reduce global solution diversity.

Results: RLVR consistently improves avg@32, raising ProRL from 54.5% to 65.4% and DAPO from 43.0% to 61.3%.The gains are accompanied by reduced answer-level entropy.
Entropy dynamics: Answer-level entropy decreases under RLVR, indicating convergence onto fewer distinct solutions and narrower effective support coverage.The effect is consistent across the reported benchmarks.
Entropy dynamics: Token-level entropy varies across RLVR models, increasing for ProRL and DAPO but remaining similar or lower for AceReason and Skywork.Prior work also reports sharp entropy collapse during early training phases.
Entropy dynamics: Higher token-level entropy does not necessarily indicate broader exploration, because models can remain locally stochastic while converging to fewer final answers.Skywork-OR1-7B has lower token-level entropy than AceReason-7B but higher answer-level entropy, demonstrating the decoupling.
Implications: The entropy trade-off is especially consequential for tasks with multiple valid answers or benefits from creative reasoning, motivating diversity-promoting exploration mechanisms.Precision gains can come at the expense of global diversity.

6. Conclusion

The paper concludes that current RLVR sharpens probability mass around known high-reward trajectories while largely preserving the base model’s support. Extending reasoning beyond that scope therefore requires explicit mechanisms that seed probability mass into underrepresented solution regions.

Conclusion: RLVR increases precision by concentrating probability mass on known high-reward trajectories while largely preserving the base model’s support.This concentration can exclude valid alternatives that the more diverse base model could recover.
Conclusion: Extending reasoning beyond the base model’s scope requires explicit exploration strategies or off-policy mechanisms that allocate mass to underrepresented solution regions.

Impact Statements

The paper frames RLVR as a method that reshapes access to correct solutions rather than necessarily introducing new capabilities. Its evidence highlights improved precision alongside potentially narrower solution diversity, clarifying the trade-offs of RLVR-based systems.

Impact Statements: The study examines how RLVR changes the accessibility of correct solutions rather than introducing new capabilities.
Impact Statements: The findings aim to clarify the trade-off between improved precision and potentially narrower solution diversity in RLVR-based systems.

A. Detailed Statistics for Support Dynamics

The detailed statistics provide per-model and per-domain support-dynamics measures underlying aggregate results. They show how RLVR variants redistribute probability mass across correct solutions while documenting the evaluation setup used for fair comparison.

Detailed Statistics for Support Dynamics: Per-model statistics report correct-completion counts for Preservation, Expansion, Shrinkage, and Out-of-Support categories across Math, Non-Math, and Overall domains.They derive SRR, NDR, SDS, and NSCR from these counts.
Detailed Statistics for Support Dynamics: The expanded tables distinguish whether single-sample accuracy gains reflect preservation, genuine discovery, or trade-offs between support expansion and shrinkage.
Detailed Statistics for Support Dynamics: Results cover ProRL-1.5B-V2, Nemotron-1-7B, Skywork-OR1-7B, AceReason-Nemotron-1-14B, Phi4-Reason-Plus-14B, and Kangheng-OVR-7B across math and non-math domains.The tables substantiate predominant preservation, limited expansion, and consistent shrinkage.
Experimental Details: The evaluation details include dataset descriptions, methodologies, and answer processing for Reasoning Gym to ensure format-compatible comparisons between base and ProRL models.
Experimental Details: Inference uses vLLM with temperature 0.6, top p 0.95, and a maximum response length of 32768 for all models.

B.2. Datasets

The evaluation spans math, visual-math, non-math, and reasoning benchmarks, with enhanced answer processing used to address format-related inconsistencies, especially for the base model on Reasoning Gym.

Datasets: The study evaluates math, visual-math, non-math, and reasoning tasks, including MATH500, MathVision, SimpleQA, LiveBench, and Reasoning Gym.Math evaluation uses complete datasets for several benchmarks, while SimpleQA uses a 10% sample and LiveBench uses a dated version.
Datasets: Reasoning Gym evaluation uses its easy set as the default configuration and its hard set as a challenging benchmark.The easy set follows the repository version updated after commit 17a8431.
Evaluation considerations: Base-model evaluation on Reasoning Gym required enhanced answer extraction and prompt design because format adherence and limited capacity caused inconsistencies.These changes produce differences between ProRL’s reported performance and the study’s Reasoning Gym results.
Answer processing: The answer-processing protocol prioritizes Reasoning Gym’s extract_answer() function, then falls back to boxed-answer extraction and final-answer extraction after </think> markers.The hierarchy bypasses later processing once a higher-priority method succeeds.

B.3.2. TASK-SPECIFIC PROCESSING MODIFICATIONS

Task-specific processing standardizes base-model and evaluation outputs to match benchmark answer formats, while shared processing is supplemented with task-dependent transformations.

General processing: The core answer-processing pipeline applies to both models, with additional steps targeting format compatibility issues in base-model responses.Task-specific modifications address benchmark requirements such as grids, JSON lists, fractions, palindromes, floating-point values, and scientific notation.
Dice: Dice processing converts base-model a/b fractions to LATEX format, while ProRL preserves a/b formatting for both answers and ground truth.The two paths reflect differing output conventions between the models.
Palindrome generation: Palindrome processing removes excess whitespace so character strings can be evaluated for reversibility without spacing interference.For example, spaced outputs are compacted into contiguous character strings.
Advanced geometry: Advanced-geometry processing removes redundant LATEX markers, converts degree symbols, and transforms numerical expressions into benchmark-compatible floating-point values.Arithmetic expressions are also evaluated into three-decimal floating-point numbers.
Power function: Power-function processing converts mixed LATEX and arithmetic scientific notation into e-notation to prevent format-incompatible errors.Examples include transformations such as “−2.36 × 10−16” to “-2.36e-16”.
Task-specific formats: Arc 1d processing inserts spaces between digits, while boxnet processing cleans JSON wrappers, converts dictionaries into lists, and filters illegal elements.These transformations align outputs with the required ground-truth formats.

B.4. Entropy Analysis

Entropy analysis samples 32 completions per problem under fixed decoding settings and measures accuracy, response length, token-level entropy, and answer-level entropy across evaluated models.

Setup: Each model generates 32 completions per problem using temperature 0.6, top p 0.95, and a maximum response length of 32768 tokens.Metrics are averaged across completions and test problems under the avg@32 protocol.
Models: The entropy analysis covers reasoning models across different training paradigms and parameter scales.The evaluated models are summarized in a dedicated table.
Entropy computation: Token-level entropy is computed with teacher forcing after generation, while answer-level entropy is based on extracted final answers.A single forward pass provides token probability distributions for each generated sequence.

C.1. Support Preservation: Why RLVR Rarely Discovers New Modes

The paper argues that standard RLVR preserves the base model’s support rather than expanding it, concentrating probability on high-reward outputs and trading single-sample precision for broader-sample coverage.

Support preservation: RLVR updates cannot assign positive probability to correct solutions that the base model assigns zero probability.On-policy gradients depend on sampled outputs, so unreachable solutions contribute no gradient signal.
Support preservation: Theorem C.1 formalizes that the RLVR policy’s support remains a subset of the base model’s support.This establishes a support-preservation guarantee for standard on-policy gradient updates.
Sampling bound: lim k→∞ pass@k_πθ(x) ≤ lim k→∞ pass@k_q(x), giving RLVR an asymptotic sampling ceiling relative to the base model.The bound assumes independent sampling across draws.
Precision–coverage trade-off: RLVR improves pass@1 by concentrating probability on known high-reward modes but can reduce pass@k for large k by narrowing coverage.The base model’s long-tail distribution may retain correct solutions that RLVR downweights.
Variational interpretation: The variational view characterizes RLVR as either a hard projection or soft trade-off that remains close to the base distribution.Exponential tilting emphasizes high-reward completions while preserving the base model’s support constraints.
Entropy–coverage trade-off: As β increases, RLVR can collapse onto a narrower subset of correct completions and miss valid alternatives accessible under the more dispersed base distribution.This reduces the long-run probability of recovering diverse solutions through large-k sampling.
Empirical implications: The empirical precision–diversity trade-off appears across mathematics, logical reasoning, factual QA, and code generation.The paper presents this cross-domain consistency as evidence that the pattern is not domain-specific.