PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-Efficient RLVR via Off-Policy Influence Guidance

Erle Zhu, Dazhi Jiang, Yuan Wang, Xujun Li, Jiale Cheng, Yuxian Gu, Yilin Niu, Aohan Zeng, Jie Tang, Minlie Huang, Hongning Wang

arXiv:2510.26491v2cs.LG

TL;DR

Heuristic RLVR data selection lacks theoretical guarantees and generalizes poorly, while influence estimation for large models faces costly rollouts and high-dimensional gradients. The paper estimates influence from offline trajectories, uses sparse projection, and builds CROPI to select data across stages. On a 1.5B model, CROPI achieves 2.66× step-level acceleration using 10% of the data per stage.

  • Problem

    RLVR data selection is largely heuristic, often based on difficulty or uncertainty, and lacks theoretical performance guarantees and generalizability across scenarios.

    Source: §1 · p. 1

  • Method

    The paper estimates data influence from offline trajectories, uses sparse random projection for high-dimensional gradients, and applies CROPI to select influential data across training stages.

    Source: §1 · p. 1, §4.2 · p. 5

  • Results

    2.66× step-level acceleration versus full-dataset training on a 1.5B model, using only 10% of the data per stage.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support influence-based data selection as a potentially efficient alternative to common heuristics for online RLVR.

    Source: §7 · p. 9

    Experiments are limited to single-turn mathematics question answering, predominantly 1.5B and 7B models, and fewer than 1000 training steps.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Data selection is a critical aspect of Reinforcement Learning with Verifiable Rewards (RLVR) for enhancing the reasoning capabilities of large language models (LLMs). Current data selection methods are largely heuristic-based, lacking theoretical guarantees and generalizability. This work proposes a theoretically-grounded approach using influence functions to estimate the contribution of each data point to the learning objective. To overcome the prohibitive computational cost of policy rollouts required for online influence estimation, we introduce an off-policy influence estimation method that efficiently approximates data influence using pre-collected offline trajectories. Furthermore, to manage the high-dimensional gradients of LLMs, we employ sparse random projection to reduce dimensionality and improve storage and computation efficiency. Leveraging these techniques, we develop \textbf{C}urriculum \textbf{R}L with \textbf{O}ff-\textbf{P}olicy \text{I}nfluence guidance (\textbf{CROPI}), a multi-stage RL framework that iteratively selects the most influential data for the current policy. Experiments on models up to 7B parameters demonstrate that CROPI significantly accelerates training. On a 1.5B model, it achieves a 2.66x step-level acceleration while using only 10\% of the data per stage compared to full-dataset training. Our results highlight the substantial potential of influence-based data selection for efficient RLVR.

1 Introduction

The paper addresses the limits of heuristic RLVR data selection with influence-based estimation and introduces CROPI to select data across training stages. On a 1.5B model, CROPI achieves 2.66× step-level acceleration while using 10% of the data per stage.

1 Introduction

  • Existing RLVR data-selection methods rely mainly on heuristics such as difficulty or uncertainty, which lack theoretical performance guarantees and generalize poorly across scenarios.The cited methods are Wang et al. (2025), Bae et al. (2025), Li et al. (2025), Zhao et al. (2025), and Sun et al. (2025).Source: §1 · p. 1
  • Influence functions estimate an individual data point’s contribution to the learning objective and offer stronger theoretical guarantees and finer-grained information than heuristic approaches.The paper grounds its use of influence functions in Hampel (1974) and Koh and Liang (2017).Source: §1 · p. 1
  • Off-policy influence estimation uses pre-collected offline trajectories to evaluate data influence without costly new rollouts, while sparse random projection reduces gradient dimensionality.The paper also reports that dropout before projection mitigates numerical noise and improves efficiency while preserving inner products.Source: §1 · pp. 1–2
  • 2.66× step-level acceleration versus full-dataset training on a 1.5B model, while using only 10% of the data per stage.Experiments cover models from 1.5B to 7B parameters and varying context lengths.Source: §1 · p. 2
  • CROPI selects the highest-influence data for the current policy checkpoint at each stage of a multi-stage reinforcement learning process.The framework leverages off-policy influence estimation for this iterative data selection.Source: §1 · pp. 1–2
  • Sparse random projection reduces the dimensionality of LLM gradients, and applying dropout before projection improves computational efficiency and inner-product preservation while mitigating numerical noise.This addresses the storage and computation demands of high-dimensional gradients.Source: §1 · p. 2

2 Preliminaries

The preliminaries model reasoning with verifiable rewards as an MDP and introduce GRPO and influence functions. They frame data attribution as estimating how individual examples contribute to objective change, a question still open for RLVR.

2 Preliminaries

  • RLVR is modeled as an MDP in which tokens are actions, generated prefixes are states, and the correctness-based reward is assigned at the terminal step.The trajectory is a sequence of state-token pairs.Source: §2 · p. 2
  • GRPO builds on PPO but estimates advantages using group-normalized returns, avoiding the critic model overhead required by PPO.The returns are from trajectories sampled under the previous policy.Source: §2 · p. 2
  • Influence functions decompose objective change into contributions from individual data points, but their use for RLVR LLM data selection remains an open question.The paper describes influence functions as a gradient-based approach to data attribution, citing Hampel (1974) and Koh and Liang (2017).Source: §2 · p. 2

3 Influence Estimation in RLVR

The section defines prompt influence through policy-gradient inner products, then addresses the rollout cost and gradient scale that make influence estimation difficult in large RLVR models. It combines off-policy gradient estimation with sparse random projection to make influence estimation more practical.

3 Influence Estimation in RLVR

  • A training prompt’s influence on a test query is measured by the inner product of their policy gradients.The section uses a first-order influence-function formulation.Source: §3 · pp. 2–3
  • Rollouts make online influence estimation costly, while full-parameter gradients create substantial storage and projection overhead.In the 1.5B setting, one rollout step averages 319.80s versus 35.91s for forward-plus-backward, an 8.91× difference.Source: §3 · p. 3

3.1 Off-Policy Gradient for Rollout-Free Estimation

  • Offline trajectories from behavior policy β provide gradients for policy πθ, with the estimator relying on their KL-constrained proximity.The paper sets β = πθ0 and reports that its off-policy gradient approximates the on-policy gradient to a certain extent.Source: §3.1 · pp. 3–4

3.2 Sparse Random Projection for Full Gradient Compression

  • Sparse random projection selects gradient dimensions before projection, and experiments find efficient, accurate inner-product rank preservation versus direct projection.The selected dimensions form a smaller projection input.Source: §3.2 · p. 4

3.3 Final Solution

  • The final POPI estimator combines off-policy gradients with sparse projection and normalizes gradient features before taking their inner product, equivalent to cosine similarity.The section names this practical off-policy influence estimation.Source: §3.3 · p. 4

4 Curriculum RL with Off-Policy Influence Guidance

The section uses POPI scores to select training prompts by their estimated influence on validation data, then applies that selection iteratively as a curriculum. At each phase, the current policy guides selection before GRPO optimization on the retained subset.

4.1 Data Selection with POPI

  • POPI scores a training prompt against a validation set using its average gradient feature across validation prompts.For multiple validation sets, Reciprocal Rank Fusion combines the prompt’s POPI ranks; higher ranks (smaller rank values) contribute more to the fused score.Source: §4.1 · pp. 4–5

4.2 Curriculum RL with Data Selection

  • CROPI iteratively filters training data toward the points estimated to be most influential.The framework uses phase-level selection to balance long-term planning with dynamic selection.Source: §4.2 · p. 5
  • At each phase, CROPI scores all training instances with the current policy, selects the top ⌊α|Dtr|⌋ samples, and runs GRPO for E steps before the next phase.After M phases, the procedure outputs the final policy.Source: §4.2 · p. 5

5 Experiments

Experiments compare CROPI with data-selection baselines on mathematical benchmarks and report stronger targeted-task performance, including a 2.66× step-level speedup using 10% of the data per phase.

5.1 Setup

  • The evaluation uses instruction-tuned models from 1.5B to 7B, 47K training problems, and standard math benchmarks; baseline comparisons use a 0.1 selection ratio on the 1.5B model.A validation set of up to 100 examples is sampled from some test sets for targeted selection; other test sets are designated untargeted.Source: §5.1 · pp. 5–6

5.2 Main Results

  • CROPI consistently outperforms full-data training and data-selection baselines on targeted tasks; on the 1.5B model it achieves 2.66× step-level speedup using 10% of the data per phase.Each phase re-scores the training pool with the current policy, selects the top 10% of prompts, and trains on that refreshed subset.Source: §5.2 · p. 6, Table 1 · p. 6, Figure 3 · p. 7
  • CROPI also improves performance on untargeted benchmarks, while static-utility baselines plateau quickly as training evolves.The reported comparison spans different model scales in targeted tasks, and the untargeted benchmarks were not used to create the selection validation set.Source: §5.2 · pp. 6–7
  • CROPI achieves a 2.16x overall speedup after accounting for data-selection time, and reaches 70% targeted accuracy with a 2.17x wall-clock speedup over standard GRPO on the 1.5B model.Selection remains non-negligible because gradients are computed over the filtered dataset.Source: §5.2 · p. 7, Table 2 · p. 7

6 Analysis

The analysis examines sparse random projection and CROPI-selected data, finding improved gradient-similarity ranking under sparsity and selected prompts that are relevant yet challenging for the evolving model.

6.1 Analysis on Sparse Random Projection

  • Sparse random projection omits a randomly selected proportion of gradient dimensions before projection; the analysis compares projected and full-parameter cosine-similarity rankings.The evaluation samples 50 training prompts and uses precision@10% to measure preservation of the top-10% similarity ranking.Source: §6.1 · pp. 7–8
  • At sparse ratio 0.1, precision@10% reaches nearly 80%, compared with around 13% when projecting the full gradient at ratio 1.The authors hypothesize that sparsity filters numerical noise amplified by projection, improving the signal-to-noise ratio; Figure 4 reports the ranking-preservation experiments.Source: Figure 4 · p. 7, §6.1 · p. 8

6.2 Analysis on data selected by CROPI

  • Top-100 POPI-selected prompts have higher semantic similarity to the validation set than randomly sampled prompts and the bottom-100 prompts.The comparison uses cosine similarity between prompt embeddings and the average validation-set embedding; Figure 5 presents the semantic-similarity analysis.Source: §6.2 · p. 8, Figure 5 · p. 8
  • For MATH-selected prompts, offline pass rate falls from 0.75 to around 0.53 while online pass rate rises from 0.75 to 0.87 across training.The selected problems become harder for the base model while remaining within the current model’s learning frontier; the reported online pass-rate range is 0.6 to 0.9.Source: §6.2 · p. 8

7 Conclusion

The paper presents influence-based selection with offline trajectories as a principled approach for RLVR and reports faster training with less data. Its analysis also contrasts the base model’s pass rates with those of the evolving model on selected prompts.

7 Conclusion

  • CROPI uses off-policy influence estimation from offline trajectories to select impactful training data in a curriculum framework.This approach is introduced to avoid the prohibitive cost of online policy rollouts.Source: §7 · p. 8
  • For the MATH top-100 prompts, the figure compares base-model offline pass rates with current-model online pass rates to show difficulty and learning progress on selected data.Offline pass rate indicates inherent difficulty; online pass rate indicates the current model’s learning progress.Source: Figure 6 · p. 9
  • Influence-based data selection achieves a speedup in RL training while using substantially less data than full-dataset training, supporting its role as a theoretically grounded, efficient alternative to common heuristics.Source: §7 · p. 9

Limitations

The paper identifies theoretical and experimental boundaries on its influence estimates and derives first-order influence from policy-gradient updates. Its derivation approximates the objective change using gradient alignment between training and test prompts.

Limitations

  • Off-policy gradient estimates lack an analysis of their errors, bias, or variance, which the authors leave for future work.The analysis is restricted to first-order influence under SGD.Source: Limitations · p. 9
  • Gradients estimated from base-model trajectories can be zero for prompts the base model answers entirely correctly or incorrectly, because their GRPO advantage is zero.The authors suggest stronger or negative examples as a possible way to provide nonzero gradient signals for all prompts.Source: Limitations · p. 9
  • Experiments cover single-turn mathematics questions, mainly 1.5B and 7B models, with fewer than 1000 training steps.The authors identify multi-turn, other reasoning, agentic and multimodal tasks, larger models, and longer runs as future directions.Source: Limitations · p. 9

A Derivation of First-order Influence Function

  • The first-order derivation approximates the next-step objective with a Taylor expansion and uses a positive-sign gradient update for stochastic gradient ascent.For its epoch-level approximation, it treats an epoch as one large-batch step and holds parameters constant within the epoch.Source: A Derivation of First-order Influence… · p. 12
  • First-order influence scores a training prompt by the inner product of its gradient with a test prompt's gradient at the same checkpoint.For multiple test samples, the derivation considers influence with respect to their distribution.Source: A Derivation of First-order Influence… · p. 12

B Derivation of Off-Policy Gradient

This material derives an off-policy gradient estimator and situates CROPI among RLVR data-selection approaches. It also describes the experimental setup and reports comparative convergence, selection, and rollout-cost findings.

B Derivation of Off-Policy Gradient

  • The estimator uses token-level importance sampling because trajectory-level importance weights can have high variance and destabilize gradient estimates.The approximation assumes the training policy and behavior policy are relatively close; the paper cites KL values usually below 0.1 during RLVR training.Source: B Derivation of Off-Policy Gradient · pp. 13–14

C.1 Data Selection for RLVR

  • Prior RLVR selection methods use global, batch, or phase-level selection, trading off adaptation to training dynamics against warmup needs, variance, and distribution instability.Heuristic utilities often rely on prompt difficulty or uncertainty, while online rollout costs motivate compromises that can miss changes across training stages.Source: §C.1 · p. 15

C.2 Influence Functions for LLM Data Selection

  • Influence functions have been used for language-model pretraining and supervised fine-tuning, but their application to RLVR remains challenging because gradients and rewards require costly rollouts.A concurrent RLHF study uses a different target function and training objective.Source: §C.2 · p. 15

D Algorithmic pseudocode of CROPI

  • CROPI iteratively scores prompts for the current policy checkpoint, selects the highest-scoring fraction, and trains that phase with GRPO.The algorithm repeats this selection and optimization across phases.Source: D Algorithmic pseudocode of CROPI · p. 16

E.1 Implementation Details

  • The experiments use Qwen2.5 1.5B and 7B models and a 1.5B distilled model, with a 47K-query math training set and model-specific validation and test benchmarks.The 1.5B-R1 experiments use a random 25K-query subset; reported test results exclude validation examples.Source: §E.1 · p. 16

E.3 Baselines in Main Results

  • CROPI is compared with difficulty- and pass-rate-based global selection, single-phase influence selection, and DAPO; the main comparison aligns baseline hyperparameters.DAPO filters perfect- or zero-pass-rate samples and replaces them from a historical-trajectory buffer.Source: §E.3 · p. 16

E.4 Additional Convergence and Time-based Comparison

  • CROPI reaches peak targeted accuracy earlier and achieves slightly higher best accuracy than standard GRPO within 2000 steps, and it outperforms GRPO at 12h, 18h, and 24h.The additional comparisons use the 1.5B setting.Source: Table 3 · p. 17, §E.4 · p. 17, Table 4 · p. 17

E.5 Random Selection Baseline

  • Phase-wise random selection of 10% of the data underperforms both full-data GRPO and CROPI, while rollout generation costs substantially more than a forward-plus-backward step.The rollout-cost comparison uses batch size 128 and sequence length capped at 8192 in the 1.5B setting.Source: §E.5 · p. 17, F Additional Analysis on Rollout Cost · p. 17

G Additional Analysis on Off-policy Gradient Estimator

The analysis compares off-policy gradient estimates with on-policy gradients at CROPI's step-200 checkpoint. Most paired gradients have cosine similarity above 0.6, but top-rank preservation remains limited.

G Additional Analysis on Off-policy Gradient Estimator

  • 40 of 50 on-policy and off-policy gradient pairs have cosine similarity greater than 0.6.The pairs come from problems where neither the base model nor the step-200 CROPI checkpoint had perfect success or complete failure across eight rollouts.Source: Figure 7 · p. 18, G Additional Analysis on Off-policy… · p. 18
  • Top-10% rank preservation is 28.80%, above a random baseline but leaving substantial room for improvement.The analysis compares row-wise rankings from 50×50 cosine-similarity matrices built from the two gradient sets.Source: G Additional Analysis on Off-policy… · p. 18

H Additional Analysis on Gradient Projection

The gradient-projection analysis finds better similarity-ranking preservation with sparse projection than with full gradients. The authors suggest reduced numerical error as a possible explanation.

H Additional Analysis on Gradient Projection

  • On the 3B model, precision@10% is only slightly above random with full gradients, but exceeds 80% at sparse ratio 0.1.The same rank-preservation pattern was observed in the 1.5B experiment.Source: H Additional Analysis on Gradient… · p. 18
  • The authors hypothesize that sparsification filters numerical error in float16 projection, but leave further investigation of this explanation to future work.They note that sparsification clusters more projected values near zero, which might help avoid precision loss in rank preservation.Source: H Additional Analysis on Gradient… · pp. 18–19

I.1 Data Source of Selected Prompts

The selected prompts shift with the validation target and training progress, becoming more task-focused and, for MATH, increasingly difficult. The appendix also reports lower semantic diversity among the most influential prompts and lower training entropy under data selection than with full data.

I.1 Data Source of Selected Prompts

  • 71% of Round 0’s top 100 prompts came from GSM8K and 6% from MATH; by Round 4 (step=800), 99% were GSM8K, with MATH nearly excluded.The text says this selection also removes GSM8K examples ineffective for MATH validation, potentially detrimental to cross-task generalization.Source: §I.1 · p. 19, Figure 10 · p. 20, Figure 11 · p. 20

I.2 Knowledge Domains of Selected Prompts

  • MSC categorization found that selected MATH prompts’ knowledge-domain distribution closely mirrors the validation set and shifts as training progresses.The analysis classified validation questions and selected problems with GPT-4o using the Mathematics Subject Classification system.Source: §I.2 · p. 20, Figure 12 · p. 21

I.3 Semantic Diversity of Selected Prompts

  • The top-100 selected prompts have lower semantic diversity than random and bottom-100 samples; multiple diverse validation sets can mitigate this, and CROPI gains on untargeted tests.The passage notes that semantically similar prompts could potentially limit generalization.Source: §I.3 · p. 20, Figure 13 · p. 22

I.4 Human Annotated Difficulty-level of Selected Prompts

  • As training progresses, CROPI increasingly favors MATH-Train samples with higher human-annotated difficulty levels.The passage interprets this trend as requiring progressively more challenging problems as model capability boundaries expand.Source: §I.4 · p. 20, Figure 14 · p. 22

I.5 Most influential questions

  • Top-100 influence scores rise steadily and stabilize around 0.5, while bottom-100 prompts have gradient-space cosine similarity below zero with the validation set.The paper visualizes these influence-score trends across selection rounds.Source: §I.5 · pp. 20, 22, Figure 15 · p. 22

J Additional Results on CROPI Experiments

Related papers

Loading 2510.26491v2…