Source-linked AI summary
LIMR: Less is More for RL Scaling
Xuefeng Li, Haoyang Zou, Pengfei Liu
TL;DR
The paper addresses limited understanding of which RL training-data requirements improve language-model reasoning and whether scaling data is necessary. It introduces LIM to select samples by alignment with learning trajectories, finding that 1,389 selected samples can match or exceed 8,523 samples and that LIMR outperforms the cited SFT approaches at 7B scale.
Problem
RL advances provide limited transparency about training-data scale, while systematic evidence on data quantity and sample selection remains limited.
Method
LIM analyzes learning dynamics to quantify sample value and select high-impact RL training data.
Results
1,389 selected samples match or exceed the 8,523-sample dataset, while LIMR achieves 16.7% higher AIME24 accuracy and surpasses LIMO and s1 by 13.0% and 22.2% on MATH500.
Takeaways & Limitations
The findings suggest that selective sample quality may matter more than simply increasing RL data quantity for improving reasoning capabilities.
Abstract
from arXiv · showhide
In this paper, we ask: what truly determines the effectiveness of RL training data for enhancing language models' reasoning capabilities? While recent advances like o1, Deepseek R1, and Kimi1.5 demonstrate RL's potential, the lack of transparency about training data requirements has hindered systematic progress. Starting directly from base models without distillation, we challenge the assumption that scaling up RL training data inherently improves performance. we demonstrate that a strategically selected subset of just 1,389 samples can outperform the full 8,523-sample dataset. We introduce Learning Impact Measurement (LIM), an automated method to evaluate and prioritize training samples based on their alignment with model learning trajectories, enabling efficient resource utilization and scalable implementation. Our method achieves comparable or even superior performance using only 1,389 samples versus the full 8,523 samples dataset. Notably, while recent data-efficient approaches (e.g., LIMO and s1) show promise with 32B-scale models, we find it significantly underperforms at 7B-scale through supervised fine-tuning (SFT). In contrast, our RL-based LIMR achieves 16.7% higher accuracy on AIME24 and outperforms LIMO and s1 by 13.0% and 22.2% on MATH500. These results fundamentally reshape our understanding of RL scaling in LLMs, demonstrating that precise sample selection, rather than data scale, may be the key to unlocking enhanced reasoning capabilities. For reproducible research and future innovation, we are open-sourcing LIMR, including implementation of LIM, training and evaluation code, curated datasets, and trained models at https://github.com/GAIR-NLP/LIMR.
1 Introduction
The paper examines whether RL reasoning performance depends more on training-data scale or on sample quality and selection. It introduces automated sample evaluation and reports that a selected subset can match or outperform full-data training, while LIMR compares favorably with SFT baselines at 7B scale.
- Motivation: RL training-data requirements remain poorly understood because prior advances provide limited transparency about their data scale.This limits systematic comparison and reuse of successful RL training practices.
- Motivation: Researchers lack clear data-scale benchmarks and systematic evidence about how sample quantity affects model performance.These gaps encourage trial-and-error resource use and complicate allocation decisions.
- Key insight: The paper challenges the assumption that larger RL datasets necessarily produce better performance, emphasizing sample quality and relevance.It frames sample selection as a potentially more fundamental factor than dataset size.
- Findings: 1,389 carefully selected RL samples achieve comparable or superior performance to the full 8,523-sample dataset.This result directly contrasts selective training with scaling the complete dataset.
- Contributions: Learning Impact Measurement (LIM) automatically predicts which RL samples contribute most to model improvement and removes the need for manual curation.The method is designed to scale to practical training workflows.
- Findings: LIMR achieves 16.7% higher AIME24 accuracy than the comparison SFT result and surpasses LIMO and s1 by 13.0% and 22.2% on MATH500.The reported AIME24 comparison is 32.5% versus 15.8%; MATH500 results are 78.0% versus 65.0% and 55.8%.
- Contributions: The authors release LIMR datasets, models, training and evaluation code, and LIM implementation details.The release is intended to support reproducibility and future research.
2 Methodology
The methodology presents Learning Impact Measurement as a systematic way to quantify training-data value in RL by analyzing learning dynamics and identifying effective samples.
- Methodology: Learning Impact Measurement quantifies and optimizes RL training-data value by analyzing learning dynamics to identify effective samples.The method targets data efficiency in reinforcement-learning training.
2.1 Learning Dynamics in RL Training
Analysis of the 8,523-problem MATH-FULL dataset shows that training samples contribute unequally to model learning and exhibit diverse learning trajectories.
- Dataset analysis: The MATH-FULL dataset contains 8,523 mathematical problems spanning difficulty levels 3–5.The analysis uses this dataset to study how training data relates to model improvement.
- Learning trajectories: Different samples display diverse learning trajectories, with some stable and others showing complex dynamics associated with significant model improvements.The observed trajectories indicate that samples do not contribute uniformly to learning.
2.2 Learning Impact Measurement (LIM)
LIM evaluates samples by how well their learning trajectories align with the model’s overall progression, providing a basis for selective sampling.
- Trajectory alignment: Figure 2 compares individual sample reward trajectories with the average reward curve to assess trajectory alignment.Higher LIM scores correspond to trajectories showing growth patterns similar to the model’s average progression.
- Trajectory alignment: The method treats alignment with the model’s overall learning progression as a measure of individual training-sample effectiveness.This operationalizes the idea that samples can contribute unequally to RL learning.
- Sample valuation: LIM uses observed learning dynamics to quantify which training samples are most effective for RL improvement.The approach is presented as a systematic alternative to treating all samples uniformly.
2.2 Learning Impact Measurement (LIM)
LIM evaluates training samples by comparing their learning patterns with the model’s overall performance trajectory, using average reward as the reference curve.
- LIM values samples whose learning patterns complement the model’s overall performance trajectory.
- The model’s average reward curve serves as the reference for measuring sample effectiveness.
- For each sample, LIM computes a normalized alignment score from its rewards across training epochs.
2.3 Baseline Data Selection Methods
LIM converts sample–trajectory alignment into a selective sampling rule, retaining samples above an adjustable quality threshold.
- Higher alignment scores indicate that a sample’s learning pattern better matches the model’s overall learning trajectory.
- With θ = 0.6, LIM selects 1,389 high-value samples from the original dataset.
2.3 Baseline Data Selection Methods
The baseline methods compare selective sampling against random selection and a linear-progress criterion, which can miss rapidly improving samples.
- RAND randomly selects 1,389 samples from MATH-FULL to match LIMR’s dataset size.
- LINEAR selects samples showing steady improvement across epochs, using θ = 0.7 to produce 1,189 samples.
- LINEAR can miss samples with rapid early gains followed by stabilization.
2.4 Reward Design
The reward design uses rule-based outcomes that distinguish correct answers, incorrectly answered but properly formatted responses, and formatting errors.
- Correct answers receive reward 1, while incorrect but properly formatted answers receive -0.5.
- Answers with formatting errors receive reward -1.
3 Experiment
Experiments compare RL training and data-selection strategies on difficult mathematical benchmarks, finding that LIMR preserves performance with substantially less data and outperforms random selection. Training dynamics and cross-benchmark results further support LIMR’s effectiveness, while matched-data comparisons favor RL over SFT for small models.
- Experimental Setup: PPO training starts from Qwen2.5-Math-7B and evaluates MATH500, AIME2024, and AMC2023.The evaluation uses repeated sampling for AIME24 and AMC23 and greedy decoding for MATH500.
- Main Results: An 80% dataset-size reduction leaves LIMR nearly on par with MATH-FULL, while MATH-RAND loses 8.1% average accuracy and MATH-LINEAR loses 2%.These results indicate that selection strategy strongly affects RL performance.
- Training Dynamics: LIMR and MATH-FULL have nearly identical accuracy curves and both significantly outperform MATH-RAND during training.LIMR’s reward rises faster and approaches 1.0, while MATH-FULL shows sequence-length instability.
- Cross-Benchmark Results: LIMR achieves performance comparable to MATH-FULL across three benchmarks while significantly outperforming the RAND baseline.Consistent results on AIME24 and AMC23 argue against overfitting to a single dataset.
- RL versus SFT: With matched small data budgets, LIMR improves over LIMO and s1 by more than 100% on AIME and by at least 10% on AMC23 and MATH500.The comparison uses 1,000 s1 samples, 817 LIMO samples, and LIMR under supervised-fine-tuning comparisons.
4 Conclusion
The paper argues that improving LLM reasoning does not require simply scaling RL data. It presents LIMR as a data-selective RL approach whose 1,389-sample subset matches or exceeds an 8,523-sample dataset across mathematical benchmarks, with potential applicability beyond mathematics.
- Conclusion: LIMR matches or exceeds the full 8,523-sample dataset across multiple challenging mathematical benchmarks using 1,389 carefully selected samples.The approach challenges the assumption that larger RL datasets are necessary for better reasoning.
- Conclusion: Learning Impact Measurement provides an automated, scalable way to select RL samples by optimizing sample quality rather than data quantity.The conclusion presents this as a practical solution for efficient RL training.
- Conclusion: The comparison with supervised fine-tuning indicates that RL combined with efficient data selection can benefit smaller models with limited data.The paper suggests potential applications beyond mathematical reasoning to other language-model RL domains.