PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't

Quy-Anh Dang, Chris Ngo

arXiv:2503.16219v2cs.LGcs.CL

TL;DR

The paper asks whether reinforcement learning can improve reasoning in small LLMs without the large datasets and computational budgets typically used for such systems. It adapts GRPO and curates mathematical reasoning data for a 1.5-billion-parameter model under strict hardware and time constraints. Open-RS reports strong benchmark gains at low cost, while prolonged training exposes instability and length-related limitations.

  • Problem

    The study examines how small LLMs behave under strict computational constraints and whether RL-based fine-tuning can elevate their reasoning performance.

  • Method

    The authors adapt GRPO and curate a compact mathematical reasoning dataset to train DeepSeek-R1-Distill-Qwen-1.5B on 4 NVIDIA A40 GPUs within 24 hours.

  • Results

    Open-RS raises AMC23 accuracy from 63% to 80% and reaches 46.7% on AIME24, surpassing o1-preview, at an approximately $42 training cost.

  • Takeaways & Limitations

    RL-based fine-tuning can produce competitive reasoning performance in small LLMs with minimal resources, supporting lightweight reasoning-capable models.

  • Takeaways & Limitations

    Prolonged training causes unstable accuracy, rising completion lengths, unreadable content, and non-English outputs, indicating reward misalignment and length constraints.

Abstract

from arXiv · show

Enhancing the reasoning capabilities of large language models (LLMs) typically relies on massive computational resources and extensive datasets, limiting accessibility for resource-constrained settings. Our study investigates the potential of reinforcement learning (RL) to improve reasoning in small LLMs, focusing on a 1.5-billion-parameter model, DeepSeek-R1-Distill-Qwen-1.5B, under strict constraints: training on 4 NVIDIA A40 GPUs (48 GB VRAM each) within 24 hours. Adapting the Group Relative Policy Optimization (GRPO) algorithm and curating a compact, high-quality mathematical reasoning dataset, we conducted three experiments to explore model behavior and performance. Our results demonstrate rapid reasoning gains - e.g., AMC23 accuracy rising from 63% to 80% and AIME24 reaching 46.7%, surpassing o1-preview - using only 7,000 samples and a $42 training cost, compared to thousands of dollars for baseline models. However, challenges such as optimization instability and length constraints emerged with prolonged training. These findings highlight the efficacy of RL-based fine-tuning for small LLMs, offering a cost-effective alternative to large-scale approaches. We release our code and datasets as open-source resources, providing insights into trade-offs and laying a foundation for scalable, reasoning-capable LLMs in resource-limited environments. All are available at https://github.com/knoveleng/open-rs.

1 Introduction

The paper examines whether reinforcement learning can improve reasoning in small LLMs under strict computational constraints. It reports competitive mathematical reasoning performance with limited resources while identifying data-efficiency, stability, and length-related challenges.

  • Motivation: Small LLMs offer a resource-efficient deployment alternative, but prior RL fine-tuning often requires hundreds of thousands to millions of samples or substantial computational cost.These constraints motivate studying reasoning improvement in resource-limited settings.
  • Research Questions and Setup: The study investigates DeepSeek-R1-Distill-Qwen-1.5B using adapted GRPO reinforcement learning on 4 NVIDIA A40 GPUs within 24 hours.Evaluation uses mathematical reasoning benchmarks as a testbed for structured problem-solving.
  • Headline Results: Open-RS achieves 46.7% on AIME24, exceeding o1-preview at 44.6%, while its training cost is approximately $42.The figure compares zero-shot pass@1 performance against model size and computational cost.
  • Contributions: The authors systematically analyze small-LLM reasoning potential under computational constraints and provide practical insights into scalability and deployment feasibility.The work connects RL-based fine-tuning to real-world applicability and releases code and datasets for reproducibility.
  • Challenges: The paper identifies data efficiency, optimization stability, and completion-length constraints as challenges for realizing small, reasoning-capable LLMs.These issues are presented alongside the reported promise of RL-based methods.

2 Methodology

The methodology combines compact mathematical-data curation with resource-efficient reinforcement learning. It uses filtering, refinement, group-relative optimization, and rule-based rewards to balance reasoning performance with computational overhead.

  • Method Overview: The approach combines a high-quality mathematics-focused dataset with a resource-efficient RL algorithm for training small LLMs.The design targets performance gains under reduced computational overhead and practical constraints.
  • Dataset Curation: The curated sources are the s1 and DeepScaleR datasets, filtered and refined to retain relevant and challenging mathematical reasoning examples.The curation process is intended to support efficient learning for small models.
  • Dataset Curation: The s1 filtering workflow reduces the corpus from 59,029 questions to 31,323 after retaining boxed-answer solutions, then to 21,533 after removing trivial questions.A further Qwen2.5-7B-Instruct filtering step removes noisy or multipart questions.
  • Dataset Curation: The final dataset contains 39,659 mathematical reasoning questions, combining 18,615 refined open-s1 examples with 21,044 open-deepscaler examples.The corpus is designed to balance scale and mathematical specificity.
  • Reinforcement Learning Algorithm: GRPO estimates baselines from group scores instead of using a separate critic model, reducing computational overhead during policy optimization.The policy samples grouped outputs from the old policy and optimizes the current policy with a KL-divergence penalty.
  • Reward Design: The reward system combines correctness, response-length efficiency, and structure through rule-based components rather than resource-intensive neural reward models.Accuracy requires a boxed final answer, while cosine reward scales rewards according to response length.

3 Experiments

Three resource-constrained reinforcement-learning experiments tested how dataset composition, length limits, and reward design affect reasoning in a 1.5B model. The models achieved rapid benchmark gains and competitive overall performance, but prolonged training remained unstable.

  • Experiment 1: High-Quality Data: 63% to 70% on AMC23 and 83% to 84% on MATH-500 within 50–100 steps, but both benchmarks deteriorated after 200 steps during Experiment 1.AMC23 fell below 60% and MATH-500 reached 80%; outputs also became unreadable and non-English as lengths increased.
  • Overall Comparison: Open-RS1, Open-RS2, and Open-RS3 averaged 53.0%, 55.7%, and 56.3%, while Open-RS3 reached 46.7% on AIME24 and Open-RS2 reached 80.0% on AMC23.Open-RS3’s AIME24 score surpassed o1-preview at 44.6% and DeepScaleR-1.5B-Preview at 43.1%.
  • Overall Comparison: The approach used 7000 samples and cost approximately $42, compared with $1088–$3629 for listed baseline models requiring substantially more resources and data.The authors characterize small-model RL as a scalable alternative to resource-intensive baselines.

4 Conclusion

The study examines RL-based reasoning enhancement for small LLMs and reports promise under resource constraints, while positioning the work against limitations in prior large-model approaches.

  • Study scope: The study investigates reasoning enhancement for a 1.5-billion-parameter model under strict computational constraints.The model is DeepSeek-R1-Distill-Qwen-1.5B, studied under a constrained setting.
  • Contribution: The work adapts RL-based fine-tuning to small LLMs and releases code and curated datasets as open-source resources.The study presents the resources as supporting reproducibility and further exploration.
  • Findings: RL-based fine-tuning improves small-LLM reasoning with minimal resources, with AMC23 accuracy rising from 63% to 80% and AIME24 reaching 46.7%.The reported cost was $42, compared with thousands of dollars for baselines.
  • Conclusion: The paper frames these results as evidence that small LLMs can achieve competitive reasoning performance despite the resource demands associated with larger-model approaches.The conclusion emphasizes the promise of RL-based methods for small LLMs.
  • Motivation: Prior RL-based reasoning methods often target large models and require substantial computational resources, leaving small-model applications under resource and privacy constraints underexplored.The paper identifies this as a gap in the literature.

B Limitations & Discussion

The paper identifies limitations and broader implications that qualify its evidence on RL-based reasoning enhancement for small LLMs.

  • Scope: The authors state that their findings require contextualization through discussion of limitations and broader implications.This framing motivates the limitations discussed later in the paper.

Limitations

The experiments are bounded by limited training time and length capacity, while multilingual drift and mathematics-only evaluation restrict interpretation and generalizability.

  • Training constraints: A 24-hour window on 4 NVIDIA A40 GPUs limited training to 500 global steps, restricting exploration of behavior beyond 200 steps.The paper contrasts 500 available steps with a potential 1500 steps for one epoch.
  • Length constraints: Maximum completion lengths of 4096 tokens in Experiment 1 and 3584 in Experiments 2 and 3 were insufficient for extremely hard open-s1 problems.The resulting truncation may underexploit small-LLM potential on tasks requiring extended reasoning chains.
  • Language drift: The multilingual base model introduced unintended language drift after 150–200 steps despite prompts intended to enforce English-only outputs.The authors characterize this as a trade-off in using a multilingual foundation.
  • Evaluation scope: Evaluation covered only mathematical reasoning benchmarks, leaving generalizability to scientific reasoning and coding unexplored.The authors recommend cautious interpretation within the specified scope.

Discussion

The discussion presents a trade-off between efficient reasoning gains and reasoning depth, with curriculum-like data and cosine rewards offering distinct but incomplete remedies.

  • Efficiency: Rapid gains in the first 50–100 steps suggest that small, high-quality datasets can bootstrap reasoning capabilities efficiently.The paper relates this observation to prior work on data efficiency in RL.
  • Optimization: Later degradation indicates sensitivity to over-optimization under fixed length constraints.The discussion notes a similar challenge in larger models such as DeepSeek-R1.
  • Training strategy: Mixed-difficulty training in Experiment 2 indicates that curriculum-like strategies could mitigate degradation.The paper presents this as an indication rather than a definitive result.
  • Reward design: The cosine reward stabilizes reasoning verbosity but sacrifices peak accuracy compared with Experiment 2.Its stabilizing effect is presented as a promising direction for controlling response length.
  • Comparative performance: Open-RS variants rival or exceed some state-of-the-art 1.5B and 7B models at a fraction of the cost and data volume.The discussion presents this efficiency as an alternative for resource-constrained environments.

Future Directions

The paper identifies research directions for addressing truncation, language drift, and limits on generalization. It concludes that RL-based fine-tuning can improve reasoning in small LLMs under stringent constraints while balancing performance and accessibility.

  • Future Directions: Longer training or multi-stage length schedules could address truncation while preserving stability.These approaches are intended to help the model handle harder problems.
  • Future Directions: Lightweight language rewards or monolingual pre-filtering could mitigate language drift and improve output consistency.
  • Future Directions: Expanding benchmarks beyond mathematics would test the generalizability of the approach.
  • Conclusion: RL-based fine-tuning can unlock substantial reasoning potential in small LLMs under stringent constraints.The paper frames this as supporting efficient reasoning-capable models that balance performance and accessibility.

C Datasets

The evaluation uses a diverse collection of mathematical benchmarks spanning different problem types and difficulty levels. The listed datasets include AIME24, AMC23, MATH-500, and Minerva.

  • Datasets: AIME24 contains 30 advanced high-school-level problems from the 2024 American Invitational Mathematics Examination.
  • Datasets: AMC23 contains 40 problems testing foundational mathematical skills from the 2023 American Mathematics Competition.
  • Datasets: MATH-500 is a 500-problem subset spanning algebra, calculus, and geometry.
  • Dataset Coverage: The diverse benchmark collection is intended to assess reasoning generalization across problem types and difficulty levels.

D Baseline Models

The baseline suite spans small mathematics-focused models, general-purpose large models, and mathematics-focused 7B models. It includes models trained with GRPO, RL, reward models, tree search, and related methods.

  • General-Purpose Large Models: General-purpose comparison models include Llama-3.1-70B-Instruct and OpenAI’s o1-preview.
  • Mathematics-Focused 7B Models: The 7B mathematics-focused baselines include Qwen-2.5-Math-7B-Instruct, rStar-Math-7B, Eurus-2-7B-PRIME, and Qwen2.5-7B-SimpleRL.Their described approaches include reward modeling, Monte Carlo Tree Search, online RL with process rewards, and related RL methods.
  • Baseline Methods: The related-work baseline description also notes PPO with rewards based on final answers.
  • Mathematics-Focused 1.5B Models: DeepSeek-R1-Distill-Qwen-1.5B serves as the original untrained baseline.
  • Mathematics-Focused 1.5B Models: DeepScaleR-1.5B-Preview uses GRPO on 40,000 math problem-answer pairs across multiple RL stages.
  • Mathematics-Focused 1.5B Models: Still-3-1.5B-Preview uses RL with a slow-thinking focus, including tree search, on 30,000 curated math examples.

E Hyperparameter Setup

The training setup is documented through a hyperparameter table for the GRPO trainer. The section directs readers to the table for the parameters used during training.

  • Hyperparameter Setup: Table 5 summarizes the parameters used during the training phase.
  • Hyperparameter Setup: The section’s training details are therefore organized as a tabulated GRPO hyperparameter setup.
  • Hyperparameter Setup: The hyperparameter configuration is presented specifically for the GRPO trainer.
Loading 2503.16219v2…