PaperlayerView exact revision on arXiv

Source-linked AI summary

LIMO: Less is More for Reasoning

Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, Pengfei Liu

arXiv:2502.03387v3cs.CLcs.AI

TL;DR

The paper addresses whether sophisticated mathematical reasoning in LLMs requires massive post-training datasets. It uses simple supervised fine-tuning with a compact, carefully curated dataset designed around pre-trained knowledge and explicit reasoning chains. LIMO achieves strong benchmark and out-of-domain results while using only 800 examples, supporting the LIMO hypothesis that minimal demonstrations can elicit advanced reasoning in knowledge-rich models.

  • Problem

    Complex mathematical reasoning in LLMs has conventionally been treated as requiring tens or hundreds of thousands of training examples.

  • Method

    LIMO curates high-quality reasoning demonstrations and applies simple supervised fine-tuning to Qwen2.5-32B-Instruct.

  • Results

    LIMO achieves 78.1% average performance across benchmarks and outperforms models trained on datasets orders of magnitude larger.

  • Takeaways & Limitations

    In knowledge-rich foundation models, sophisticated reasoning can emerge from minimal, precisely orchestrated demonstrations that use inference-time computation.

Abstract

from arXiv · show

We challenge the prevailing assumption that complex reasoning in large language models (LLMs) necessitates massive training data. We demonstrate that sophisticated mathematical reasoning can emerge with only a few examples. Specifically, through simple supervised fine-tuning, our model, LIMO, achieves 63.3\% accuracy on AIME24 and 95.6\% on MATH500, surpassing previous fine-tuned models (6.5\% on AIME24, 59.2\% on MATH500) while using only 1\% of the training data required by prior approaches. Furthermore, LIMO exhibits strong out-of-distribution generalization, achieving a 45.8\% absolute improvement across diverse benchmarks, outperforming models trained on 100x more data. Synthesizing these findings, we propose the Less-Is-More Reasoning Hypothesis (LIMO Hypothesis): In foundation models where domain knowledge has been comprehensively encoded during pre-training, sophisticated reasoning can emerge through minimal but strategically designed demonstrations of cognitive processes. This hypothesis suggests that the threshold for eliciting complex reasoning is not dictated by task complexity but rather by two key factors: (1) the completeness of the model's pre-trained knowledge base and (2) the effectiveness of post-training examples in serving as "cognitive templates" that guide reasoning.

1 Introduction

The paper argues that complex reasoning need not require massive fine-tuning datasets when foundation models already encode rich knowledge. LIMO tests this premise with curated demonstrations and simple supervised fine-tuning, reporting strong mathematical and generalization performance.

  • Reasoning training is conventionally treated as requiring tens or hundreds of thousands of examples because tasks involve multi-step deduction, domain knowledge, and structured solutions.
  • Modern foundation models may already contain extensive mathematical knowledge from large-scale pre-training, creating conditions for more data-efficient reasoning elicitation.
  • LIMO identifies prerequisite knowledge in model parameters and effective minimal exemplars as the two factors governing complex-reasoning elicitation.
  • 800 curated training samples with simple SFT yield 63.3% accuracy on AIME and 95.6% on MATH, using 1% of prior models’ training data.The approach also outperforms models trained on 100x more data across unseen scenarios.
  • The paper’s contributions include establishing the LIMO hypothesis, constructing the dataset, demonstrating benchmark performance, and analyzing data-selection principles through ablations.

2 Related Work

Related work has generally improved reasoning by scaling pre-training or post-training data, increasing test-time computation, or using reinforcement learning. Prior alignment results show that small curated datasets can generalize, but whether this extends to computationally demanding reasoning remained uncertain.

  • Large-scale pre-training and post-training datasets have been used to capture cognitive patterns and teach reasoning by scaling questions and solutions.
  • Test-time scaling improves reasoning by increasing generated tokens through methods such as parallel sampling and symbolic tree search.
  • Small curated instruction datasets can teach formats and generalize to unseen tasks, but their applicability to reasoning tasks remained uncertain because of reasoning complexity.

3 LIMO Dataset

The LIMO dataset operationalizes the hypothesis that rich pre-trained knowledge and explicit, high-quality reasoning chains can make minimal demonstrations effective. Its pipeline filters difficult problems, evaluates diverse solution qualities, and retains the highest-scoring 800 question–solution pairs.

  • LIMO Hypothesis: The LIMO hypothesis requires latent prerequisite knowledge and reasoning chains that decompose complex problems into detailed, logical, traceable steps.
  • Dataset Formalization: The dataset represents verifiable questions, reasoning chains of intermediate steps, and answers, with quality depending on both question and solution quality.
  • Question Selection: Candidate questions are selected for difficulty, complex reasoning, knowledge integration, and diverse problem-solving approaches.
  • Question Selection: The source pool combines established mathematics datasets and historical AIME problems spanning high-school through advanced competition levels.
  • Question Filtering: A multi-stage filtration pipeline removes problems solved within four attempts by a smaller model, then evaluates survivors with a stronger reasoning model.
  • Solution Curation: Three reasoning models generate multiple solutions, which authors collaboratively examine for quality and diversity.
  • Solution Scoring: Solution scoring measures elaboration, self-verification, exploration, and adaptive granularity using weighted, length-normalized rule-based metrics.The weights are 30%, 20%, 25%, and 25%, respectively.
  • Final Dataset: From 2,125 questions, the pipeline ranks each problem’s highest-scoring solution and selects the top 800 pairs for the LIMO Dataset.

4 Training Recipe

The training recipe relies on simple supervised fine-tuning of a knowledge-rich base model using a few hundred carefully selected examples. It assumes that long-chain reasoning at test time allows the model to integrate meta-reasoning into coherent solution chains.

  • The recipe assumes substantial pre-trained reasoning knowledge and the ability to perform long-chain reasoning at test time.
  • With a few hundred carefully selected SFT examples, the model learns to integrate meta-reasoning tasks into cohesive reasoning chains.
  • LIMO fine-tunes Qwen2.5-32B-Instruct with full-parameter SFT using DeepSpeed ZeRO-3 and FlashAttention-2.
  • Optimization uses a 5.0e-6 learning rate with cosine decay and no warmup phase.

5 Evaluation Framework

The evaluation framework measures mathematical reasoning both within the training distribution and on diverse out-of-distribution tasks, using established and newly constructed benchmarks.

  • The framework covers both in-domain and out-of-distribution evaluations across established benchmarks and novel multilingual tests.This design examines generalization beyond the training distribution.
  • In-domain Evaluation: In-domain evaluation includes AIME24, MATH500, and AMC23 mathematical competitions and benchmarks.
  • Out-of-distribution Evaluation: Out-of-distribution evaluation spans OlympiadBench, CHMath, Gaokao, Kaoyan, and GradeSchool.These benchmarks differ from the training data across mathematical challenges and educational settings.
  • Performance metrics: Performance is measured with pass@1 in a zero-shot chain-of-thought setting across all benchmarks.Larger benchmarks use greedy single-sample decoding, while smaller benchmarks use four samples at temperature 0.6 and unbiased pass@1.

6.1 Baselines

LIMO is compared with prominent reasoning models, its base model, and larger supervised fine-tuning datasets using a shared backbone to isolate training-data effects.

  • Baselines include OpenAI-o1-preview, QwQ-32B-Preview, and the base model Qwen2.5-32B-Instruct.These comparisons cover advanced reasoning models and LIMO’s underlying model.
  • Comparative fine-tuning experiments use mainstream open-source reasoning datasets on the same LLM backbone as LIMO.This setup attributes performance differences to training-data characteristics rather than backbone differences.
  • The comparison datasets contain substantially more samples than LIMO’s 800-example training set.

6.2 Main Results

LIMO outperforms the compared models across in-domain and out-of-domain benchmarks, while its carefully curated 800 examples outperform substantially larger training datasets.

  • In-domain Performance: LIMO achieves the best results across all in-domain benchmarks.
  • In-domain Performance: 63.3% accuracy on AIME24 surpasses QwQ-32B-Preview at 50.0% and OpenAI-o1-preview at 44.6%.
  • In-domain Performance: 95.6% accuracy on MATH500 surpasses QwQ-32B-Preview at 89.8% and OpenAI-o1-preview at 85.5%.
  • Out-of-domain Generalization: 67.6% accuracy on OlympiadBench exceeds QwQ-32B-Preview at 58.5% and the base model at 45.3%.LIMO also reaches 84.2% on CHMath and 76.2% on GradeSchool.
  • Comparison with Larger Datasets: LIMO’s curated 800 problems yield 78.1%, outperforming larger datasets including NuminaMath-100k at 32.3% and OpenThoughts-114k at 58.3%.The comparison attributes the stronger result to targeted selection and curation rather than data quantity alone.
  • Overall Performance: 78.1% is LIMO’s highest average performance across all benchmarks, exceeding OpenAI-o1-preview, QwQ-32B-Preview, and other baselines.The evaluation reports this result despite LIMO using 800 training examples versus more than 100k for other fine-tuned models.

6.3 Analysis

The analysis tests how reasoning-chain quality, question difficulty, pre-trained model choice, model size, and dataset size affect LIMO’s mathematical reasoning performance.

  • Reasoning Chain Quality: L5-trained models achieve the highest AIME24 and MATH500 results, with performance decreasing consistently across lower reasoning-chain quality levels.The substantial L5–L1 gap underscores the importance of curating high-quality training data.
  • Question Quality: Advanced-500 training yields 91.2% on MATH500 without in-domain training data, indicating that harder questions improve reasoning across datasets.The question sets progress from Simple-500 through Complex-500 to Advanced-500.
  • Pre-trained Knowledge: 63.3% AIME2024 accuracy and 95.6% MATH500 accuracy with Qwen2.5-based LIMO exceed Qwen1.5 by 54.1 and 30.4 percentage points, respectively.The comparison controls architecture and fine-tuning while varying the pre-trained model choice.
  • Model Size: AIME24 performance rises from 2.5 for the 3B model to 68.3 for the 72B model, while MATH500 gains are less dramatic.The models use the same supervised fine-tuning recipe and 800 high-quality samples.
  • Sample Efficiency: 400 samples raise AIME24 accuracy from 16.5 to 57.5 and MATH500 accuracy from 79.4 to 94.8, with diminishing returns beyond 800 samples.The reported gains between LIMO-800 and LIMO-1k2 are +0.9 on AIME24 and -0.2 on MATH500.

7 Conclusion

LIMO shows that complex mathematical reasoning can be achieved with surprisingly few examples, challenging the assumption that massive training data is necessary. In knowledge-rich foundation models, minimal, precisely orchestrated demonstrations can elicit sophisticated reasoning using inference-time computation.

  • LIMO achieves complex mathematical reasoning with surprisingly few examples, contradicting the assumption that massive training data is necessary.The approach uses only 1% of the training data required by previous approaches.
Loading 2502.03387v3…