PaperlayerView exact revision on arXiv

Source-linked AI summary

NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions

Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Ilia Kulikov, Kyunghyun Cho, Dong Wang, Yuandong Tian, Jason E Weston, Xian Li

arXiv:2502.13124v4cs.CL

TL;DR

Reasoning beyond math and coding is limited by a shortage of diverse, high-quality questions. NaturalReasoning addresses this gap with a 2.8-million-question dataset and evaluates it for distillation and unsupervised self-training, finding consistent benchmark improvements and effective external or self-rewarding approaches.

  • Problem

    Existing reasoning datasets often cover narrow domains with short, easily verified solutions, while broader reasoning problems are open-ended.

  • Method

    NaturalReasoning generates challenging, diverse questions and reference answers from pretraining corpora using LLMs, then evaluates them in distillation and self-training settings.

  • Results

    2.8 million questions support consistent reasoning-benchmark improvements through scaled distillation, while self-rewarding enables effective unsupervised self-training.

  • Takeaways & Limitations

    NaturalReasoning provides a broad dataset for transferring and self-training LLM reasoning across multiple domains.

  • Takeaways & Limitations

    The study focuses mainly on large-scale offline training, and models trained on the data may exhibit undesirable behavior not examined in the work.

Abstract

from arXiv · show

Scaling reasoning capabilities beyond traditional domains such as math and coding is hindered by the lack of diverse and high-quality questions. To overcome this limitation, we introduce a scalable approach for generating diverse and challenging reasoning questions, accompanied by reference answers. We present NaturalReasoning, a comprehensive dataset comprising 2.8 million questions that span multiple domains, including STEM fields (e.g., Physics, Computer Science), Economics, Social Sciences, and more. We demonstrate the utility of the questions in NaturalReasoning through knowledge distillation experiments which show that NaturalReasoning can effectively elicit and transfer reasoning capabilities from a strong teacher model. Furthermore, we demonstrate that NaturalReasoning is also effective for unsupervised self-training using external reward models or self-rewarding. To foster future work, we publicly release NaturalReasoning at https://huggingface.co/datasets/facebook/natural_reasoning.

1 Introduction

NaturalReasoning addresses the shortage of diverse, open-ended reasoning data by providing 2.8 million challenging questions with reference answers. The dataset supports post-training through knowledge distillation and self-training.

  • Existing reasoning datasets often target narrow domains with short, easily verified solutions, whereas broader reasoning problems are frequently open-ended.
  • The dataset combines easy-to-verify answers with open-ended solutions, supporting reasoning research across broader domains.
  • NaturalReasoning uses pretraining corpora and LLMs to create diverse real-world reasoning questions without extra human annotation.
  • NaturalReasoning questions improve sample efficiency in knowledge distillation compared with existing datasets.
  • Self-rewarding techniques using NaturalReasoning can achieve performance comparable to some strong external reward models.

2 Data Collection

NaturalReasoning is built by using LLMs to identify reasoning-rich pretraining documents, synthesize grounded questions, verify reference answers, and filter redundancy and benchmark contamination.

  • Question generation: LLMs annotate pretraining documents to identify sophisticated reasoning traces before question generation.The sources are DCLM-baseline and FineMath pretraining corpora.
  • Question generation: For selected documents, an LLM composes self-contained, challenging questions based on document content.This produces questions that are not directly extracted from the source text.
  • Dataset comparison: NATURALREASONING is compared with existing reasoning datasets using statistics including question count, domain coverage, and question length.Table 1 defines question length by number of words.
  • Answer verification: Each synthesized question is checked for a reference answer derivable from the source document, when possible.The pipeline also generates an additional response with Llama-3-70B-Instruct.
  • Filtering: Near-duplicate questions are removed with word-level locality-sensitive min-hashing using a similarity threshold of 0.55.The filtering targets redundant variations with the same core reasoning task.

3 Data Analysis

NaturalReasoning is a large, diverse, and difficult reasoning dataset with broad domain coverage, long questions, long model responses, and mostly multiword reference answers.

  • Utility: Models trained on NATURALREASONING achieve better performance across general reasoning benchmarks than models trained on OpenThoughts.Reference answers also support filtering for knowledge distillation and reinforcement learning with verifiable rewards.
  • Scale and coverage: 2.8M unique questions make NATURALREASONING substantially larger than OpenMathInstruct-2, NuminaMath, and MetaMathQA.The compared datasets contain 607K, 860K, and 395K questions, respectively.
  • Scale and coverage: NATURALREASONING covers more diverse domains than large open reasoning datasets that primarily focus on mathematics.Its topic coverage includes non-Math areas such as Physics, Computer Science, and Social Science.
  • Difficulty and diversity: 55 words is the average NATURALREASONING question length, exceeding OpenMathInstruct-2 (46), WebInstruct (34), NuminaMath (48), and MetaMathQA (41).The authors associate longer questions with richer context and multi-step requirements.
  • Difficulty and diversity: 434 words is NATURALREASONING’s median Llama3.3-70B-Instruct response length, the longest among the compared datasets.Response length is used as a proxy for question difficulty.
  • Reference answers: At least 50.9% of questions have long reference answers of 10 or more words, while 81.68% have answers derivable from pretraining data.Single-word answers account for 10.7%, and 2–9-word answers account for 20.0%.

4 Experimental Setup

The experiments test whether NaturalReasoning transfers reasoning capabilities through supervised knowledge distillation and supports unsupervised self-training across diverse reasoning benchmarks.

  • Experimental settings: The study evaluates NATURALREASONING in knowledge distillation and unsupervised self-training settings.Knowledge distillation uses supervised finetuning to transfer reasoning capabilities to student models.
  • Experimental settings: Student experiments include Llama3.1-8B and Qwen2.5-7B model families.The distillation experiments examine whether NATURALREASONING enables steeper scaling than existing datasets.
  • Evaluation: Evaluation covers MATH, GPQA, GPQA-Diamond, and MMLU-Pro, with broader NLP tasks such as writing evaluated in Appendix H.All trained models use zero-shot evaluation, vLLM inference, and greedy decoding.

5 Steeper Scaling with Challenging and Diverse questions

NaturalReasoning questions support more sample-efficient scaling than comparison datasets across averaged reasoning benchmarks. Math-specialized data performs best on MATH but shows weaker or plateauing transfer to GPQA and MMLU-Pro.

  • 5.1 Results: Figure 4 reports Llama3.1-8B-Base scaling results, including average performance across three benchmarks and benchmark-level breakdowns by dataset size.Qwen2.5-7B shows the same reported superiority of NATURALREASONING in Appendix E.
  • 5.1 Results: NATURALREASONING requires fewer training examples to achieve superior average performance across MATH, GPQA, and MMLU-Pro.With 1.5M examples, it outperforms Llama3.1-8B-Instruct, while OpenMathInstruct-2 and WebInstruct do not surpass that baseline at 2.8M.
  • 5.1 Results: OpenMathInstruct-2 achieves the highest MATH scores, increasing from 50.83 at 500K to 59.25 at 2.8M.Its specialization is strongest on the MATH benchmark.
  • 5.1 Results: OpenMathInstruct-2’s GPQA accuracy plateaus around 27–26, while MMLU-Pro fluctuates without significant improvement as dataset size increases.The pattern indicates weaker transfer beyond math reasoning.
  • 5.1 Results: WebInstruct’s GPQA performance peaks at 29.02 at 500K, drops to 25.37 at 1.5M, and reaches only 26.12 at 2.8M.This illustrates inconsistent scaling despite increased training data.

6 Eliciting Long Chain-of-Thought

The authors test whether NaturalReasoning contains questions complex enough to benefit from long chain-of-thought responses generated by a stronger reasoning model. Random subsets perform comparably to heavily curated datasets, and larger subsets improve performance consistently.

  • 6 Eliciting Long Chain-of-Thought: DeepSeek-R1 generated responses for 1K randomly sampled NATURALREASONING questions, ranging from 745 to 14.6K tokens with a 4,430-token average.These responses were used to fine-tune Llama-3.3-70B-Instruct and compare against s1K-1.1 and LIMO.
  • 6 Eliciting Long Chain-of-Thought: A random 1K-question subset of NATURALREASONING matches or slightly exceeds performance from s1K-1.1 and LIMO after distilling DeepSeek-R1 responses.The comparison is reported in Table 2 using Llama-3.3-70B-Instruct Pass@1.
  • 6 Eliciting Long Chain-of-Thought: Performance increases monotonically when the random NATURALREASONING subset grows from 1K to 10K and 100K questions across every benchmark.The authors attribute the gains to added questions maintaining high quality.
  • 6 Eliciting Long Chain-of-Thought: Fine-tuning on 100K randomly sampled NATURALREASONING questions brings Llama-3.3-70B-Instruct close to DeepSeek-R1-Distill-Llama-70B.This result is presented as an additional scaling outcome in Table 2.

7 Unsupervised Self-Training

NaturalReasoning supports unsupervised self-training evaluated on GPQA-Diamond and MMLU-Pro using RFT, DPO, and several reward-scoring strategies. Self-reward methods improve over the baseline and can outperform external reward models, especially when low-confidence responses are filtered.

  • 7 Unsupervised Self-Training: The self-training study constructs SelfTrain-15k by retrieving NATURALREASONING questions from GPQA seeds and evaluates on held-out GPQA-Diamond questions.The setup is designed to avoid using test questions for data selection.
  • 7 Unsupervised Self-Training: Self-consistency selects responses by final-answer frequency, whereas self-scoring rewards the model’s judged validity using the log-probability difference between “yes” and “no”.Both strategies operate on 32 sampled candidate responses per question within RFT or DPO.
  • 7.1 Results: Self-training improves Llama3.1-8B-Instruct beyond its 40.81 average baseline across GPQA-Diamond and MMLU-Pro.Almost all evaluated self-training methods improve performance.
  • 7.1 Results: Self-score-filtered SFT and self-score-filtered DPO achieve 35.02 on GPQA-Diamond, while filtered DPO reaches the highest overall score of 43.67.These self-reward methods are comparable to or better than the external reward-model results.
  • 7.1 Results: Self-score filtering improves self-scoring results in both RFT and DPO, reaching 42.54 AVG versus 42.35 AVG for RFT and 43.67 AVG versus 43.22 AVG for DPO.The filtering removes responses or preference pairs whose highest-ranked or preferred response has a self-score below zero.

8 Related Work

Related work includes synthetic reasoning datasets bootstrapped from existing benchmarks, harvested from online question-answer sources, or built from pretraining corpora. These approaches differ in domain coverage, quality curation, and reliance on human annotation or external reward models.

  • Synthetic Reasoning Data: STaR and MetaMath bootstrap new reasoning data from existing datasets, while OpenMathInstruct-2 and related methods synthesize math questions and solutions.These approaches rely on high-quality source data and often remain tied to mathematical domains.
  • Synthetic Reasoning Data: OpenMathInstruct-2, Xwin-Math, and related datasets harvest Mathematics Stack Exchange answers but apply limited quality curation and remain math-specific.The resulting coverage is narrower than multidomain pretraining-corpus approaches.
  • Synthetic Reasoning Data: WebInstruct spans multiple domains by harvesting question-answer pairs from pretraining corpora but depends on carefully crafted rule-based filters.This distinguishes its construction from approaches using LLMs throughout generation and curation.
  • Unsupervised Self-training: Prior unsupervised self-training methods typically depend on human-annotated final answers or external reward models.Manual verification is especially resource-intensive for complex, multi-step reasoning, and reward-model training often requires human evaluation.

9 Conclusion

NATURALREASONING is a 2.8-million-question dataset designed to enhance LLM reasoning across diverse domains. Experiments show consistent benchmark improvements with scaling and support self-training using external or self-generated rewards.

  • NATURALREASONING contains 2.8 million challenging questions spanning mathematics, physics, computer science, economics, social sciences, and other domains.
  • Scaling NATURALREASONING data in distillation experiments consistently improves performance on reasoning benchmarks.
  • NATURALREASONING supports unsupervised self-training with external reward models or self-rewarding.

Limitation & Impact Statement

The paper reports broad dataset coverage, strong scaling behavior, and promising reinforcement-learning results, while identifying limited RL exploration and potential model biases as important boundaries. Supporting analyses examine clustering, question quality, answer formats, and implementation details.

  • Limitations: NATURALREASONING’s preliminary online RL experiments show promising gains with limited training, but broader RL exploration remains future work.The study uses General Verifier rewards and reports only preliminary results.
  • Impact boundary: Models trained on this data may exhibit unexamined undesirable behavior or pre-existing biases, motivating comprehensive evaluation.
  • Clustering analysis: NATURALREASONING and WebInstruct are compared through topic clustering, with the clustering procedure described in Section 3.3.
  • Question formats: Example questions are organized by single-word, short, and long reference-answer formats.
  • Evaluation and implementation: Question quality is evaluated with expert human ratings, while generation uses specified decoding temperatures and response sampling settings.Two expert annotators independently rate sampled questions; generation settings vary by pipeline stage.
  • Scaling experiments: Scaling results for Qwen2.5-7B average performance across MATH, GPQA, and MMLU-Pro at different training-data sizes.

F.1 Data Filtering In Knowledge Distillation

The paper filters distillation data by verifying generated responses against reference answers and evaluates related uses in knowledge distillation, cross-domain transfer, and verifiable-reward reinforcement learning. Filtering improves benchmark scores despite reducing dataset size, while RL results remain preliminary.

  • F.1 Data Filtering In Knowledge Distillation: Reference-answer filtering retains responses whose final judgments indicate agreement with the reference answer.Llama3.3-70B-Instruct performs the verification using a chain-of-thought judging prompt.
  • F.1 Data Filtering In Knowledge Distillation: 7,646 filtered examples outperform 12,349 unfiltered examples on GPQA-Diamond and MMLU-Pro after reference-answer verification.Scores are 32.15 vs. 31.82 on GPQA-Diamond and 50.06 vs. 49.92 on MMLU-Pro.
  • F.2 Reinforcement Learning With Verifiable Rewards: Preliminary GRPO training with General Verifier rewards produces noticeable gains after only 50 optimization steps.The experiment uses short-reference-answer questions and Llama3.1-8B-Instruct.
  • Knowledge distillation: NATURALREASONING-trained students outperform OpenThoughts-trained students on three of four benchmarks under matched 100K-sample distillation.DeepSeek-R1 is the teacher and Llama-3.1-8B-Instruct is the student.
  • Cross-domain evaluation: Training on NATURALREASONING improves performance across diverse non-reasoning subjects in SuperGPQA evaluation.
  • I Prompts: The paper provides prompts for document reasoning annotation, question generation, quality scoring, answer matching, and self-scoring.
Loading 2502.13124v4…