PaperlayerView exact revision on arXiv

Source-linked AI summary

MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning

Run-Ze Fan, Zengzhi Wang, Pengfei Liu

arXiv:2507.16812v2cs.CLcs.AIcs.LG

TL;DR

Open-source scientific reasoning lacks large-scale datasets with high-quality, verifiable answers, despite its relevance to AI scientists and researchers. The paper constructs TextbookReasoning and MegaScience and evaluates scientific fine-tuning. Its experiments report strong performance and efficiency, including average-performance gains over official instruct models for MegaScience-trained base models.

  • Problem

    Open-source scientific reasoning remains underdeveloped, with limited large-scale, high-quality datasets and concerns about benchmark reliability, decontamination, and reference-answer quality.

    Source: §1 · pp. 2–3

  • Method

    The paper builds TextbookReasoning from university textbooks and forms MegaScience through dataset-specific selection studies, with step-by-step solutions annotated for datasets other than TextbookReasoning.

    Source: §1 · p. 3

  • Results

    MegaScience-trained Llama3.1, Qwen2.5, and Qwen3 base models outperform their official instruct models in average performance, and the datasets support efficient training and inference with state-of-the-art scientific-domain performance.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors present their datasets and empirically grounded curation strategies as resources for scientific-reasoning research and domain-specific dataset assembly.

    Source: §8 · p. 14

    Less capable models, including Llama-series and smaller Qwen2.5 models, can struggle with MegaScience without supplemental foundational data or lower-difficulty warmup training.

    Source: §6 · p. 13

Abstract

from arXiv · show

Scientific reasoning is critical for developing AI scientists and supporting human researchers in advancing the frontiers of natural science discovery. However, the open-source community has primarily focused on mathematics and coding while neglecting the scientific domain, largely due to the absence of open, large-scale, high-quality, verifiable scientific reasoning datasets. To bridge this gap, we first present TextbookReasoning, an open dataset featuring truthful reference answers extracted from 12k university-level scientific textbooks, comprising 650k reasoning questions spanning 7 scientific disciplines. We further introduce MegaScience, a large-scale mixture of high-quality open-source datasets totaling 1.25 million instances, developed through systematic ablation studies that evaluate various data selection methodologies to identify the optimal subset for each publicly available scientific dataset. Meanwhile, we build a comprehensive evaluation system covering diverse subjects and question types across 15 benchmarks, incorporating comprehensive answer extraction strategies to ensure accurate evaluation metrics. Our experiments demonstrate that our datasets achieve superior performance and training efficiency with more concise response lengths compared to existing open-source scientific datasets. Furthermore, we train Llama3.1, Qwen2.5, and Qwen3 series base models on MegaScience, which significantly outperform the corresponding official instruct models in average performance. In addition, MegaScience exhibits greater effectiveness for larger and stronger models, suggesting a scaling benefit for scientific tuning. We release our data curation pipeline, evaluation system, datasets, and seven trained models to the community to advance scientific reasoning research.

1 INTRODUCTION

Open-source scientific reasoning is underdeveloped despite its importance, amid gaps in benchmark reliability, decontamination, reference-answer quality, and training-data quality. The paper introduces textbook-derived questions and a curated dataset mixture, reporting strong performance and efficiency for scientific fine-tuning.

1 INTRODUCTION

  • Scientific reasoning remains underdeveloped relative to mathematics and coding, while multiple-choice benchmarks can inflate performance without reflecting computational reasoning.The introduction also identifies fragile overlap-based decontamination and substantial benchmark overlap in existing science post-training datasets.Source: §1 · p. 2
  • Web-scraped or LLM-generated reference answers can be unreliable, while distilled long chain-of-thought data may overthink and burden small-model training and inference.The authors argue these issues hinder principled, efficient, and generalizable knowledge transfer.Source: §1 · pp. 2–3
  • TextbookReasoning provides 650k scientific reasoning questions with truthful reference answers, extracted from nearly 12k university-level textbooks across seven disciplines.Its automated curation pipeline includes textbook digitalization, dual QA extraction, deduplication, refinement, filtering, and LLM-based decontamination.Source: §1 · p. 3
  • MegaScience combines 1.25 million instances from open-source datasets, using ablations to select an approach for each dataset and annotating step-by-step solutions except for TextbookReasoning.The paper also reports shorter responses: 410 tokens for TextbookReasoning and 721 for MegaScience.Source: §1 · p. 3
  • Fine-tuning on the datasets achieves state-of-the-art scientific-domain performance and efficient training and inference; MegaScience-trained Llama3.1, Qwen2.5, and Qwen3 base models outperform official instruct models on average.MegaScience is reported to be more effective for larger and stronger models, suggesting a scaling benefit for scientific instruction tuning.Source: §1 · p. 3
  • The authors release their data curation pipeline, evaluation system, datasets, and trained models to the community for scientific reasoning research.The contribution list specifies seven trained models.Source: §1 · p. 3

2 TEXTBOOKREASONING DATA CURATION

TextbookReasoning is built from university-level scientific textbooks through collection, extraction, deduplication, refinement, filtering, and benchmark decontamination. The resulting dataset contains verified reference answers and questions spanning seven disciplines.

2 TEXTBOOKREASONING DATA CURATION

  • TextbookReasoning comprises 651k reasoning questions from 12.8k university-level scientific textbooks across seven disciplines, with verified reference answers.The authors motivate textbooks as expert-crafted sources with systematic knowledge structures suited to knowledge learning.Source: §2 · p. 3

2.1 TEXTBOOKS COLLECTION AND DIGITIZATION

  • The collection process filters web-crawled books by public-access metadata and academic level, then digitizes the retained 12.8k books across seven disciplines.The corpus includes 6,057 computer science and artificial intelligence books, the largest listed subject group, and 158 economics books, the smallest.Source: §2.1 · p. 4

2.2 DUAL Q-A PAIRS EXTRACTION

  • Dual Q-A extraction applies high-standard multi-step-reasoning criteria and lower-standard completeness criteria to textbook chunks, producing 945k extracted pairs.More than 60% of mathematics chunks contain questions, compared with fewer than 10% in other disciplines; extraction preserves original information relative to question synthesis [Li et al., 2023a].Source: §2.2 · p. 4

2.3 QUESTION DEDUPLICATION

  • Word-level locality-sensitive min-hashing removes questions with similarity above 0.6 to reduce redundant variants of the same reasoning task.Source: §2.3 · p. 5

2.4 Q-A PAIR REFINEMENT

  • Question-answer pairs are refined with source documents to restore missing context and reasoning, then defective pairs are filtered out.DeepSeek-V3 refines pairs and adds explanations or reformats answers; Llama3.3-70B-Instruct identifies pairs with missing reasoning and filters pairs with unresolved defects.Source: §2.4 · p. 5

2.5 LLM-BASED QUESTION DECONTAMINATION

  • LLM-based decontamination retrieves each question’s five most similar benchmark examples and removes it if Llama3.3-70B-Instruct judges any pair to be a paraphrase.The process targets overlap across widely used scientific-reasoning benchmarks because n-gram methods can miss paraphrased or translated questions.Source: §2.5 · p. 5

3 MEGASCIENCE DATA CURATION

MegaScience combines selected open-source scientific datasets through data selection and solution-annotation procedures to form a 1.25-million-instance mixture.

3 MEGASCIENCE DATA CURATION

  • The curation recipe combines dataset collection, selection, and solution annotation into the 1.25-million-instance MegaScience mixture.Figure 5 presents the overall recipe.Source: §3 · pp. 5–6, Figure 5 · p. 6

3.1 SOURCING FROM PUBLIC DATASETS

  • The source datasets are NaturalReasoning, Nemotron-Science, and TextbookReasoning; SCP-116K is excluded for inferior scientific-reasoning performance.The selection process draws from multiple public datasets and explores data-selection and solution-annotation techniques.Source: §3 · p. 5, §3.1 · p. 6

3.2 QUESTION DEDUPLICATION AND DECONTAMINATION

  • NaturalReasoning and Nemotron-Science undergo question deduplication and LLM-based question decontamination.Source: §3.2 · p. 6

3.3 DATA SELECTION

  • The three selection methods are response-length selection, difficulty selection, and random selection, compared using equal-sized subsets after difficulty selection sets the instance count.Difficulty scores average 16 Qwen2.5-7B-Instruct responses judged against reference answers; samples scoring above 9 or below 1 are filtered out.Source: §3.3 · p. 6
  • Random selection works best for NaturalReasoning, difficulty selection for Nemotron-Science, and no subset method surpasses using all TextbookReasoning instances.Source: §3.3 · p. 6

3.4 SOLUTION ANNOTATION

  • TextbookReasoning retains refined solutions; DeepSeek-V3 annotates step-by-step solutions for NaturalReasoning and Nemotron-Science, with responses over 4,096 tokens removed.The length filter removes approximately 8,000 instances; the authors report that overly long outputs often contain repetition or redundancy.Source: §3.4 · p. 7

4 MEGASCIENCE EVALUATION FRAMEWORK

The evaluation framework supports reproducible, fair assessment across scientific subjects and question formats, with dedicated answer-extraction strategies. Its components include an open toolkit, a benchmark suite, extraction methods, and recommended settings.

4 MEGASCIENCE EVALUATION FRAMEWORK

  • The framework targets reproducibility, broad subject and question-type coverage, fair model comparison, and accurate answer extraction.It comprises an open evaluation toolkit, a scientific-reasoning suite, answer-extraction strategies, and recommended settings.Source: §4 · p. 7

4.1 LANGUAGE MODEL OPEN SCIENCE EVALUATION

  • The open evaluation system supports conversation and base models, configurable benchmarks and prompts, parallel multi-model runs, and instance-level outputs.Source: §4.1 · pp. 7–8

4.2 MEGASCIENCE EVALUATION SUITE

  • The benchmark suite covers general scientific reasoning, specialized subjects, and mathematical reasoning across diverse question formats.Its stated categories include general scientific reasoning, specific scientific reasoning, and mathematics.Source: §4.2 · p. 8

4.3 ANSWER EXTRACTION STRATEGY

  • Answer extraction first detects final-answer indicator phrases, then extracts content across mathematical and textual formats; for multiple-choice items, option content can recover a missing label.Source: §4.3 · pp. 8–9

5 SUPERVISED FINETUNING

Across the reported experiments, MegaScience performs strongly against scientific-dataset and instruction-tuned baselines, with results also examining model scale, curation components, efficiency, and question difficulty.

5.1 SETUP

  • LLM-based decontamination identifies 19K leaked instances in SCP-116K, 66K in NaturalReasoning, and 164K in Nemotron-Science.The paper applies this method to baseline datasets because n-gram decontamination can be circumvented by minor textual variations.Source: §5.1 · p. 9

5.2 MAIN EXPERIMENTS

  • TextbookReasoning beats Nemotron-Science by 20.62% on SciBench and 5.23% on OlympicArena, while remaining competitive on multiple-choice evaluations.Nemotron-Science leads on some multiple-choice benchmarks, a pattern the authors attribute to its all-MCQ training data.Source: §5.2 · p. 9
  • MegaScience ranks best on 7 of 14 benchmarks and second-best on 3 more, with a 2.21% overall average improvement over Qwen2.5-7B-Instruct.It scores 48.75% on SciBench and 40.23% on OlympicArena, the highest results on those challenging computational tasks.Source: §5.2 · pp. 9–10, Table 6 · p. 10

5.3 PUSHING THE FRONTIER IN SCIENCE DOMAIN WITH MEGASCIENCE

  • Qwen2.5-7B, all Qwen3 models, and Llama3.1-8B trained on MegaScience outperform their corresponding official instruction-tuned counterparts in average performance.The comparison uses Qwen3 non-thinking mode; the reported scaling pattern is not uniform across every model size.Source: §5.3 · p. 10, Table 7 · p. 11

5.4 ABLATION STUDY

  • Removing refinement drops TextbookReasoning’s overall average from 58.33% to 13.15%, while DeepSeek-V3 refinement outperforms Llama3.3-70B-Instruct across evaluation categories.Removing supplementary CoT lowers overall performance to 57.33%; removing decontamination raises it to 58.57%.Source: §5.4 · p. 11, Table 8 · p. 12, Table 9 · p. 12

5.5 ANALYSIS

  • Models trained on MegaScience show strong inference performance despite shorter training responses, and Qwen3-8B-MegaScience averages 1,080 response tokens versus 1,345 for Qwen2.5-7B-MegaScience.Figure 6 presents the performance–response-length trade-off; the authors report a negative correlation between training-response length and downstream performance across datasets.Source: Figure 6 · p. 12, §5.5 · pp. 12–13
  • Refinement slightly outperforms difficulty-aware distillation overall, while distillation improves mathematical reasoning; the authors suggest reference documents may reduce refinement’s hallucination risk.Source: Table 11 · p. 13, §5.5 · p. 13
  • TextbookReasoning has shorter average responses than NaturalReasoning—898.5 versus 1,124.7 tokens—but a broader, more varied response-length distribution.The authors interpret this distribution as indicating more diverse question complexity; Figure 7 shows the response-length distributions.Source: §5.5 · p. 13

6 DISCUSSION

The discussion identifies two limits on generalizing MegaScience results: lower-capacity models may struggle with complex data, and proxy-model choices can shape data development.

6 DISCUSSION

  • Less capable models may struggle with complex MegaScience reasoning data without supplemental foundational data or lower-difficulty warmup training.The reported difficulties include abbreviated responses and elevated repetition rates.Source: §6 · p. 13
  • Using Qwen2.5-7B as the proxy ties optimized data mixtures to that model’s capabilities, and lower-capacity models may not reproduce its gains.The authors identify proxy selection as a source of bias in data development and call for attention to capability alignment and broader generalizability.Source: §6 · pp. 13–14

7 RELATED WORKS

Prior scientific-reasoning datasets rely mainly on web-scraped or LLM-synthesized data, with limitations in answer quality, efficiency, decontamination, and evaluation coverage. This work uses textbooks, selected short-CoT annotations, semantic decontamination, and a broad evaluation framework as alternatives.

7 RELATED WORKS

  • Scientific reasoning has become a focus as research shifts from knowledge-based abilities toward reasoning capabilities, alongside advances in test-time scaling.Prior dataset approaches include web-question scraping and LLM synthesis from seed data.Source: §7 · p. 14
  • Scientific-reasoning datasets mainly use web-scraped questions or LLM-synthesized questions and solutions, but face limitations in answer reliability, reasoning efficiency, decontamination, and benchmark coverage.Existing evaluations often emphasize multiple-choice questions, which do not adequately reflect computational skills.Source: §7 · p. 14
  • The paper uses textbooks, data selection, short-CoT annotation, and LLM-based semantic benchmark decontamination, alongside a 15-benchmark evaluation framework spanning several question types.The framework includes multiple-choice, computational, true/false, and open-ended problem-solving tasks.Source: §7 · p. 14

8 CONCLUSION AND FUTURE WORK

The conclusion presents TextbookReasoning and MegaScience as open scientific-reasoning resources, with supervised fine-tuning results outperforming official instruct counterparts. Future work includes reinforcement learning and testing whether concise long-CoT formats can improve performance at comparable response lengths; the appendices point to prompts for extraction, refinement, decontamination, and difficulty selection.

8 CONCLUSION AND FUTURE WORK

  • Supervised fine-tuning on Qwen-2.5, Qwen-3, and Llama3 series models yielded models that significantly outperform their official instruct counterparts.MegaScience contains 1.25 million instances, and TextbookReasoning comprises 650k questions with step-by-step solutions from authoritative textbooks.Source: §8 · p. 14
  • The authors have not explored reinforcement learning for scientific reasoning, and propose testing RL on SFT models for long-CoT reasoning; computing constraints also left long-CoT compression uninvestigated.The proposed compression question is whether concise reasoning could improve performance at response lengths comparable to MegaScience.Source: §8 · pp. 14–15

A.1 PROMPTS FOR Q-A PAIRS EXTRACTION

  • The appendix points to prompts used to extract question-answer pairs across seven scientific domains.The domains are biology, chemistry, computer science, economics, mathematics, medicine, and physics.Source: §A.1 · p. 20

A.2 PROMPTS FOR Q-A PAIRS REFINEMENT

  • The appendix identifies prompts for question-answer refinement, chain-of-thought screening, and defective-pair filtering.Source: §A.2 · p. 20

A.3 PROMPTS FOR QUESTION DECONTAMINATION

  • The appendix points to the prompt used for LLM-based question decontamination.Source: §A.3 · p. 20

A.4 PROMPTS FOR DIFFICULTY SELECTION

  • The appendix points to prompts for annotating reference answers and evaluating student answers.Source: §A.4 · p. 20

B ANSWER EXTRACTION RULES AND PATTERNS

The section points readers to a table of answer-extraction patterns.

B ANSWER EXTRACTION RULES AND PATTERNS

C TRAINING DETAILS

The section points readers to a table containing training details.

C TRAINING DETAILS

D DETAILED RESULTS

The supplied passages identify the Qwen3 comparison reported in Table 13 and point to detailed MegaScience results in Tables 12 and 13. They also list subject-specific extraction prompts and prompts for later Q-A processing.

D DETAILED RESULTS

  • Detailed MegaScience results are reported in Tables 12 and 13.The supplied passages do not include the tables’ result values.Source: D DETAILED RESULTS · p. 20
  • Table 13 compares Qwen3 models trained on MegaScience with official instruction-tuned models, using non-thinking mode for fair comparison because of short CoT.Bold indicates the best result.Source: Table 13 · p. 21
  • High- and low-standard Q-A extraction prompts are listed for biology, chemistry, and computer science and artificial intelligence.The prompts are identified in Figures 8–13.Source: Figure 8 · p. 21, Figure 9 · p. 22, Figure 10 · p. 23
  • High- and low-standard Q-A extraction prompts are listed for economics, mathematics, and physics.The prompts are identified in Figures 15–20.Source: Figure 14 · p. 27, Figure 15 · p. 28, Figure 16 · p. 29
  • The listed processing prompts cover Q-A refinement, reasoning checks, defective-pair filtering, decontamination, reference-answer annotation, and response evaluation.These prompts are identified in Figures 22–27.Source: Figure 22 · p. 34, Figure 23 · p. 35, Figure 24 · p. 36

Related papers

Loading 2507.16812v2…