PaperlayerView exact revision on arXiv

Source-linked AI summary

SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domain

Dakuan Lu, Xiaoyu Tan, Rui Xu, Tianchu Yao, Chao Qu, Wei Chu, Yinghui Xu, Yuan Qi

arXiv:2501.15587v2cs.CLcs.AIcs.IR

TL;DR

Higher-education science lacks the curated resources available for mathematics. The paper introduces SCP-116K and an automated extraction pipeline; its hybrid distillation approach reports 58.59% accuracy on GPQA-diamond, 11.12 percentage points above the base model.

  • Problem

    High-quality scientific problem-solution pairs for undergraduate-to-doctoral content are scarce compared with curated mathematics resources.

    Source: §1 · p. 1

  • Method

    The authors build SCP-116K from heterogeneous sources using an automated pipeline for extraction, matching, and quality filtering.

    Source: §1 · pp. 1–2

  • Results

    58.59% accuracy on GPQA-diamond, the hybrid approach improved 11.12 percentage points over base Qwen2.5-32B-Instruct.

    Source: §4.3 · p. 5

  • Takeaways & Limitations

    SCP-116K and its extraction pipeline offer a scalable resource and approach for expanding scientific reasoning data across disciplines.

    Source: §5 · pp. 5–6

Abstract

from arXiv · show

Recent breakthroughs in large language models (LLMs) exemplified by the impressive mathematical and scientific reasoning capabilities of the o1 model have spotlighted the critical importance of high-quality training data in advancing LLM performance across STEM disciplines. While the mathematics community has benefited from a growing body of curated datasets, the scientific domain at the higher education level has long suffered from a scarcity of comparable resources. To address this gap, we present SCP-116K, a new large-scale dataset of 116,756 high-quality problem-solution pairs, automatically extracted from heterogeneous sources using a streamlined and highly generalizable pipeline. Our approach involves stringent filtering to ensure the scientific rigor and educational level of the extracted materials, while maintaining adaptability for future expansions or domain transfers. By openly releasing both the dataset and the extraction pipeline, we seek to foster research on scientific reasoning, enable comprehensive performance evaluations of new LLMs, and lower the barrier to replicating the successes of advanced models like o1 in the broader science community. We believe SCP-116K will serve as a critical resource, catalyzing progress in high-level scientific reasoning tasks and promoting further innovations in LLM development. The dataset and code are publicly available at https://github.com/AQA6666/SCP-116K-open.

1 Introduction

Despite progress in mathematical reasoning and scientific LLM capabilities, comparable curated resources for higher-education science remain scarce. SCP-116K addresses this gap with a large problem-solution dataset and an automated extraction pipeline.

1 Introduction

  • Higher-education science lacks resources comparable to curated mathematics datasets, and high-quality scientific problem-solution pairs remain scarce.The paper describes this scarcity as a bottleneck to advancing scientific reasoning capabilities.Source: §1 · p. 1
  • SCP-116K provides 116,756 problem-solution pairs spanning scientific disciplines and undergraduate-to-doctoral content.The dataset is intended for training and evaluating LLMs in scientific reasoning.Source: §1 · pp. 1–2
  • An automated pipeline extracts scientific content from heterogeneous sources, using image-based rendering, multimodal parsing, problem-solution matching, and quality controls.The matching combines numerical identifiers with semantic similarity; filtering aims to preserve academic rigor and practical utility.Source: §1 · pp. 1–2
  • The work also benchmarks current LLMs on the dataset and openly releases the dataset and extraction pipeline to support further STEM reasoning research.The authors aim to lower barriers to developing specialized STEM models.Source: §1 · p. 2

2 Related Work

The related work contrasts existing scientific datasets and benchmarks with the paper’s higher-education training resource, and situates it among recent reasoning models and methods.

Scientific Question-Answering Datasets

  • Existing resources such as CAMEL, ScienceQA, and the Kaggle LLM Science Exam either use synthetic data or target lower educational levels, unlike SCP-116K’s authentic advanced materials.SCP-116K is described as containing 116,756 extracted problem-solution pairs.Source: §2 · p. 2

Benchmarking Standards

  • GPQA evaluates graduate-level scientific reasoning, while SCP-116K is positioned as a complementary training resource for challenging evaluations.GPQA contains 448 expert-crafted questions across biology, physics, and chemistry.Source: §2 · p. 2

Advanced Scientific Reasoning Models

  • o1, QwQ-32B-preview, and STILL-2 exemplify recent advances in scientific reasoning, with STILL-2 reporting 56.1% accuracy on GPQA using 3,900 distillation examples.The paper connects these developments to the importance of high-quality training data.Source: §2 · pp. 2–3

3 Methodology

The methodology presents a six-stage pipeline for retrieving, processing, segmenting, extracting, filtering, and matching scientific problem-solution content; it also collects model-generated solutions for distillation and benchmarking.

3.1 Document Retrieval and Filtering

  • Keyword search over 6.69 million academic documents yielded 4,270 candidates, then GPT-4o filtering retained 467 university-to-doctoral physics, chemistry, or biology sources.Documents were initially targeted by “problem” or “question” in their titles.Source: §3.1 · p. 3

3.2 Unified Preprocessing

  • Unified preprocessing renders diverse formats as images, then uses GPT-4o to convert them into markdown with LaTeX scientific notation while preserving semantic structure.The approach is intended to handle formulas and chemical equations consistently across formats.Source: §3.2 · p. 3

3.3 Content Segmentation

  • GPT-4o segmentation identifies natural boundaries such as chapters, sections, problems, and solutions while respecting a maximum token constraint.The stated aim is to preserve logical units while supporting downstream processing.Source: §3.3 · p. 3

3.4 Structured Extraction

  • Generative extraction identifies problems, solutions, and numerical identifiers, producing an initial pool of about 190,000 problems and 80,000 solutions.Extracted content retains contextual relationships for subsequent refinement.Source: §3.4 · p. 3

3.5 Quality Filtering

  • Two filtering stages remove incomplete extractions and entries dependent on external elements, yielding 116,756 problems and 70,000 solutions.The second stage excludes references to equations, figures, or other problems absent from the extracted content.Source: §3.5 · p. 3

3.6 Problem-Solution Matching

  • Numerical matching pairs shared problem numbers, while Stella semantic similarity addresses inconsistent numbering; GPT-4o verifies ranked candidates, yielding 43,000 pairs.The matching process generates up to four candidate solutions per problem.Source: §3.6 · pp. 3–4

3.7 Model Response Collection

  • The authors collect o1-mini and QwQ-32B-preview solutions for every problem, with GPT-4o-mini validating correspondence to extracted ground-truth solutions.The collection is intended to support knowledge distillation and provide baseline performance metrics.Source: §3.7 · p. 4

4 Experiments

Experiments assess model performance on SCP-116K and test distillation on GPQA-diamond. SCP-116K-enhanced approaches improve over the base model, with the hybrid approaching o1-mini’s performance.

4.3 Results and Analysis

  • 58.59% (hybrid) and 58.08% (SCP-116K STILL-2) exceeded base Qwen2.5-32B-Instruct by 11.12 and 10.61 percentage points, respectively, on GPQA-diamond.The reported comparison is among the distillation approaches and the base model.Source: Table 1 · p. 5, §4.3 · p. 5

4.1 Baseline Performance on SCP-116K

  • On SCP-116K, o1-mini achieved 58.40% accuracy and QwQ-32B-preview achieved 55.79%.These results establish baseline performance on the dataset.Source: §4.1 · pp. 4–5

4.2 Knowledge Distillation Experiments

  • Distillation tests included transferring o1-mini responses, original STILL-2 data, replacing its CAMEL science QA with 1,000 SCP-116K examples, and a hybrid data mix.The hybrid combined 1,000 SCP-116K pairs selected for semantic similarity to GPQA-diamond questions with STILL-2 data.Source: §4.2 · p. 5

4.3 Results and Analysis

  • The hybrid approach approached o1-mini’s 60.61% accuracy on GPQA-diamond despite using a significantly smaller model.The authors suggest the dataset and distillation methodology may help bridge the performance gap with larger models.Source: §4.3 · p. 5

5 Conclusion

SCP-116K pairs a large scientific problem-solution dataset with a generalizable extraction pipeline. The authors report improvements from hybrid distillation and plan to expand the dataset and develop models for scientific reasoning.

5 Conclusion

  • Hybrid distillation achieves notable improvements in model performance on challenging scientific reasoning tasks.The conclusion also presents the dataset’s utility as a foundation for future scientific-reasoning research.Source: §5 · p. 5
  • SCP-116K contains 116,756 high-quality scientific problem-solution pairs alongside a generalizable automated extraction pipeline.The authors present these resources as addressing a gap in scientific reasoning.Source: §5 · p. 5
  • The authors plan to use the scalable pipeline to expand the dataset’s scale and disciplinary coverage.They also identify more sophisticated models optimized for scientific reasoning as a future direction.Source: §5 · p. 6
  • The authors see continued model development and training on an expanded dataset as potential routes to further scientific-reasoning improvements.They frame this as a future possibility, not an established result.Source: §5 · p. 6

A.1 GPT-4o Prompts

The appendix provides GPT-4o prompts used in the pipeline for transparency and reproducibility, covering document filtering, preprocessing, segmentation, structured extraction, and pair verification.

A.1 GPT-4o Prompts

  • The appendix presents GPT-4o prompts used throughout the pipeline for transparency and reproducibility.The prompts include document filtering, preprocessing, segmentation, extraction, and verification tasks.Source: §A.1 · p. 6, §A.1.1 · p. 6, §A.1.2 · p. 6

A.1.1 Document Filtering

  • Document filtering asks whether a book is a physics, chemistry, or biology textbook or problem book for undergraduate-to-doctoral students.The prompt supplies book title and author metadata and requests a Yes/No determination.Source: §A.1.1 · p. 6

A.1.2 Unified Preprocessing

  • Unified preprocessing prompts GPT-4o to convert document images into Markdown in reading order, using LaTeX for formulas and describing important illustrations.It instructs the model to ignore headers, footers, and non-essential visuals, and to return “empty” for blank pages.Source: §A.1.2 · p. 6

A.1.3 Content Segmentation

  • Content segmentation identifies chapter, section, subsection, and problem beginnings by line number while ignoring headers, footers, sub-question markers, and solution indicators.The prompt asks for the identified line numbers in JSON format.Source: §A.1.3 · pp. 6–7

A.1.4 Structured Extraction

  • The extraction prompt specifies JSON fields for problem or solution numbers and their full content, with an empty list when neither is present.The prompt also asks for the extracted results after reasoning.Source: §A.1.4 · p. 7
  • Structured extraction asks GPT-4o to identify complete problems or solutions, preserve multi-part problem statements, and avoid changing or omitting content.The extracted results are requested as a list of JSON objects.Source: §A.1.4 · p. 7

A.1.5 Problem-Solution Verification

  • Problem-solution verification checks that each item is genuinely a problem or solution, that the pair matches, and that the solution is correct and complete.The prompt permits a final answer without workings, provided it includes a final answer and can be independently understood and verified.Source: §A.1.5 · p. 8

Related papers

Loading 2501.15587v2…