PaperlayerView exact revision on arXiv

Source-linked AI summary

SPARK: Skeleton-Guided Reasoning Synthesis from Large-Scale Scientific Literature

Yu Li, Wei Li, Xin Gao, Mengyuan Sun, Xiaoyang Wang, Qizhi Pei, Lijun Wu

arXiv:2608.30214v1cs.AI

TL;DR

Scientific reasoning datasets often emphasize factual recall and routine computation while underrepresenting mechanism, evidence, and hypothesis evaluation. SPARK reconstructs paper-level claim-evidence-derivation structures into reasoning skeletons and uses them to generate and verify diverse scientific tasks. Spark-234K achieves strong benchmark performance with substantially fewer examples, while its evaluation relies mainly on automated consistency checks and remains limited to supervised fine-tuning.

  • Problem

    Existing scientific reasoning data underrepresents evidence-based, mechanism-driven, and hypothesis-evaluation reasoning found in scientific discovery.

    Source: §1 · pp. 1–2, §6.1 · p. 8

  • Method

    SPARK distills papers into claim-centered reasoning skeletons, generates tasks from four scientific perspectives, and verifies their self-containment and consistency.

    Source: §1 · p. 2

  • Results

    Spark-234K outperforms larger baseline datasets across base-model families and parameter scales, including average margins of +3.10, +4.47, and +3.43 points.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Spark-234K achieves strong downstream scientific reasoning performance with substantially fewer training examples, highlighting dataset construction quality alongside scale.

    Source: §5.2 · p. 6, §7 · p. 9

    Experiments are limited to supervised fine-tuning, and large-scale quality control relies primarily on automated consistency checks with human validation on 300 samples.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Scientific reasoning remains challenging for open-source models, largely due to the lack of high-quality scientific reasoning data. Existing datasets are often dominated by factual recall or formulaic problem solving, with limited emphasis on mechanism understanding, evidence-grounded reasoning, and hypothesis evaluation. To address this, we introduce SPARK (Scientific Paper Abstracted Reasoning sKeleton), a paper-oriented synthesis framework built on Sci-Base, a large-scale corpus of research papers spanning 10 scientific disciplines. Instead of directly converting papers into question-answer pairs, SPARK treats the claim-evidence-derivation structure of a paper as the fundamental unit of reasoning synthesis. Specifically, SPARK (1) distills each paper into a compact reasoning skeleton capturing its central claims and supporting evidence, enabling self-contained question generation, and (2) synthesizes reasoning tasks from four scientific perspectives: mechanistic reasoning, hypothesis falsification, quantitative derivation, and boundary calibration. A final consistency verification stage further removes unsupported or contradictory outputs. Using this framework, we construct Spark-234K, a scientific reasoning dataset with substantially higher difficulty and diversity than existing resources. Experiments show that Spark-234K consistently outperforms existing scientific reasoning datasets while achieving stronger performance with significantly fewer training samples.

1 Introduction

Scientific reasoning data often underrepresents evidence-based and mechanism-driven reasoning. SPARK addresses this by synthesizing tasks from paper-level claim-evidence-derivation structures, producing a compact dataset that performs strongly with fewer examples.

  • Existing scientific corpora mainly emphasize factual recall, standard derivations, and routine calculations rather than evidence-based and mechanism-driven reasoning.Source: §1 · p. 1
  • SPARK treats a paper’s claim-evidence-derivation chain as the fundamental unit for scientific reasoning data synthesis.Source: §1 · p. 2
  • Each paper is distilled into a compact reasoning skeleton containing claims, evidence, assumptions, quantitative relations, and boundary conditions.Source: §1 · p. 2
  • SPARK generates tasks through mechanistic reasoning, hypothesis falsification, quantitative derivation, and boundary calibration, then verifies consistency.Source: §1 · p. 2
  • 234K synthesized instances from 370K frontier papers outperform larger baseline datasets, with average margins of +3.10, +4.47, and +3.43 points across evaluated settings.Source: §1 · p. 2

2 Paper Collection and Preparation

SPARK draws from SCI-BASE, a structured, multi-disciplinary scientific literature corpus. The collection process filters papers for parsing quality, recency, language, argument clarity, and concrete observables while balancing disciplinary coverage.

  • SCI-BASE contains approximately 3.36M papers across ten disciplines, with structured metadata and parsed paragraphs, tables, captions, and equations.Source: §2 · p. 2
  • The pipeline restricts papers to January 2024–March 2026 and removes non-English, incompletely parsed, or review articles lacking a clear claim-evidence trajectory.Source: §2 · p. 2
  • Selection prioritizes papers with concrete observables, including equations, statistical results, tables, and experimental comparisons.Source: §2 · p. 2
  • Balanced sampling mitigates the source corpus’s heavy skew toward medicine and life sciences.Source: §2 · p. 2

3 Dataset Construction

SPARK reconstructs each paper as a claim-centered reasoning skeleton, generates questions from complementary scientific perspectives, and applies layered quality checks. The resulting pipeline preserves evidence trajectories while enforcing self-containment and consistency.

Skeleton Extraction

  • SPARK treats a paper as an argument, identifying its core conclusion and reconstructing the supporting evidence trajectory as a compact skeleton.Source: §3 · p. 2, §3.1 · p. 3
  • Skeleton extraction avoids context diffusion from full-paper prompting and fragmentation from section-level or chunk-based representations.Source: §3.1 · p. 2
  • The normalized input preserves equations, table text, captions, and numerical results while removing formatting artifacts and non-argument sections.Source: §3.1 · p. 3

Multi-Perspective Question Generation

  • Questions probe mechanisms, competing hypotheses, quantitative relations, and validity boundaries instead of only surface facts or plug-and-chug computation.Source: §3.2 · pp. 3–4
  • SPARK generates at most one question per evidence-supported perspective and skips perspectives unsupported by the extracted skeleton.Source: §3.2 · p. 4

Skeleton-Grounded Quality Check

  • Rule-based and model-based filters remove malformed, non-self-contained, inconsistent, incoherent, or weakly supported question-answer pairs.Source: §3.2 · p. 4, §3.3 · p. 4, §3.4 · p. 4
  • Skeleton-grounded evaluation keeps quality control computationally lightweight by avoiding massive majority voting and iterative rewriting.Source: §3.4 · p. 4
  • Spark-234K applies semantic deduplication within disciplines and 13-gram matching against evaluation sets to reduce duplication and benchmark leakage.Source: §3.5 · p. 4

4 Dataset Statistics and Analysis

Spark-234K is a 234K-instance scientific reasoning dataset with broad disciplinary and perspective coverage, high diversity, and predominantly advanced reasoning tasks. Its evaluations indicate strong correctness and self-containment, including after quality control.

  • Spark-234K contains 234K synthesized scientific reasoning instances derived from 370K filtered seed papers.Source: §4 · p. 4

Disciplinary coverage

  • Physics, Mathematics & Computer Science, and Chemistry are the largest disciplinary segments, while ten disciplines provide broad thematic coverage.Source: §4.1 · p. 4

Reasoning perspectives

  • Mechanistic reasoning (35.6%) and quantitative derivation (30.2%) dominate the dataset, alongside boundary calibration (20.4%) and hypothesis falsification (13.9%).Source: §4.1 · p. 4

Length and diversity

  • Spark-234K has the longest average questions at 377.61 tokens, while responses average 3139.45 tokens and rank second overall.Source: §4.2 · p. 5
  • Despite its compact size, Spark-234K achieves the highest Vendi Score (384.57) and Centroid Distance (0.6183) among compared datasets.Source: §4.2 · p. 5

Quality

  • Automated checks report 93.28% answer consistency, while expert evaluations agree at 90.3% for correctness, 97.7% for self-containment, and 94.3% for difficulty.Source: §4.2 · pp. 5–6

Difficulty

  • Over 93% of questions require high-order reasoning, with L4 at 58.5% and L5 at 34.6%, while L1 and L2 comprise only 0.02% and 1.2%.This distribution arises without explicit difficulty-based filtering.Source: §4.2 · p. 5, Figure 5 · p. 6

Quality

  • Quality control raises Spark-234K’s self-containment rate to 99.73%, following a 98.55% rate before filtering.Source: §4.2 · p. 5, §4.1 · p. 5

5 Experiments

Experiments compare Spark-234K across scientific benchmarks, model backbones, disciplines, data scales, and SPARK design ablations. Spark-234K delivers strong performance with fewer examples, especially on challenging reasoning tasks, while each synthesis perspective contributes complementary signal.

Main results

  • Across three backbones, Spark-234K achieves the highest macro-average performance using 234K examples, less than one-fifth the volume of MegaScience.For Qwen3-8B, the average score is 60.83 versus 56.36 for the strongest baseline.Source: §5.2 · p. 6
  • On Qwen3-14B, Spark-234K reaches GPQA-M 53.71 and GPQA-D 55.05, outperforming models trained on substantially larger corpora.The gains are particularly pronounced on complex, multi-step reasoning benchmarks.Source: §5.2 · pp. 6–7

Domain-specialized results

  • Spark-234K achieves the highest macro averages across evaluated chemistry, computer science, medicine, and mathematics benchmarks.Source: §5.3 · p. 7, Table 3 · p. 8
  • On Qwen3-14B MedQA-US, Spark-234K scores 79.36 versus 72.58 for MegaScience.Source: §5.3 · p. 7

Data efficiency and scaling

  • Performance improves consistently as the number of synthesized training records increases, indicating that the model has not saturated on the provided reasoning patterns.Source: §5.4 · p. 7, Figure 6 · p. 7

Ablations

  • Replacing skeleton extraction with whole-paper, chunked, or summarized inputs reduces performance by 9.15, 5.63, and 6.71 points, respectively.Whole-paper input performs worst because long documents diffuse central claims and yield shallow questions.Source: §5.5 · p. 8
  • Removing any reasoning perspective causes a performance drop under fixed data volume, with quantitative derivation producing the largest effect at −4.64.Source: §5.5 · p. 8

6 Related Work

Scientific reasoning datasets have expanded in scale and coverage, but many remain centered on settled knowledge or synthetic mixtures. SPARK instead reconstructs paper-level claim, evidence, and derivation chains to produce self-contained, mechanism-oriented questions.

  • Scientific post-training resources range from textbook-derived QA and domain-specific instruction data to broad synthetic collections.Source: §6.1 · p. 8
  • Existing scale and coverage improvements do not guarantee reasoning depth because textbooks favor settled knowledge and synthetic mixtures may diffuse scientific structure.Source: §6.1 · p. 8
  • Human annotations provide scientific grounding but resist scaling, motivating automated task synthesis from full papers, chunks, or retrieved passages.Source: §6.2 · p. 8
  • Whole-paper processing can diffuse core claims, while arbitrary chunking can sever cross-section evidence chains.Source: §6.2 · p. 8
  • SPARK distills claims, evidence, and derivations into a compact reasoning skeleton that yields self-contained, mechanism-oriented questions.Source: Table 4 · p. 8, §6.2 · p. 9

7 Conclusion

SPARK addresses limitations in scientific reasoning data by synthesizing from claim–evidence–derivation structures, reasoning skeletons, four scientific perspectives, and strict consistency checks. The resulting Spark-234K dataset is reported to offer greater difficulty and diversity while improving base-model scientific reasoning capabilities.

  • SPARK treats a paper’s claim–evidence–derivation structure as the fundamental unit for scientific reasoning-data synthesis.Source: §7 · p. 9
  • The framework distills papers into reasoning skeletons, generates questions across four scientific perspectives, and applies strict consistency verification.Source: §7 · p. 9
  • Spark-234K offers greater difficulty and diversity than existing scientific reasoning data, according to the conclusion.Source: §7 · p. 9
  • Experiments demonstrate that Spark-234K substantially improves the scientific reasoning capabilities of base models.Source: §7 · p. 9

Limitations

The paper identifies limitations in experimental scope and dataset verification, while the construction pipeline filters, samples, and quality-controls a broad scientific corpus. Automated checks provide large-scale control, but comprehensive expert evaluation remains limited.

Limitations

  • Experiments are limited to supervised fine-tuning because of computational-resource constraints, leaving reinforcement learning for future work.Source: Limitations · p. 9
  • Comprehensive expert verification of the entire dataset is prohibitively expensive, so large-scale quality control relies primarily on automated consistency checks.Source: Limitations · p. 9
  • 300 samples across all ten disciplines receive stratified human validation, while substantially larger-scale expert evaluation remains future work.Source: Limitations · p. 9

Corpus scope

  • SCI-BASE contains ∼3.36M papers across ten disciplines, with Medicine & Health Sciences and Life Sciences accounting for 61.3% of the distribution.Source: A The Source Corpus: SCI-BASE · p. 12

Data construction

  • Filtering retains papers with clear claim–evidence trajectories and concrete observables, while excluding incomplete, non-English, review, or insufficiently structured documents.Source: B Construction Funnel · p. 12

Quality control

C Training Details

Training uses LlamaFactory with standardized hardware and batching settings, including adjusted gradient accumulation for 32-GPU runs. Table 7 summarizes the remaining shared hyperparameters.

Training configuration

D Evaluation Details

Evaluation covers general science reasoning and domain-specific transfer using standardized prompting, decoding, and answer extraction. Baseline comparisons use widely adopted open-source scientific SFT datasets with complete chain-of-thought traces.

Benchmark organization

  • Evaluation benchmarks comprise general science reasoning and domain-specific groups spanning chemistry, computer science, medicine, and mathematics.Source: D Evaluation Details · p. 13
  • The general group covers expert science QA, quantitative scientific problem solving, and broad multidisciplinary science understanding.Source: D Evaluation Details · p. 13

Evaluation implementation

  • Evaluation reuses MegaScience2’s codebase and uses zero-shot inference on 8 GPUs with temperature 0, reporting avg@3 or avg@5 by benchmark.Source: D Evaluation Details · p. 13

Baseline comparison

  • Baseline datasets are selected for prior adoption, public accessibility, and complete chain-of-thought reasoning traces suitable for supervised fine-tuning.Source: §E.1 · pp. 13–14

E.2 Controlled Comparison with WildSci

SPARK is compared with WildSci through matched supervised fine-tuning and differs in its skeleton-guided processing, adaptive question generation, and skeleton-grounded verification. Spark-234K outperforms the SFT-adapted WildSci data across all four overlapping benchmarks.

  • WildSci provides verified multiple-choice answers but lacks reasoning traces required for direct supervised fine-tuning comparison.Source: §E.2 · p. 14
  • The controlled comparison filters WildSci questions by decontamination, valid answer generation, and agreement with gold options before sampling 20K instances.Source: §E.2 · p. 14
  • 2.30 points: Spark-234K averages 50.02 versus 47.72 for SFT-adapted WildSci across four overlapping benchmarks.Source: §E.2 · p. 14
  • SPARK extracts a paper-level reasoning skeleton and evidence trajectory before generating questions, whereas WildSci processes the full paper directly.Source: §E.2 · p. 14
  • SPARK generates at most one question per supported perspective and skips unsupported perspectives, adapting question production to each paper’s evidence.Source: §E.2 · p. 14
  • SPARK verifies answers against the extracted reasoning skeleton rather than relying on model voting to identify divergent or unanswerable items.Source: §E.2 · p. 14

I Human Validation of Quality Assessments

A stratified audit compares automated quality assessments with independent expert judgments across correctness, self-containment, and difficulty. The validation also documents the metrics and procedures used to assess dataset diversity and self-containment.

Human validation

Assessment dimensions

Human validation

Diversity metrics

Self-containment

  • Baseline self-containment failures can omit definitions, equations, or variables, making fluent questions unusable for standalone reasoning.Source: K Self-Containment Bad Cases · p. 17

Related papers

Loading 2608.30214v1…