Source-linked AI summary
SPARK: Skeleton-Guided Reasoning Synthesis from Large-Scale Scientific Literature
Yu Li, Wei Li, Xin Gao, Mengyuan Sun, Xiaoyang Wang, Qizhi Pei, Lijun Wu
TL;DR
Scientific reasoning datasets often emphasize factual recall and routine computation while underrepresenting mechanism, evidence, and hypothesis evaluation. SPARK reconstructs paper-level claim-evidence-derivation structures into reasoning skeletons and uses them to generate and verify diverse scientific tasks. Spark-234K achieves strong benchmark performance with substantially fewer examples, while its evaluation relies mainly on automated consistency checks and remains limited to supervised fine-tuning.
Problem
Existing scientific reasoning data underrepresents evidence-based, mechanism-driven, and hypothesis-evaluation reasoning found in scientific discovery.
Method
SPARK distills papers into claim-centered reasoning skeletons, generates tasks from four scientific perspectives, and verifies their self-containment and consistency.
Results
Spark-234K outperforms larger baseline datasets across base-model families and parameter scales, including average margins of +3.10, +4.47, and +3.43 points.
Takeaways & Limitations
Spark-234K achieves strong downstream scientific reasoning performance with substantially fewer training examples, highlighting dataset construction quality alongside scale.
Takeaways & Limitations
Experiments are limited to supervised fine-tuning, and large-scale quality control relies primarily on automated consistency checks with human validation on 300 samples.
Abstract
from arXiv · showhide
Scientific reasoning remains challenging for open-source models, largely due to the lack of high-quality scientific reasoning data. Existing datasets are often dominated by factual recall or formulaic problem solving, with limited emphasis on mechanism understanding, evidence-grounded reasoning, and hypothesis evaluation. To address this, we introduce SPARK (Scientific Paper Abstracted Reasoning sKeleton), a paper-oriented synthesis framework built on Sci-Base, a large-scale corpus of research papers spanning 10 scientific disciplines. Instead of directly converting papers into question-answer pairs, SPARK treats the claim-evidence-derivation structure of a paper as the fundamental unit of reasoning synthesis. Specifically, SPARK (1) distills each paper into a compact reasoning skeleton capturing its central claims and supporting evidence, enabling self-contained question generation, and (2) synthesizes reasoning tasks from four scientific perspectives: mechanistic reasoning, hypothesis falsification, quantitative derivation, and boundary calibration. A final consistency verification stage further removes unsupported or contradictory outputs. Using this framework, we construct Spark-234K, a scientific reasoning dataset with substantially higher difficulty and diversity than existing resources. Experiments show that Spark-234K consistently outperforms existing scientific reasoning datasets while achieving stronger performance with significantly fewer training samples.
1 Introduction
Scientific reasoning data often underrepresents evidence-based and mechanism-driven reasoning. SPARK addresses this by synthesizing tasks from paper-level claim-evidence-derivation structures, producing a compact dataset that performs strongly with fewer examples.
- Existing scientific corpora mainly emphasize factual recall, standard derivations, and routine calculations rather than evidence-based and mechanism-driven reasoning.
- SPARK treats a paper’s claim-evidence-derivation chain as the fundamental unit for scientific reasoning data synthesis.
- Each paper is distilled into a compact reasoning skeleton containing claims, evidence, assumptions, quantitative relations, and boundary conditions.
- SPARK generates tasks through mechanistic reasoning, hypothesis falsification, quantitative derivation, and boundary calibration, then verifies consistency.
- 234K synthesized instances from 370K frontier papers outperform larger baseline datasets, with average margins of +3.10, +4.47, and +3.43 points across evaluated settings.
2 Paper Collection and Preparation
SPARK draws from SCI-BASE, a structured, multi-disciplinary scientific literature corpus. The collection process filters papers for parsing quality, recency, language, argument clarity, and concrete observables while balancing disciplinary coverage.
- SCI-BASE contains approximately 3.36M papers across ten disciplines, with structured metadata and parsed paragraphs, tables, captions, and equations.
- The pipeline restricts papers to January 2024–March 2026 and removes non-English, incompletely parsed, or review articles lacking a clear claim-evidence trajectory.
- Selection prioritizes papers with concrete observables, including equations, statistical results, tables, and experimental comparisons.
- Balanced sampling mitigates the source corpus’s heavy skew toward medicine and life sciences.
3 Dataset Construction
SPARK reconstructs each paper as a claim-centered reasoning skeleton, generates questions from complementary scientific perspectives, and applies layered quality checks. The resulting pipeline preserves evidence trajectories while enforcing self-containment and consistency.
- Skeleton Extraction: SPARK treats a paper as an argument, identifying its core conclusion and reconstructing the supporting evidence trajectory as a compact skeleton.
- Skeleton Extraction: Skeleton extraction avoids context diffusion from full-paper prompting and fragmentation from section-level or chunk-based representations.
- Skeleton Extraction: The normalized input preserves equations, table text, captions, and numerical results while removing formatting artifacts and non-argument sections.
- Multi-Perspective Question Generation: Questions probe mechanisms, competing hypotheses, quantitative relations, and validity boundaries instead of only surface facts or plug-and-chug computation.
- Multi-Perspective Question Generation: SPARK generates at most one question per evidence-supported perspective and skips perspectives unsupported by the extracted skeleton.
- Skeleton-Grounded Quality Check: Rule-based and model-based filters remove malformed, non-self-contained, inconsistent, incoherent, or weakly supported question-answer pairs.
- Skeleton-Grounded Quality Check: Skeleton-grounded evaluation keeps quality control computationally lightweight by avoiding massive majority voting and iterative rewriting.
- Skeleton-Grounded Quality Check: Spark-234K applies semantic deduplication within disciplines and 13-gram matching against evaluation sets to reduce duplication and benchmark leakage.
4 Dataset Statistics and Analysis
Spark-234K is a 234K-instance scientific reasoning dataset with broad disciplinary and perspective coverage, high diversity, and predominantly advanced reasoning tasks. Its evaluations indicate strong correctness and self-containment, including after quality control.
- Spark-234K contains 234K synthesized scientific reasoning instances derived from 370K filtered seed papers.
- Disciplinary coverage: Physics, Mathematics & Computer Science, and Chemistry are the largest disciplinary segments, while ten disciplines provide broad thematic coverage.
- Reasoning perspectives: Mechanistic reasoning (35.6%) and quantitative derivation (30.2%) dominate the dataset, alongside boundary calibration (20.4%) and hypothesis falsification (13.9%).
- Length and diversity: Spark-234K has the longest average questions at 377.61 tokens, while responses average 3139.45 tokens and rank second overall.
- Length and diversity: Despite its compact size, Spark-234K achieves the highest Vendi Score (384.57) and Centroid Distance (0.6183) among compared datasets.
- Quality: Automated checks report 93.28% answer consistency, while expert evaluations agree at 90.3% for correctness, 97.7% for self-containment, and 94.3% for difficulty.
- Difficulty: Over 93% of questions require high-order reasoning, with L4 at 58.5% and L5 at 34.6%, while L1 and L2 comprise only 0.02% and 1.2%.This distribution arises without explicit difficulty-based filtering.
- Quality: Quality control raises Spark-234K’s self-containment rate to 99.73%, following a 98.55% rate before filtering.
5 Experiments
Experiments compare Spark-234K across scientific benchmarks, model backbones, disciplines, data scales, and SPARK design ablations. Spark-234K delivers strong performance with fewer examples, especially on challenging reasoning tasks, while each synthesis perspective contributes complementary signal.
- Main results: Across three backbones, Spark-234K achieves the highest macro-average performance using 234K examples, less than one-fifth the volume of MegaScience.For Qwen3-8B, the average score is 60.83 versus 56.36 for the strongest baseline.
- Main results: On Qwen3-14B, Spark-234K reaches GPQA-M 53.71 and GPQA-D 55.05, outperforming models trained on substantially larger corpora.The gains are particularly pronounced on complex, multi-step reasoning benchmarks.
- Domain-specialized results: Spark-234K achieves the highest macro averages across evaluated chemistry, computer science, medicine, and mathematics benchmarks.
- Domain-specialized results: On Qwen3-14B MedQA-US, Spark-234K scores 79.36 versus 72.58 for MegaScience.
- Data efficiency and scaling: Performance improves consistently as the number of synthesized training records increases, indicating that the model has not saturated on the provided reasoning patterns.
- Ablations: Replacing skeleton extraction with whole-paper, chunked, or summarized inputs reduces performance by 9.15, 5.63, and 6.71 points, respectively.Whole-paper input performs worst because long documents diffuse central claims and yield shallow questions.
- Ablations: Removing any reasoning perspective causes a performance drop under fixed data volume, with quantitative derivation producing the largest effect at −4.64.
6 Related Work
Scientific reasoning datasets have expanded in scale and coverage, but many remain centered on settled knowledge or synthetic mixtures. SPARK instead reconstructs paper-level claim, evidence, and derivation chains to produce self-contained, mechanism-oriented questions.
- Scientific post-training resources range from textbook-derived QA and domain-specific instruction data to broad synthetic collections.
- Existing scale and coverage improvements do not guarantee reasoning depth because textbooks favor settled knowledge and synthetic mixtures may diffuse scientific structure.
- Human annotations provide scientific grounding but resist scaling, motivating automated task synthesis from full papers, chunks, or retrieved passages.
- Whole-paper processing can diffuse core claims, while arbitrary chunking can sever cross-section evidence chains.
- SPARK distills claims, evidence, and derivations into a compact reasoning skeleton that yields self-contained, mechanism-oriented questions.
7 Conclusion
SPARK addresses limitations in scientific reasoning data by synthesizing from claim–evidence–derivation structures, reasoning skeletons, four scientific perspectives, and strict consistency checks. The resulting Spark-234K dataset is reported to offer greater difficulty and diversity while improving base-model scientific reasoning capabilities.
- SPARK treats a paper’s claim–evidence–derivation structure as the fundamental unit for scientific reasoning-data synthesis.
- The framework distills papers into reasoning skeletons, generates questions across four scientific perspectives, and applies strict consistency verification.
- Spark-234K offers greater difficulty and diversity than existing scientific reasoning data, according to the conclusion.
- Experiments demonstrate that Spark-234K substantially improves the scientific reasoning capabilities of base models.
Limitations
The paper identifies limitations in experimental scope and dataset verification, while the construction pipeline filters, samples, and quality-controls a broad scientific corpus. Automated checks provide large-scale control, but comprehensive expert evaluation remains limited.
- Limitations: Experiments are limited to supervised fine-tuning because of computational-resource constraints, leaving reinforcement learning for future work.
- Limitations: Comprehensive expert verification of the entire dataset is prohibitively expensive, so large-scale quality control relies primarily on automated consistency checks.
- Limitations: 300 samples across all ten disciplines receive stratified human validation, while substantially larger-scale expert evaluation remains future work.
- Corpus scope: SCI-BASE contains ∼3.36M papers across ten disciplines, with Medicine & Health Sciences and Life Sciences accounting for 61.3% of the distribution.
- Data construction: Filtering retains papers with clear claim–evidence trajectories and concrete observables, while excluding incomplete, non-English, review, or insufficiently structured documents.
- Quality control: 14.97% of generated QA pairs are removed during skeleton-grounded checking, including 65.87% of rejected pairs for inconsistency with the original paper skeleton.
C Training Details
Training uses LlamaFactory with standardized hardware and batching settings, including adjusted gradient accumulation for 32-GPU runs. Table 7 summarizes the remaining shared hyperparameters.
- Training configuration: Fine-tuning uses LlamaFactory, with experiments generally running on 16 GPUs, per-device batch size 4, and 2 gradient accumulation steps.
- Training configuration: 32-GPU runs use gradient accumulation of 1 to maintain a consistent global batch size.
- Training configuration: Table 7 summarizes the remaining shared hyperparameters.
D Evaluation Details
Evaluation covers general science reasoning and domain-specific transfer using standardized prompting, decoding, and answer extraction. Baseline comparisons use widely adopted open-source scientific SFT datasets with complete chain-of-thought traces.
- Benchmark organization: Evaluation benchmarks comprise general science reasoning and domain-specific groups spanning chemistry, computer science, medicine, and mathematics.
- Benchmark organization: The general group covers expert science QA, quantitative scientific problem solving, and broad multidisciplinary science understanding.
- Evaluation implementation: Evaluation reuses MegaScience2’s codebase and uses zero-shot inference on 8 GPUs with temperature 0, reporting avg@3 or avg@5 by benchmark.
- Baseline comparison: Baseline datasets are selected for prior adoption, public accessibility, and complete chain-of-thought reasoning traces suitable for supervised fine-tuning.
E.2 Controlled Comparison with WildSci
SPARK is compared with WildSci through matched supervised fine-tuning and differs in its skeleton-guided processing, adaptive question generation, and skeleton-grounded verification. Spark-234K outperforms the SFT-adapted WildSci data across all four overlapping benchmarks.
- WildSci provides verified multiple-choice answers but lacks reasoning traces required for direct supervised fine-tuning comparison.
- The controlled comparison filters WildSci questions by decontamination, valid answer generation, and agreement with gold options before sampling 20K instances.
- 2.30 points: Spark-234K averages 50.02 versus 47.72 for SFT-adapted WildSci across four overlapping benchmarks.
- SPARK extracts a paper-level reasoning skeleton and evidence trajectory before generating questions, whereas WildSci processes the full paper directly.
- SPARK generates at most one question per supported perspective and skips unsupported perspectives, adapting question production to each paper’s evidence.
- SPARK verifies answers against the extracted reasoning skeleton rather than relying on model voting to identify divergent or unanswerable items.
I Human Validation of Quality Assessments
A stratified audit compares automated quality assessments with independent expert judgments across correctness, self-containment, and difficulty. The validation also documents the metrics and procedures used to assess dataset diversity and self-containment.
- Human validation: The audit samples 300 questions across ten disciplines, with two blinded domain experts evaluating each discipline’s sample and resolving disagreements through discussion.
- Assessment dimensions: Experts independently solve questions for correctness, judge independent answerability for self-containment, and assign one of five reasoning levels for difficulty.
- Human validation: 90.3% agreement with independently derived expert answers supports the reliability of the dataset’s original answers.
- Human validation: 97.7% and 94.3% agreement with original labels support the reliability of automated self-containment and difficulty assessments.
- Diversity metrics: Vendi Score measures effective independent semantic modes from normalized eigenvalues of a cosine-similarity kernel.
- Diversity metrics: Centroid Distance measures embedding dispersion around the dataset centroid, with higher values indicating broader semantic coverage.
- Self-containment: Baseline self-containment failures can omit definitions, equations, or variables, making fluent questions unusable for standalone reasoning.