PaperlayerView exact revision on arXiv

Source-linked AI summary

Expert-validated STEM QA

Kihwan Han, Saurabh Patil, Chinmayee Shukla, Abhinav Sharma, Marko Pavlovic, Anshuman Lall, Mahesh Joshi

arXiv:2608.28591v1cs.AI

TL;DR

Existing STEM benchmarks have documented gaps in saturation, taxonomy coverage, question format, and answer reliability. Expert-validated STEM QA presents a reviewed 398-item dataset, on which models scored below 25%; fine-tuning on a separate dataset improved performance.

  • Problem

    Existing STEM benchmarks show saturation, skewed taxonomies, scientist-misaligned question formats, and possible answer inaccuracies.

    Source: §1 · pp. 1–2

  • Method

    The authors built a 398-item Physics, Chemistry, Biology, and Mathematics dataset using an expert-informed taxonomy and structured, multi-stage review.

    Source: §1 · p. 2, §2.1.1 · p. 3

  • Results

    Frontier and open-source models scored below 25% of total benchmark scores; fine-tuning Qwen 3.5 9B Q5_K_M improved HLE-verified STEM-subset overall scores by 15% of baseline (p=0.045).

    Source: §4 · p. 8, §3.4 · p. 8

  • Takeaways & Limitations

    The authors conclude that expert-validated STEM QA could help advance AI-in-STEM research.

    Source: §4.3 · p. 9

    The authors could not systematically evaluate models’ reasoning processes because access to those processes varies across models, especially proprietary ones.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

Recent advancements in AI are helping scientists achieve breakthroughs in fields such as mathematics, medicine, and materials sciences. New evaluation datasets for AI models contribute to such advancement in AI. In the STEM domain, frontier models have consumed most of the available online data, creating the need for human-created datasets that codify the knowledge of leading experts in the domain. There are several STEM datasets available for the research community in this field. However, there are some gaps in these datasets, leaving room for improvement. Examples of gaps include (1) saturation in model performance on these datasets, leaving no head-room for meaningful evaluations, (2) skewed taxonomy distributions, (3) multiple choice question format that is misaligned with how scientists use AI in the real world, and (4) inaccurate answers and rationales partially led by a contest-based data collection and a time-bound review process. In this study, we present 'Expert-validated STEM QA', a high-quality, expert-validated STEM dataset (N=398) in Physics, Chemistry, Biology, and Mathematics, created by 241 domain experts. We (1) carefully designed a taxonomy with balanced distribution, (2) vetted question contributors with quality-driven incentive, (3) conducted multiple rounds of reviews with revisions validated by domain experts based on consensus, and (4) created the dataset in verifiable question and answer format. Our study demonstrated low performance ($<25\%$) of frontier AI models on the dataset as a benchmark. Post-training on a separate, private version of the dataset (N=2,000) increased performance of the open source model by $15\%$ relative to the baseline model (p=0.045) on the STEM subset of HLE-verified dataset, indicating potential utility of the dataset for model training. We have open-sourced a portion of our dataset for the AI research community.

1 INTRODUCTION

Expert-validated STEM QA addresses gaps in scientific-reasoning benchmarks with an expert-reviewed dataset spanning four STEM disciplines. The introduction frames this effort against scientists’ need for multi-step reasoning and growing use of AI in research.

1 INTRODUCTION

  • Scientific research builds incrementally and requires multi-step reasoning, while AI is increasingly adopted by scientists and supports advances such as protein and biomolecular-complex structure prediction [16].The introduction situates the dataset among existing scientific-reasoning benchmarks [25] [28] [26] [18] [19] [21] [12].Source: §1 · p. 1
  • Existing scientific benchmarks can be saturated, use formats misaligned with scientific practice, skew taxonomies, omit domains, or contain inaccurate or ambiguous questions [25] [25] [26] [28] [28] [21] [12].Online-resource curation can raise data-leakage concerns [26] [18] [19], and some datasets lack human verification [18] [21], risking misleading performance estimates and apparent confidence.Source: §1 · pp. 1–2
  • The authors present Expert-validated STEM QA, a 398-item dataset spanning Physics, Chemistry, Biology, and Mathematics, built with a designed taxonomy and consensus-based expert review.Its construction also used quality-driven incentives; the question, explanation, and answer form a triplet, produced through a four-level review process with revisions.Source: §1 · p. 2, Figure 1 · p. 2

2 MATERIALS AND METHODS

The methods describe dataset design, expert annotation and review, dataset characterization, model assessment, post-training, and statistical testing. The workflow aimed to produce expert-validated question, explanation, and answer instances across PCMB domains.

2.1.1 Dataset design

  • SMEs and team leads defined question–explanation–answer instances, PCMB domains, proficiency, volume, and an expertise-based taxonomy informed by GPQA, journals, research papers, and SME discussion.Annotators contributed within their own expertise rather than being assigned uniform coverage across taxonomy categories.Source: §2.1.1 · p. 3
  • Review rubrics used atomic Pass/Fail judgments; annotator metadata documented web-searchability and whether model failures reflected reasoning limits rather than ambiguity.Automated reviewers assessed criteria deemed reliable for agents, calibrated against trusted reviewers’ golden examples until held-out precision and recall each reached at least 80%.Source: §2.1.1 · p. 3

2.1.2 Talent pool of annotators, reviewers, and team leads

  • Candidates completed reasoning and domain-specific screening, technical interviews, and bootcamp training; approximately 90% were off-boarded, while remaining contributors, primarily doctoral researchers with 5–10 years’ experience, became reviewers.Team leads with academic or industry experience held regular meetings to maintain quality.Source: §2.1.2 · pp. 3–4

2.1.3 Annotation and review process

  • Examples underwent agentic review, L1 checks of integrity and formatting, L2 expert review of scientific validity and accuracy, revisions or documented disputes, and final team-lead spot checks.Agentic review addressed lower-level criteria such as linguistic clarity and web-searchability; L2 review was intended to mitigate individual reviewer bias.Source: §2.1.3 · p. 4

2.2.1 Dataset specifications

  • Dataset characterization covered total volume, domain and taxonomy breakdowns, and token-count distributions for questions and combined explanations and answers.The analysis also specified the dataset in multiple dimensions before assessing model performance.Source: §2.2.1 · p. 4, §2.2 · p. 4

2.2.2 Assessment of model performance on the dataset

  • Ten model variants across proprietary and open-source families were assessed using modified HLE prompts; responses were graded by GPT-4o and Qwen 3.5-4B-UD-Q4_K_XL against manual grading.Open-source reasoning-model consistency was assessed using pass rate and pass@k across 8 independent trials.Source: §2.2.2 · pp. 4–5

2.3 Post-training

  • Qwen 9B with reasoning disabled was fine-tuned using LoRA on final-answer labels from a separate proprietary dataset (N=2,000), then evaluated against its baseline on HLE-verified STEM subsets.The held-out external test set comprised 1,309 instances; HLE-verified was selected for structural similarity and verified question-and-answer accuracy.Source: §2.3 · p. 5

2.4 Statistical analysis

  • Confidence-score comparisons used Mann–Whitney U tests after normality tests, with Bonferroni correction; post-training changes in correct and incorrect responses were assessed using McNemar’s test.Tests used α=0.05.Source: §2.4 · p. 5

3 RESULTS

All 10 evaluated models scored below 25% on the dataset, while fine-tuning Qwen 3.5 9B on a separate private version improved its score on the HLE-verified STEM subset. Results also characterize the dataset’s composition and reveal patterns in model confidence and reasoning.

3.1 Annotator, reviewer, and team lead statistics

  • The recruitment process yielded 241 annotators, reviewers, and team leads, 80% of whom held PhDs in their areas of expertise.Source: §3.1 · p. 5

3.2 Dataset generation results

  • 398 examples were balanced across four domains, but Mathematics and Biology had concentrated L1 categories; Biology’s large “other” L2 group comprises diverse singleton topics.Mathematics examples concentrated in discrete mathematics (49%) and Biology in molecular biology (70%); the L2 categories within the largest L1 categories were well distributed in all domains. The L1 skews reflected annotator expertise, and Biology’s molecular-biology skew also reflected its broad conceptual scope.Source: §3.2 · p. 5, Figure 2 · p. 6
  • Token counts showed domain-specific patterns, with Chemistry driving upper long tails and Mathematics questions shorter than those in other domains.Combined response token counts were comparable for Mathematics, Physics, and Chemistry, but lower for Biology.Source: §3.2 · p. 6, Figure 3 · p. 6

3.3 Model performance on the dataset

  • Below 25%: all 10 models scored under this threshold, with proprietary models outperforming open-source models overall.Higher reasoning effort was associated with higher scores across models, and larger open-source models scored higher; Qwen 3.5 open-source models were relatively weak in Chemistry.Source: §3.3 · p. 6, Figure 4 · p. 7
  • Above 80%: median confidence for nearly all models exceeded this level despite low scores, indicating frequent high-confidence errors.GPT models expressed lower confidence than other model families; confidence distributions for incorrect answers had more pronounced lower tails than those for correct answers, with a GPT 5.2 medium-effort exception. Significant differences were reported for Claude Opus 4.5 high reasoning and Qwen 3.5 9B reasoning off after Bonferroni correction.Source: §3.3 · p. 7, Figure 4 · p. 7
  • Reasoning traces showed open-source models entering extended loops that exhausted completion-token budgets and appearing to retrieve training memories instead of generating multi-step reasoning.For Qwen 3.5 27B and 9B, most pass rates across eight trials were below 4/8; the 27B pass@k curve stayed higher, but neither curve had a steep elbow.Source: §3.3 · p. 7, Figure 5 · p. 8

3.4 Post-training results

  • 15% of baseline performance was gained on the HLE-verified STEM subset by fine-tuning Qwen 3.5 9B Q5_K_M on a separate proprietary dataset.The gain was statistically significant (p=0.045); improvements were 22% of baseline on the verifiable question-and-answer subset and 5.8% on the multiple-choice subset. The effect size was moderate, and some baseline-correct answers became incorrect after fine-tuning.Source: §3.4 · p. 8, Figure 6 · p. 8

4 DISCUSSIONS

The discussion presents Expert-validated STEM QA as an expert-reviewed benchmark with low model scores and illustrative training utility, while emphasizing leakage prevention, holistic evaluation, and limits that shape future work.

4 DISCUSSIONS

  • Frontier and open-source models scored below 25% overall, while post-training on a separate private dataset (N=2,000) increased the open-source model’s score by 15% of baseline performance (p=0.045).The dataset contains N=398 questions and was created through consensus-driven review by 241 domain experts, 80% of whom earned PhDs.Source: §4 · p. 8

4.1 Highlights of the dataset

  • Expert validation and consensus-based review address context-dependent correctness, while verified search-proof evidence and private data help limit leakage and preserve evaluation headroom.The authors also report expert-verified responses intended to favor genuine multi-step reasoning over questions designed to exploit model weaknesses.Source: §4.1 · pp. 8–9
  • Post-training experiments illustrate potential training utility, but the authors caution that practical use requires more sophisticated reasoning strategies and larger data volumes.A larger human-expert-validated dataset is available to support pre-, mid-, and post-training.Source: §4.1 · p. 9

4.2 Findings in relation to prior literature

  • Model evaluation should extend beyond answer accuracy to hallucination and reasoning efficiency, as models showed high confidence in incorrect answers and sometimes repetitive reasoning loops.The authors frame the loops as an empirical observation motivating systematic study, including metrics such as deep-thinking ratio.Source: §4.2 · p. 9

4.3 Limitations and future directions

  • Interpretation is limited by unavailable reasoning traces, the dataset’s smaller volume (N=398), and SFT on a non-reasoning model, which is susceptible to catastrophic forgetting.Future work includes systematic reasoning-process assessment, more sophisticated post-training such as reinforcement learning with data mixture, and evaluation of more recent models; the authors aim to advance AI research in STEM.Source: §4.3 · p. 9

A APPENDIX · A.1 Taxonomy · A.1.2 Chemistry

The chemistry taxonomy organizes topics across analytical, biochemical, general, electrochemical, inorganic, medicinal, organic, physical, and polymer chemistry.

A.1.2 Chemistry

  • Analytical Chemistry covers separation processes and NMR, while Biochemistry includes amino acids, enzymes, proteins, natural products, and synthesis.Source: §A.1.2 · p. 14
  • General Chemistry spans catalysis, reactions, molecular structure, and mechanisms; Electrochemistry covers the Debye-Hückel limiting law, electrosynthesis, Gibbs free energy, and redox reactions.Source: §A.1.2 · p. 14
  • Inorganic Chemistry includes organometallic chemistry, point groups, reaction products, and transition metal complexes; Medicinal Chemistry includes drug design, reaction products, and synthesis.Source: §A.1.2 · p. 14
  • Organic Chemistry covers nomenclature, reaction mechanisms and products, reactions and synthesis, and reactivity.Source: §A.1.2 · p. 14
  • Physical Chemistry includes chemical kinetics, equilibrium chemistry, statistical thermodynamics, and thermodynamics.Source: §A.1.2 · p. 14
  • Polymer Chemistry covers crystalline phase behavior, molar mass determination, network polymers, polymers in solution, and polymer kinetics.Source: §A.1.2 · p. 15

A.1.4 Biology

The Biology taxonomy spans molecular, cellular, organismal, and applied topics, with additional coverage of neurobiology and proteomics.

A.1.4 Biology

  • Biology coverage includes biochemistry, cell biology, genetics, immunology, microbiology, and molecular biology, with topics ranging from protein function and signaling to host–pathogen interactions.Listed areas also include cancer cell growth, gene expression, mutations, immunotherapy, virology, and antibiotic resistance.Source: §A.1.4 · p. 15
  • General Biology topics extend across development, disease, cellular processes, and experimental methods, including gene editing, DNA repair, tissue remodeling, and PCR techniques.The list also covers cancer complications, diabetes-related complications, imaging, immunohistochemistry, and synthetic biology and metabolic engineering.Source: §A.1.4 · p. 15
  • Neurobiology includes brain tumor biology, synaptic transmission mechanisms, and NMR brain diagnostics, while Proteomics includes plant–pathogen protein biochemistry and signal transduction profiling.Source: §A.1.4 · p. 16

A.2 Prompts for model response and grading, and label for post-training

The appendix provides prompts for model responses and grading during scoring, plus a model-response prompt and label for post-training.

A.2 Prompts for model response and grading, and label for post-training

A.3 Review rubrics

This section presents the review rubrics.

A.3 Review rubrics

  • Table S1 is titled “Review rubrics.”The passage provides the table title but not its rubric criteria.Source: Table S1 · p. 17

A.4 Annotator, reviewer, and team lead credentials … A.7 Additional post-training results

The appendices document annotator and reviewer credentials, model response characteristics, and additional baseline-versus-post-training analyses on HLE-verified STEM tasks.

A.4 Annotator, reviewer, and team lead credentials

  • Table S2 presents credential breakdowns for annotators, reviewers, and team leads.Source: Table S2 · p. 18

A.5 Token count distribution

  • Figure S5 plots model completion-token-count distributions after excluding examples with no model response.Source: Figure S5 · p. 18

A.6 Example model responses

  • Figure S6 shows reasoning snapshots from Qwen3.5-27B and Qwen3.5-9B, highlighting repeated phrases and expressions of confusion in sky blue.Source: Figure S6 · p. 19

A.7 Additional post-training results

  • Table S3 reports baseline and post-training correct-versus-incorrect task counts for Qwen 3.5 9B Q5_K_M on HLE-verified STEM overall.Source: Table S3 · p. 19
  • Table S4 reports baseline and post-training correct-versus-incorrect task counts for Qwen 3.5 9B Q5_K_M on HLE-verified STEM MCQ.Source: Table S4 · p. 19
  • Table S5 reports baseline and post-training correct-versus-incorrect task counts for Qwen 3.5 9B Q5_K_M on HLE-verified STEM VQA.Source: Table S5 · p. 19
Loading 2608.28591v1…