Source-linked AI summary
SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research
Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, Kai Yu
TL;DR
Existing scientific LLM benchmarks are often objective-only, vulnerable to data leakage, and limited in disciplinary or capability coverage. SciEval addresses these gaps with a multi-dimensional benchmark combining objective, subjective, and dynamically generated questions, while experiments show strong performance from GPT-series and Claude-series models but generally weak benchmark performance.
Problem
Existing scientific benchmarks are often objective-only, discipline-specific, and exposed to potential data leakage, limiting comprehensive evaluation of scientific LLM abilities.
Method
SciEval evaluates LLMs across four Bloom’s-taxonomy-based dimensions using about 18,000 questions spanning chemistry, physics, and biology, with objective, subjective, and dynamically generated data.
Results
Most LLMs perform poorly on SciEval, while GPT-series and Claude-series models are exceptions; GPT-4 is strongest overall, but dynamic questions remain challenging.
Takeaways & Limitations
SciEval provides a benchmark for assessing scientific LLM capabilities across multiple dimensions while mitigating potential data leakage.
Takeaways & Limitations
Claude-Instant and Claude were unavailable in CoT and 3-Shot evaluations because of API limitations.
Abstract
from arXiv · showhide
Recently, there has been growing interest in using Large Language Models (LLMs) for scientific research. Numerous benchmarks have been proposed to evaluate the ability of LLMs for scientific research. However, current benchmarks are mostly based on pre-collected objective questions. This design suffers from data leakage problem and lacks the evaluation of subjective Q/A ability. In this paper, we propose SciEval, a comprehensive and multi-disciplinary evaluation benchmark to address these issues. Based on Bloom's taxonomy, SciEval covers four dimensions to systematically evaluate scientific research ability. In particular, we design a "dynamic" subset based on scientific principles to prevent evaluation from potential data leakage. Both objective and subjective questions are included in SciEval. These characteristics make SciEval a more effective benchmark for scientific research ability evaluation of LLMs. Comprehensive experiments on most advanced LLMs show that, although GPT-4 achieves SOTA performance compared to other LLMs, there is still substantial room for improvement, especially for dynamic questions. The codes and data are publicly available on https://github.com/OpenDFM/SciEval.
1 Introduction
SciEval addresses limitations in existing scientific LLM benchmarks by combining broad, multi-level evaluation with objective, subjective, and dynamically generated data. It covers multiple scientific fields and uses Dynamic Data to reduce potential data leakage.
- Existing scientific benchmarks are often discipline-specific, objective-only, and vulnerable to data leakage, limiting comprehensive assessment of scientific LLM abilities.
- SciEval contains about 18,000 challenging English questions across chemistry, physics, and biology, with multiple sub-topics.
- SciEval evaluates scientific capability across basic knowledge, knowledge application, scientific calculation, and research ability using Bloom’s taxonomy.
- The benchmark combines objective questions for standardized knowledge evaluation with subjective Experimental Data questions covering twelve basic science experiments.
- Dynamic Data is generated from basic scientific principles, updated regularly, and maintained alongside a stable version to reduce potential leakage and support fair comparisons.
2 Related Work
Related work includes broad multi-task benchmarks and domain-specific scientific evaluations. These benchmarks span general reasoning, medical, mathematical, and educational science tasks.
- General-purpose benchmarks evaluate language models across broad task collections, including language understanding, common-sense reasoning, standardized exams, and Chinese knowledge.
- Scientific and downstream benchmarks target narrower areas such as medical question answering, mathematical reasoning, and school-level multimodal science questions.
3 The SciEval Dataset
SciEval structures scientific-research evaluation across multiple cognitive levels and four knowledge dimensions, using Static, Dynamic, and Experimental Data. Its data collection combines curated sources, generated questions, dynamic scientific-principle-based instances, and experiments.
- 3.1 Scientific Research Evaluation System: SciEval applies Bloom’s cognitive domain across six levels to evaluate scientific capability at four dimensions: Basic Knowledge, Knowledge Application, Scientific Calculation, and Research Ability.The dimensions range from fundamental knowledge and problem solving to research tasks including formulation, experimental design, data analysis, and summarization.
- 3.2 Data Types: The dataset uses Static Data for all four dimensions, Dynamic Data for Knowledge Application and Scientific Calculation, and Experimental Data for twelve scientific experiments.Dynamic Data is regularly updated, while Static Data remains constant for comparison.
- 3.2 Static Data: Static Data is collected from biology, chemistry, and physics materials, preprocessed with rule-based methods, and converted into evaluated question formats with GPT-4 assistance.GPT-4 processes question titles, simplifies ground-truth answers, generates three wrong answers, and classifies knowledge domains; generated content is manually checked.
- 3.2 Static Data: Static Data is diversified with MedQA, PubMedQA, and Reagent Selection, including USMLE test data, 1000 expert-annotated PubMedQA items, and 40% of Reagent Selection data.PubMedQA items are framed as judgment questions, while Reagent Selection items are formulated as multiple-choice questions.
- 3.2 Dynamic Data: Dynamic Data generates chemistry questions from PubChem molecular information and physics questions from manually written scripts based on physics formulas.Users receive regenerated versions regularly, while a stable version is maintained for fair model-performance comparison.
- 3.3 Data Statistics: For Static Data, each source, knowledge domain, and discipline contributes five randomly selected development items, with remaining data split 1:9 between validation and test sets.The development set supports few-shot learning.
4 Experiment
SciEval evaluates accessible LLMs across static, dynamic, and experimental scientific data using answer-only, chain-of-thought, and 3-shot settings. GPT-4 generally leads, but results reveal substantial weaknesses in dynamic questions, scientific calculation, physics, and experimental-result analysis.
- 4.1 Experiment Setup: Accuracy is used for objective questions, MSE for numerical chemistry answers, BLEU and exact match for string answers, and manual assessment for experimental responses.Dynamic physics questions are multiple-choice, whereas chemistry questions include molecular-weight and SMILES-expression tasks.
- Answer-Only Setting: GPT-4 performs significantly better than other models on Static Data, while only GPT-4, GPT-3.5-turbo, and Claude-v1.3 exceed 60% average accuracy.The leaderboard is sorted by average Static Data accuracy.
- Answer-Only Setting: Experimental results show strong performance on experimental principles and design but unsatisfying performance when models analyze experiment results.GPT-series and Claude-series models achieve good overall results, whereas two other evaluated models do not.
- CoT Setting and 3-Shot setting: Chain-of-thought improves only GPT-series performance on Static Data, while roughly half of models improve with 3-shot prompting.On Dynamic Data, CoT and 3-shot prompting significantly improve most chemistry results, but physics performance remains nearly random; GPT-4 reaches 51.01 accuracy with 3-shot prompting.
5 Conclusion
SciEval is presented as a benchmark for evaluating LLM scientific capabilities across three fundamental science fields and four dimensions, using objective, subjective, and dynamically generated data.
- SciEval contains about 18,000 challenging scientific questions covering three fundamental science fields.
- The benchmark assesses LLM scientific ability across four dimensions and includes both objective and subjective questions.
- SciEval employs dynamic data generation to mitigate potential data leakage and supports comprehensive experiments across advanced LLMs.
- Most evaluated LLMs perform poorly on SciEval, except GPT-series and Claude-series models.