Source-linked AI summary
SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, Wei Wang
TL;DR
Existing scientific benchmarks often assess high-school subjects or elementary computations, leaving complex college-level scientific reasoning insufficiently examined. The paper introduces SciBench, evaluates diverse LLMs and prompting strategies on curated collegiate problems, and analyzes their errors by problem-solving skill. The best reported performance reaches 43.22%, while prompting strategies show mixed skill-specific effects rather than a universally superior approach.
Problem
Existing LLM benchmarks often focus on high-school or grade-level problems and elementary computations, limiting evaluation of complex scientific reasoning.
Method
SciBench curates college-level Mathematics, Chemistry, and Physics problems, evaluates diverse LLMs with multiple prompting strategies, and classifies errors across ten problem-solving abilities.
Results
43.22% is the best reported SciBench score, while prompting strategies produce mixed effects across scientific problem-solving skills.
Takeaways & Limitations
SciBench provides a framework for testing advanced scientific problem-solving abilities and reveals substantial room for improvement in current LLMs.
Takeaways & Limitations
SciBench currently covers a select group of scientific disciplines, with future extensions suggested for biology, engineering, social sciences, and humanities.
Abstract
from arXiv · showhide
Most of the existing Large Language Model (LLM) benchmarks on scientific problem reasoning focus on problems grounded in high-school subjects and are confined to elementary algebraic operations. To systematically examine the reasoning capabilities required for solving complex scientific problems, we introduce an expansive benchmark suite SciBench for LLMs. SciBench contains a carefully curated dataset featuring a range of collegiate-level scientific problems from mathematics, chemistry, and physics domains. Based on the dataset, we conduct an in-depth benchmarking study of representative open-source and proprietary LLMs with various prompting strategies. The results reveal that the current LLMs fall short of delivering satisfactory performance, with the best overall score of merely 43.22%. Furthermore, through a detailed user study, we categorize the errors made by LLMs into ten problem-solving abilities. Our analysis indicates that no single prompting strategy significantly outperforms the others and some strategies that demonstrate improvements in certain problem-solving skills could result in declines in other skills. We envision that SciBench will catalyze further developments in the reasoning abilities of LLMs, thereby ultimately contributing to scientific research and discovery.
1. Introduction
Existing benchmarks often use grade-level or high-school problems and basic computations, leaving complex scientific reasoning insufficiently tested. SCIBENCH addresses this gap with college-level problems, broad model and prompting evaluations, and skill-focused error analysis.
- Motivation: Existing benchmarks often use grade-level or high-school problems, basic computations, or primarily mathematical questions.This limits assessment of the reasoning required for complex scientific problem solving.
- Motivation: College-level Physical Chemistry examples show CoT can produce correct formulas but fail final numerical calculations, while external tools can trigger equation misunderstandings.In one example, λ1 is misplaced in the numerator rather than the denominator when Python is used.
- Contributions: SCIBENCH contains 869 open-ended problems from college Chemistry, Physics, and Mathematics textbooks, including 177 multimodal problems with visual contexts.The problems require multi-step reasoning, scientific concepts, domain-specific knowledge, and complex numerical computation.
- Contributions: The evaluation covers open-source and proprietary unimodal and multimodal LLMs under zero-shot, few-shot, CoT, and external-tool prompting strategies.External scientific-computing libraries include Python and Wolfram language.
- Contributions: The study compares generated solutions with correct solutions and uses human-assisted self-refinement to classify deficiencies into ten scientific problem-solving skills.The analysis examines which skills are affected by different prompting configurations.
2. Related Work
Prior benchmarks broaden scientific coverage but often emphasize lower-level subjects, basic arithmetic, or text-only questions. SCIBENCH instead targets college-level scientific concepts, advanced computation, and multimodal reasoning across Mathematics, Physics, and Chemistry.
- Existing benchmarks: Earlier benchmarks include mathematical, multimodal, multidisciplinary, and scientific-capability evaluations, but their task formats and educational levels vary.Examples include GSM8K, LILA, ScienceQA, BIG-Bench, SciEval, and JEEBench.
- Existing benchmarks: Most existing datasets emphasize lower educational levels and basic arithmetic, while many remain confined to textual-only problems without graphs or diagrams.These limitations leave advanced scientific computation and visual reasoning incompletely assessed.
- SCIBENCH: SCIBENCH focuses on college-level Mathematics, Physics, and Chemistry problems requiring scientific concepts, knowledge retrieval, calculus, differential equations, statistics, and probability.Its design emphasizes applying relevant principles rather than only recognizing them.
- SCIBENCH: SCIBENCH also includes multimodal problems that require integrating visual elements with scientific problem solving.This directly addresses the visual omissions identified in prior benchmarks.
3. The SCIBENCH Dataset
SCIBENCH is constructed from college textbooks and course exams to evaluate advanced scientific problem solving under realistic, difficult conditions. The dataset emphasizes free-response tasks, detailed solutions, complex computation, and visual elements.
- Selection criteria: Selected problems require domain-specific knowledge, adept calculation, and advanced numerical computation such as calculus and differential equations.Detailed solutions are included to support fine-grained analysis of model limitations.
- Selection criteria: The benchmark includes visual elements because many real-world scientific problems require integrating textual and visual information.This supports evaluation of multimodal problem-solving capabilities.
- Data sources: The dataset collects college-level textbook and course-exam problems across scientific and computational domains.Ten textbooks span Physics, Chemistry, and Mathematics, while exams cover Computer Science and Mathematics courses.
- Evaluation realism: A closed exam subset is designed to reduce the likelihood that questions appeared in LLM training data.The exam questions were manually extracted from PDFs and formatted into LaTeX documents.
- Answer processing: Free-response questions are preferred over multiple-choice questions, while textbook answers are standardized mainly as single numerical values for automated evaluation.The preprocessing also stores units separately and provides step-by-step solutions in LaTeX.
4. Experiments
The experiments evaluate diverse language models and prompting strategies on textbook, multimodal, and closed-exam scientific problems. Results show substantial variation across models, prompts, and tools, with GPT-4 generally leading but performance remaining challenging on exams.
- Experiment Setup: The evaluation covers seven unimodal LLMs, multiple prompting strategies, multimodal models, and closed collegiate exams.The textbook experiments include proprietary and open-source models, while additional evaluations cover visual-context problems and exam settings.
- Results and Analysis: The textbook results are reported as accuracy percentages, with weighted averages based on the number of problems in each textbook.Table 3 highlights the best score in bold and the second-best score with underlining.
- Results and Analysis: GPT-4-Turbo outperforms Mistral-7B by 34.76% in the zero-shot textbook setting.The benchmark differentiates model performance, with proprietary models consistently outperforming LLaMA-2 and Mistral across textbook settings.
- Results and Analysis: LLaMA-2-70B rises from 2.41% to 8.40% with few-shot prompting, while GPT-4 increases from 30.36% to 43.22% with external tools.Prompting and tool effectiveness varies significantly across models; GPT-4-Turbo performs better zero-shot but can trail GPT-4 in few-shot Python-assisted computation.
- Additional Experiments: GPT-4 achieves a 57.54% average score on the closed exam dataset but remains below student averages in reported Data Mining exams.GPT-4 scores 64.44% and 42.67% on the midterm and final, compared with student averages of 80.18% and 72.71%.
- Additional Experiments: Wolfram Language decreases few-shot performance by 6.70% for Claude2 and 6.17% for LLaMA-2-70B relative to CoT prompting.The paper offers syntax errors during translation into Wolfram Language as a plausible explanation for the deterioration.
5. Error Analysis of Prompting Strategies
The evaluation protocol distills LLM error reasons into ten scientific problem-solving skills and uses an LLM verifier to compare deficiencies across prompting settings. Results show that prompting strategies improve some abilities while sometimes worsening others.
- Evaluation Protocol: An LLM verifier classifies each incorrect solution by the specific deficient skill, producing error profiles for direct comparison across strategies.The protocol was designed to reduce the cost and time required for human attribution of error causes.
- Evaluation Protocol: Human annotators analyzed incorrect LLM and correct reference solutions, distilling 112 error annotations into ten essential problem-solving skills.The skills include logical decomposition, assumption identification, spatial perception, causal reasoning, problem deduction, abstract reasoning, scientific literacy, and code conversion.
- Prompting Results: 13.6% of zero-shot CoT errors were attributed to calculation ability, versus 29.0% for the vanilla zero-shot baseline.CoT therefore substantially reduced calculation-related errors in this comparison.
- Prompting Results: 32.2% of zero-shot CoT errors involved causal ability and 25.4% involved logical decomposition, compared with 18.3% for each skill under zero-shot prompting.The result indicates that CoT's calculation improvement was accompanied by higher error rates in these other abilities.
- Prompting Results: Few-shot CoT reduced causal-reasoning errors by 12.8% but increased errors associated with other skills, including logical decomposition.The comparison between zero-shot and few-shot CoT did not show universal improvement across scientific problem-solving skills.
6. Conclusion
SCIBENCH evaluates college-level scientific problem solving across diverse models and prompting strategies. The study finds that LLMs remain weak at problem solving despite strong performance on introductory mathematical benchmarks and assistance from tools.
- Conclusion: SCIBENCH covers college-level problems in Mathematics, Physics, Chemistry, Computer Science, and Mathematics exams.Its evaluation spans open-source and proprietary models, unimodal and multimodal settings, and varied prompting strategies.
- Conclusion: LLMs show impressive performance on introductory mathematical benchmarks but weak mastery of scientific problem-solving ability.The conclusion frames this gap as a limitation of current LLM capabilities.
- Conclusion: Current LLMs do not achieve satisfactory problem-solving performance even with assistance from various tools.The paper presents SCIBENCH and its evaluation protocol as a foundation for future research on understanding and enhancing these capabilities.
Reproducibility Statement
The paper supports reproducible research by documenting dataset processing, data collection interface design, experiment details, and LLM prompts, while releasing the dataset and code.
- Reproducibility Statement: The authors provide dataset-processing details, the data-collection UI design, experiment details, and LLM prompts.These materials are distributed across Section 3 and Appendices A.3 and C.
- Reproducibility Statement: The SCIBENCH dataset and code are publicly available through the paper's repository.The release is intended to foster reproducible research.
Ethical Statement
The dataset was manually examined for sensitive or ethical concerns, and exam-question inclusion was authorized by course instructors. Its textbook-question collection follows a limited-use Fair Use approach.
- Ethical Statement: Questions were sourced from science textbooks and exams and manually checked for potential sensitive background or ethical concerns.The authors state that inclusion of exam questions was authorized by the respective course instructors.
- Ethical Statement: The textbook dataset is intended solely for academic use and selects only a limited number of questions from each textbook.The authors describe this collection as adhering to Fair Use Law in the United States.
Impact Statement
SCIBENCH advances LLM evaluation by targeting collegiate-level mathematics, chemistry, and physics problems rather than primarily high-school subjects and basic algebra. Its scope is currently limited to a select group of scientific disciplines, leaving room for broader extensions.
- SCIBENCH evaluates LLMs on collegiate-level problems in mathematics, chemistry, and physics.
- The benchmark addresses existing evaluations’ emphasis on high-school subjects and basic algebraic operations.
- SCIBENCH’s current scope covers only a select group of scientific disciplines.
- Future extensions could add biology, computer science, engineering, social sciences, and humanities.
Supplementary Material for SCIBENCH
The supplementary material documents SCIBENCH’s textbook and exam data, annotation and preprocessing procedures, prompts, evaluation protocol, and examples of benchmark problems. It also provides supplementary figures and tables for datasets, experiments, and error analysis.
- Textbook Examples: The textbook examples span collegiate Chemistry, Physics, and Mathematics topics, while supplementary figures show textbook and multimodal examples.Examples include physical chemistry, quantum chemistry, differential equations, probability, calculus, and physics problems.
- Data Annotation: Seven annotators gathered textbook data with a web-based tool, preserving problem images, answers, and textbook ordering for verification.
- Datasets: SCIBENCH’s supplementary datasets include textbook-derived scientific problems and a closed exam dataset.The exam dataset contains 103 problems from seven collegiate Computer Science and Math exam sets.
- Supplementary Results: Supplementary tables report exam-dataset statistics and zero-shot total scores, while figures document examples and experimental comparisons.
- Prompts: The supplementary material specifies zero-shot, few-shot, Chain-of-Thought, Python, and Wolfram Language prompting settings.
- Evaluation Protocol: The evaluation protocol classifies model errors across ten problem-solving abilities using an LLM verifier and human review.The listed abilities include logical decomposition, assumption identification, spatial perception, and causal reasoning.
C.3. Additional Experiment on Wolfram Language
The additional Wolfram Language experiment compares external-tool prompting with Chain-of-Thought and Python and analyzes associated errors. Wolfram Language produces a notable code-conversion error rate, largely from syntax and variable-rule violations.
- Wolfram Language external-tool results and error analysis are compared with Chain-of-Thought and Python.
- 46.9% error rate occurs in code conversion skill when using Wolfram Language.Reported causes include protected symbols, disallowed underscores in variable names, and incorrect function usage.
- External tools can weaken other skills, particularly code conversion skills.
- The LLM verifier correctly classified 123 of 151 annotated samples, for an accuracy rate of 81.45%.Two human annotators assessed classification accuracy across different settings.
D.2. Examples
The examples illustrate how SCIBENCH documents problems, correct solutions, model outputs, and categorized errors across scientific reasoning tasks. They include failures involving causal reasoning, calculation, spatial perception, and other listed problem-solving abilities.
- Error Examples: Figure S9 contrasts ChatGPT solutions with and without Chain-of-Thought, assigning different errors to causal reasoning and calculation skills.
- Error Categories: The supplementary examples cover code conversion, scientific literacy, logical reasoning, abstract reasoning, assumption identification, causal reasoning, and problem deduction.
- Spatial Perception: The Hückel molecular-orbital example is classified as a spatial-perception error because the problem requires visual representations.The model solution provides equations without the visual representation needed to identify bonding and antibonding orbitals.
- Error Categories: The examples also include logical decomposition and analysis, alongside model answers and correct-solution derivations for textbook problems.