PaperlayerView exact revision on arXiv

Source-linked AI summary

SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence

Yiheng Wang, Yixin Chen, Shuo Li, Yifan Zhou, Bo Liu, Hengjian Gao, Jiakang Yuan, Jia Bu, Wanghan Xu, Yuhao Zhou, Xiangyu Zhao, Zhiwang Zhou, Fengxiang Wang, Haodong Duan, Songyang Zhang, Jun Yao, Han Deng, Yizhou Wang, Jiabei Xiao, Jiaqi Liu, Encheng Su, Yujie Liu, Weida Wang, Junchi Yao, Shenghe Zheng, Haoran Sun, Runmin Ma, Xiangchao Yan, Bo Zhang, Dongzhan Zhou, Shufei Zhang, Peng Ye, Xiaosong Wang, Shixiang Tang, Wenlong Zhang, Lei Bai

arXiv:2512.22334v3cs.AIcs.CL

TL;DR

Scientific benchmarks often measure isolated or surface-level performance rather than the breadth of scientific reasoning across disciplines and modalities. SciEvalKit brings together expert-curated benchmarks and seven capability dimensions in a unified evaluation toolkit. Its evaluations find that knowledge understanding is stronger than symbolic reasoning and code generation, with those weaknesses persisting in vision-enabled and instruction-tuned models.

  • Problem

    Existing evaluations often emphasize surface-level correctness or narrow task metrics and do not systematically assess scientific reasoning across capabilities, disciplines, and modalities.

    Source: §1 · p. 3

  • Method

    SciEvalKit combines expert-curated benchmarks across six scientific disciplines with a seven-dimension taxonomy and a unified evaluation toolkit for language and multimodal models.

    Source: §1 · pp. 3–4, §2.1 · p. 5

  • Results

    Models perform moderately to strongly in knowledge understanding, but symbolic reasoning and code generation remain underdeveloped, including in vision-enabled or instruction-tuned models.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The authors suggest future gains are less likely to come from scale alone and more from execution-aware code, symbolic and program-of-thought reasoning, and tighter visual grounding with scientific semantics.

    Source: §5 · p. 15

Abstract

from arXiv · show

We introduce SciEvalKit, a unified benchmarking toolkit designed to evaluate AI models for science across a broad range of scientific disciplines and task capabilities. Unlike general-purpose evaluation platforms, SciEvalKit focuses on the core competencies of scientific intelligence, including Scientific Multimodal Perception, Scientific Multimodal Reasoning, Scientific Multimodal Understanding, Scientific Symbolic Reasoning, Scientific Code Generation, Science Hypothesis Generation and Scientific Knowledge Understanding. It supports six major scientific domains, spanning from physics and chemistry to astronomy and materials science. SciEvalKit builds a foundation of expert-grade scientific benchmarks, curated from real-world, domain-specific datasets, ensuring that tasks reflect authentic scientific challenges. The toolkit features a flexible, extensible evaluation pipeline that enables batch evaluation across models and datasets, supports custom model and dataset integration, and provides transparent, reproducible, and comparable results. By bridging capability-based evaluation and disciplinary diversity, SciEvalKit offers a standardized yet customizable infrastructure to benchmark the next generation of scientific foundation models and intelligent agents. The toolkit is open-sourced and actively maintained to foster community-driven development and progress in AI4Science.

1. Introduction

Scientific intelligence requires more than general reasoning or narrow task scores: it spans conceptual, symbolic, hypothesis-driven, procedural, and multimodal capabilities. SciEvalKit organizes evaluation around seven such dimensions, and its model analysis finds persistent weaknesses in symbolic reasoning and code generation.

1. Introduction

  • The proposed seven-dimensional taxonomy covers scientific knowledge understanding, code generation, symbolic reasoning, hypothesis generation, multimodal perception, multimodal reasoning, and multimodal understanding.The taxonomy is grounded in structural, relational, and multirepresentational demands of scientific reasoning.Source: §1 · p. 3
  • Existing benchmarks often isolate factual, coding, or visual tasks and miss realistic scientific workflows, expert-level image semantics, and differences between scientific capability dimensions.General-purpose models also score below 60 on rigorous scientific scenarios despite the strongest model approaching 90 on general tasks.Source: §1 · pp. 3–4, Figure 2 · p. 4
  • SciEvalKit is an open-source toolkit and leaderboard evaluating LLMs and multimodal LLMs across seven dimensions, with more than 15 expert-curated benchmarks spanning six scientific disciplines.Its scoring combines direct text-answer evaluation, code execution, semantic LLM judging, and expert-aligned criteria.Source: §1 · p. 4
  • Models show substantial disparities: knowledge understanding is moderate-to-strong, while symbolic reasoning and code generation remain underdeveloped, even in vision-enabled or instruction-tuned models.The paper presents these shortcomings as evidence of gaps in readiness for real scientific problem solving.Source: §1 · p. 4

2. Benchmark Suite

SciEvalKit organizes scientific evaluation by capability and discipline, using an expert-aligned benchmark suite that combines text-only and multimodal tasks. Its first release is a curated subset spanning six fields and seven capability dimensions.

2.1. Core Competencies Taxonomy of Scientific Intelligence

  • The taxonomy distinguishes multimodal perception, understanding, and reasoning from text-only knowledge understanding, code generation, symbolic reasoning, and hypothesis generation.The dimensions respectively address entity localization, scientific visual-text interpretation, cross-modal inference, domain knowledge, executable procedures, symbolic manipulation, and open-ended explanatory synthesis.Source: §2.1 · pp. 4–5

2.2. Scientific Discipline Coverage

  • Benchmarks span life science, chemistry, earth science, physics, astronomy, and materials science, representing procedural, mechanistic, and context-dependent reasoning as well as factual knowledge.Examples include biomolecular reasoning, thermodynamics and structure–property analysis, geospatial interpretation, and physical-law modeling.Source: §2.2 · p. 6

2.3.1. Principles of Expert-Aligned Benchmark Design

  • Benchmark selection uses multi-round domain-expert consultation and criteria for scientific validity, expert calibration, capability coverage, modality and task diversity, and community recognition.Experts propose authentic research challenges; selected tasks are manually checked for formulation, rationale, and scoring correctness.Source: §2.3.1 · p. 6

2.3.2. Benchmark Overview

  • The first release evaluates a curated subset of the benchmark pool through text-only and multimodal tasks across six disciplines, selected for validity, capability coverage, and task diversity.The authors say future releases will add modalities, disciplines, and community-proposed agent tasks.Source: §2.3.2 · p. 7

3. Evaluation Framework

SciEvalKit separates data, inference, evaluation, and reporting into cooperating layers, while unified interfaces connect datasets and models. It evaluates capability-specific performance and supports different scoring modes for scientific answer formats.

3.1. Abstraction Layer

  • Four cooperating layers—Dataset, Model Inference, Evaluation & Testing, and Report & Storage—form an end-to-end pipeline with boundaries that allow components to be extended independently.The layers handle task ingestion, model outputs, capability-aligned scoring, and reproducible logging and metric storage.Source: §3.1 · pp. 7–8

3.2. Unified interface for prompt construction and prediction

  • Standardized dataset prompts represent ordered typed content segments, and a unified .generate() interface supports local and API-based models across text and multimodal tasks.Researchers can add datasets through .build_prompt() and .evaluate(), or swap models by exposing .generate(), without modifying the inference or scoring pipelines.Source: §3.2 · pp. 8–9

3.3. Capability-Oriented Evaluation

  • Capability-oriented scores average benchmark performance within each dimension, distinguishing five competencies in this section: perception, understanding, reasoning, code generation, and symbolic reasoning.The section also describes hypothesis generation and knowledge understanding as capability categories, with ResearchBench representing hypothesis generation and knowledge scores averaged across assigned benchmarks.Source: §3.3 · pp. 9–10

3.4. Evaluation Modes

  • Evaluation combines normalized rule-based matching, semantic LLM judging, and execution-based verification to accommodate heterogeneous scientific answers.Scientific code tasks can be scored by passed unit tests; other formats use strategies such as option extraction, exact or relaxed matching, and semantic metrics or judging.Source: §3.4 · pp. 10–11

4. Evaluation Results

SciEvalKit results show uneven scientific capabilities across evaluated models: knowledge understanding is relatively strong, while formal and multimodal reasoning remain more discriminative. General-purpose models lead overall, though some open-weight models are competitive on specific capabilities.

4. Evaluation Results

  • Contemporary models largely solve factual recall but struggle with executable logic and equation-level manipulation; Gemini 3 Pro and GPT-5 have the highest average, most even scores, while Qwen3-VL-235B-A22B approaches leading proprietary models.Source: §4 · p. 12
  • GPT-5.1 scores decrease across nearly every axis relative to GPT-5, and general-purpose models rather than science-tuned models achieve the highest scores.The passage characterizes the decrease as consistent with incremental alignment refinements rather than new scientific competence.Source: §4 · p. 12
  • Scientific code generation is the weakest competency across evaluated models, but Qwen3-Max consistently achieves the highest scores in that capability.Knowledge understanding clusters more tightly than code generation and symbolic reasoning, which show greater dispersion.Source: §4 · p. 14
  • Code generation shows the largest performance variance among text capabilities and often correlates positively with symbolic reasoning across models.The authors suggest both depend on formal abstraction, stepwise logical consistency, and tolerance for strict correctness constraints.Source: §4 · p. 14
  • Gemini-3-Pro has the strongest overall multimodal performance, while Qwen3-VL-235B-A22B leads perception without matching that strength in understanding or reasoning.Multimodal reasoning provides the strongest discriminative signal; perception is comparatively saturated, and strong visual grounding alone does not ensure scientific semantic integration.Source: §4 · p. 15

5. Conclusion and Discussion

SciEvalKit unifies scientific benchmarks, capability dimensions, and disciplines in one evaluation pipeline. The authors suggest that future gains may depend more on execution-aware coding, symbolic reasoning, and visual grounding than on scale alone.

5. Conclusion and Discussion

  • SciEvalKit brings high-quality benchmarks, seven capability dimensions, and six scientific disciplines together in one evaluation pipeline to expose LLM scientific capabilities.Source: §5 · p. 15
  • The authors suggest future gains are less likely to come from scale alone and more from execution-aware code, symbolic and program-of-thought reasoning, and tighter visual-grounding integration.The conclusion also lists an agent track with tool-use and verification loops, expanded multimodal tasks, and quarterly community releases as planned work.Source: §5 · p. 15

Project Contributors / Community Contributors

A.2. Full Evaluation Results Across Core Benchmarks

This section presents full quantitative results across multimodal and text-only scientific tasks, with separate tables covering benchmark evaluations.

A.2. Full Evaluation Results Across Core Benchmarks

  • Table 5 covers five scientific benchmarks, while Table 6 compares model performance across six benchmarks, including text-only tasks such as causal inference and mathematical derivation.The Table 6 description also includes knowledge retrieval and scientific QA.Source: Table 6 · p. 22, Table 5 · p. 22, §A.2 · p. 23
  • The full results cover evaluated models across both multimodal and text-only scientific tasks.Source: §A.2 · p. 23
  • Table 4 evaluates LLMs on SLAKE, SFE, and MSEarth, spanning visual reasoning on medical and scientific diagrams, code-based visualization, and entity localization.Source: Table 4 · p. 21, §A.2 · p. 23

B. Benchmark Description

The section describes benchmarks spanning scientific multimodal understanding, code generation, domain-specific knowledge and reasoning, and research-workflow simulation.

Multimodal benchmarks

  • MSEarth pairs curated Earth-science figures with refined captions for domain-informed inference, while SLAKE tests clinical-image visual question answering using annotated structures and a built knowledge graph.MSEarth questions cover five Earth-system spheres; SLAKE includes CT, X-ray, and MRI images.Source: B. Benchmark Description · p. 23
  • SFE evaluates scientific perception, understanding, and reasoning across five disciplines using 66 tasks and 830 verified visual question-answer pairs.Its tasks progress through signal perception, attribute understanding, and comparative reasoning.Source: B. Benchmark Description · p. 23

Code-generation benchmarks

  • AstroVisBench tests executable astronomy workflows and research-quality visualizations, while SciCode tests Python solutions to scientist-curated research problems [30] [19].AstroVisBench draws on 110 notebooks; SciCode covers 80 tasks across 16 subfields.Source: B. Benchmark Description · p. 23

Other scientific benchmarks

  • ClimaQA uses expert-refined questions from graduate climate-science textbooks [55], ChemBench probes chemical knowledge and discipline-based reasoning, and ResearchBench assesses paper- and review-writing workflows [81].ClimaQA has expert-curated Gold and programmatically generated Silver subsets; ResearchBench includes 1,000 paper-writing and 200 review-writing tasks.Source: B. Benchmark Description · p. 24

Domain-specific knowledge benchmarks

  • ProteinLMBench tests protein properties, descriptions, and sequence understanding across 944 six-choice questions [25], while MaScQA targets materials-science and metallurgical reasoning across 14 domains.ProteinLMBench items combine language context with amino-acid sequence spans.Source: B. Benchmark Description · p. 24

Domain-specific reasoning benchmarks

  • TRQA evaluates biomedical reasoning and literature-based inference, while CMPhysBench emphasizes condensed-matter symbolic derivation, algebraic manipulation, and physical interpretation [58].Source: B. Benchmark Description · p. 24

Disciplinary reasoning benchmarks

  • PHYSICS tests undergraduate-level reasoning across five physics areas [57], while EarthSE covers five Earth-system spheres and 114 subfields, with Earth-Silver targeting professional-level difficulty [56].Source: B. Benchmark Description · p. 24

C.1. MaScQA

A MaScQA example asks how to promote contiguous planar metallic-film growth while avoiding island growth. The supplied response selects option A and explains the choice through interfacial energy and wetting.

C.1. MaScQA

  • Option A recommends an adhesion layer with very low interfacial energy with the deposited metal film, which the response says promotes wetting and planar growth.The explanation treats the adhesion layer as the substrate for the top metal film.Source: §C.1 · p. 25
  • The response rejects annealing as a route to continuity, stating that it usually destroys thin-film continuity rather than promoting it.Source: §C.1 · p. 25
  • Gemini 3 Pro is identified as the only model that answers the MaScQA question correctly.Source: §C.1 · p. 26

C.2. Chembench

Chembench examples cover molecular-structure questions, while the accompanying SciCode and physics prompts address scientific programming and a launch-velocity problem.

C.2. Chembench

  • 174 normal vibrational modes are reported for C60 using the 3N−6 formula for a nonlinear molecule.The response identifies 60 atoms and subtracts three translational and three rotational degrees of freedom.Source: §C.2 · p. 26
  • Four 1H NMR signals are expected when exchangeable protons are visible, counting four distinct proton environments.The response treats the two methylene protons as equivalent.Source: §C.2 · pp. 26–27

C.3. SciCode

  • The SciCode prompt requests a conjugate-gradient function, and its code snippet initializes residuals and updates the solution iteratively until tolerance is met.The fragments show residual and search-direction initialization, solution and residual updates, and a loop condition based on tolerance.Source: §C.3 · pp. 27–28

C.4. PHYSICS

  • The physics prompt asks for the minimum horizontal launch velocity that keeps the stone’s contact angle at or below θ.The scenario specifies launch from height h and neglects air resistance.Source: §C.4 · p. 28

C.5. CMPhysBench

The examples span a sudden potential change, environmental and biological questions, drug-development status, and research-hypothesis generation. They include worked responses for several of these tasks.

C.5. CMPhysBench

  • A sudden switch from V1 to V2 leaves the wave function ψ0 unchanged, and the probability of measuring the new ground state is the squared overlap with ψ0.The response frames this as a sudden finite change and refers to the new potential’s energy eigenstates.Source: §C.5 · pp. 29–30

C.6. ClimaQA

  • The ClimaQA item asks why radar reflectivity increases between points 3 and 4 in a stratiform-precipitation profile.It presents four candidate explanations involving humidity, snowflake changes, and ice-crystal concentration.Source: §C.6 · p. 30

C.7. EarthSE

  • In the EarthSE example, small infrequent fires and low browsing are associated with coexistence of Abies alba and Quercus ilex, while disturbance and warmer summers reduce A. alba biomass.The response also says moderate browsing eliminates A. alba regeneration and that Q. ilex can dominate under higher disturbance.Source: §C.7 · pp. 30–31

C.8. ProteinLMBench

  • The ProteinLMBench response identifies the sequence as an alcohol:NAD+ oxidoreductase and gives primary- and secondary-alcohol oxidation reactions.Both listed reactions produce NADH and H+; the response describes broad substrate specificity.Source: §C.8 · p. 31

C.9. TRQA

  • The TRQA example reports Phase I as the highest development stage for PDE2A-targeting drugs for cognitive impairment in schizophrenia through April 2025, naming BI 474121.Source: §C.9 · p. 31

C.10. ResearchBench

  • The ResearchBench response proposes supernovae, AGB stars, and interstellar dust growth as contributors to early-universe dust and observed dust masses in early galaxies.The prompt supplies a question about dust formation above redshift 4 and a primordial Type II supernova study as its core inspiration.Source: §C.10 · p. 32

C.11. MSEarth

The MSEarth example asks which aquifer has the highest spread of the Fe–Mn hazard.

C.11. MSEarth

  • The MSEarth question asks which aquifer has the highest spread of the Fe–Mn hazard among three aquifers or none of the above.Source: §C.11 · pp. 32–33

C.12. AstroVisBench

The examples cover astronomical catalog processing and visualization, image-modality identification, and band-structure interpretation. The AstroVisBench processing example encounters an API error, while its visualization is judged functionally informative despite minor aesthetic differences.

C.12. AstroVisBench

  • The AstroVisBench processing code fails on an external API 404, while its visualization conveys the same spatial pattern as the target with minor aesthetic differences.The judgment says the visualization’s data positions and coordinate ranges are identical, and labels the differences minor.Source: §C.12 · pp. 35–36

C.13. SLAKE

  • The SLAKE example asks which modality was used to take an image.Source: §C.13 · p. 36

C.14. SFE

  • The SFE response classifies the material as metallic and says it exhibits neither a direct nor an indirect bandgap.Source: §C.14 · p. 37
Loading 2512.22334v3…