Source-linked AI summary
SciCode: A Research Coding Benchmark Curated by Scientists
Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng Du, Tianhua Tao, Ofir Press, Jamie Callan, Eliu Huerta, Hao Peng
TL;DR
As language models surpass most humans except domain experts, established evaluations increasingly struggle to remain challenging and realistic. SciCode addresses this gap with a scientist-curated coding benchmark spanning diverse natural science fields, and finds that even the best-performing tested model solves only 4.6% of main problems in the most realistic setup.
Problem
Established benchmarks increasingly struggle to keep pace with language-model performance, making challenging and realistic evaluation more difficult.
Method
SciCode is a scientist-curated benchmark of 80 natural-science coding problems decomposed into 338 subproblems, with scientific background, gold-standard solutions, and test cases for evaluation.
Results
4.6% of main problems are solved by Claude3.5-Sonnet, the best-performing tested model, in the most realistic evaluation setup.
Takeaways & Limitations
SciCode provides a guideline for building future code language models for varied scientific applications while showing that the benchmark remains very challenging for current models.
Takeaways & Limitations
Many problems were simplified and given additional background during revisions because most were too challenging even for the best models.
Abstract
from arXiv · showhide
Since language models (LMs) now outperform average humans on many challenging tasks, it has become increasingly difficult to develop challenging, high-quality, and realistic evaluations. We address this issue by examining LMs' capabilities to generate code for solving real scientific research problems. Incorporating input from scientists and AI researchers in 16 diverse natural science sub-fields, including mathematics, physics, chemistry, biology, and materials science, we created a scientist-curated coding benchmark, SciCode. The problems in SciCode naturally factorize into multiple subproblems, each involving knowledge recall, reasoning, and code synthesis. In total, SciCode contains 338 subproblems decomposed from 80 challenging main problems. It offers optional descriptions specifying useful scientific background information and scientist-annotated gold-standard solutions and test cases for evaluation. Claude3.5-Sonnet, the best-performing model among those tested, can solve only 4.6% of the problems in the most realistic setting. We believe that SciCode demonstrates both contemporary LMs' progress towards becoming helpful scientific assistants and sheds light on the development and evaluation of scientific AI in the future.
1 Introduction
SciCode addresses increasingly saturated LM evaluations with a scientist-curated benchmark of realistic scientific code-generation problems. Its 80 main problems and 338 subproblems remain highly challenging for current models, even when scientific background is provided.
- Benchmark proposal: SciCode contains 80 main problems decomposed into 338 subproblems across diverse natural science fields.The benchmark covers mathematics, physics, chemistry, biology, materials science, and other natural science areas.
- Benchmark proposal: Problems include scientific background when necessary, detailed instructions, gold-standard solutions, and multiple test cases for automatic evaluation.Models implement Python functions for subproblems and integrate them into complete main-problem solutions.
- Design choices: SciCode emphasizes high-quality, realistic, and uncontaminated problems annotated, revised, and verified by at least two senior researchers.Problems are sourced from scientists’ everyday research tasks or influential papers and are designed to avoid overlap with publicly available data.
- Implications: SciCode is intended to guide future AI methods for accelerating scientific research, an area the authors describe as less advanced partly because of limited commercial incentive.The benchmark is presented as a resource for developing scientific AI systems.
- Evaluation results: 4.6% of main problems were solved by Claude3.5-Sonnet in the most realistic setup, compared with 1.5% for Claude3-Opus and GPT-4o.Deepseek-Coder-v2 solved 3.1%, while several other open-source models solved no complete main problems.
- Evaluation results: 12.3% of main problems were solved by the best model when scientist-provided background knowledge was available.The authors report substantial and consistent improvements from background knowledge, but performance remained limited.
2 SciCode
SciCode is built from realistic research coding tasks across 16 natural science subfields and structures each complex task as coordinated subproblems. Its scientist-led annotation, validation, and evaluation design supports granular testing, yet the benchmark remains difficult for current models.
- 2.1 Challenging and Realistic Scientific Coding Problems: SciCode covers 16 natural science subfields with research-level coding problems drawn from scientists’ workflows and published methods.The benchmark spans mathematics, physics, chemistry, biology, materials science, and related areas.
- 2.1 Challenging and Realistic Scientific Coding Problems: The dataset contains 80 main problems, with a median of 3 subproblems per main problem and a maximum of 15.Fifteen main problems are reserved for development and 65 main problems for testing.
- 2.2 A Main Problem with Multiple Subproblems: Each main problem defines an overall research objective, while subproblems divide it into smaller parts with docstrings specifying inputs and expected outputs.This decomposition enables more systematic problem solving and granular evaluation.
- 2.1 Challenging and Realistic Scientific Coding Problems: SciCode problems are selected from numerical methods, system simulations, scientific calculations, and methods used in influential or Nobel Prize-winning studies.These categories target coding tasks requiring substantial scientific knowledge and reasoning.
- 2.3 Evaluation Design: Evaluation combines numerical input-output tests with domain-specific cases drawn from real scientific workflows.The design aims to preserve scientific fidelity while keeping evaluation practical and solutions accessible.
- 2.3 Problem Validation: Problems undergo in-domain scientist review, out-of-domain clarity review, and GPT-4-assisted validation with scientist error analysis.Scientists revise workflows and test cases to address ambiguity and prevent false positives.
3 Experiments
SciCode evaluates language models on multi-step scientific coding tasks under standard and augmented settings. Models perform substantially better on isolated subproblems than on complete main problems, while background knowledge helps but does not eliminate the challenge.
- 3.2 Main Results: Under the standard setup, Claude3.5-Sonnet achieves the best main-problem performance at 4.6% pass@1.The standard setup excludes background knowledge and carries generated solutions across previous subproblems.
- 3.2 Main Results: 14%-26% pass@1 rates on subproblems contrast with much lower performance on the more realistic main-problem evaluation.Subproblem evaluation is easier because it does not require completing the full integrated solution.
- 3.2 Main Results: Open-weight models also fail to complete main problems despite solving some subproblems correctly.This pattern is reported alongside the broader finding that SciCode is difficult for current language models.
- 3.3 Additional Results with Other Evaluation Settings: With scientific background knowledge, all models substantially improve on both subproblem and main-problem evaluations.Background-conditioned evaluation also uses generated solutions for previous subproblems, and its deltas compare against the standard setup.
- 3.3 Additional Results with Other Evaluation Settings: With background knowledge, Claude3.5-Sonnet reaches 35.4% pass@1 on subproblems and 12.3% on main problems.GPT-4o also reaches 35.4% on subproblems, while GPT-4o and GPT-4-Turbo reach 9.2% on main problems.
- 3.3 Additional Results with Other Evaluation Settings: Performance generally improves as models condition on more gold solutions from previous steps, but declines after conditioning on more than 9 previous solutions.The analysis provides gold solutions for earlier subproblems without scientific background knowledge.
4 Related Work
Related work spans code language models, execution-based code-generation evaluation, and scientific language models. Existing benchmarks cover basic programming and increasingly realistic software tasks, while scientific models address complex scientific data and reasoning tasks.
- Language models for code: Code language models evolved from CodeBERT and CodeT5 through Codex and later specialized and generalist code-generation models.The cited work describes rapid growth in both the quantity and quality of code-generation models.
- Evaluating code generation: Execution-based evaluation became more prominent as models improved at producing executable code.SciCode omits n-gram and general text-based metrics because surface-form matching is limited for scientific coding.
- Evaluating code generation: Code-generation benchmarks range from simple function completion to data science, repository-level completion, and software-engineering tasks.MBPP and HumanEval primarily test translating natural-language instructions into Python programs.
- Language models for science: Scientific language models target complex tasks involving scientific text and data modalities such as molecules and proteins.Galactica is cited as a general-purpose scientific model supporting citation prediction, scientific reasoning, document generation, and molecular property prediction.
5 Conclusion
SciCode is a scientist-curated benchmark spanning diverse scientific domains and evaluated across contemporary AI models. The evaluation shows that it remains very challenging while supporting future scientific code-model development.
- SciCode is a scientific research benchmark curated by professional natural scientists.
- The benchmark represents 16 diverse scientific domains.
- Evaluations with ten contemporary state-of-the-art AI models show that SciCode remains very challenging.
- The authors present SciCode as a guideline for building future code language models for varied scientific applications.
A.1 Prompt
The appendix presents two prompt variants for SciCode evaluation: one without background information and one with scientists’ background information.
- Table 4 presents the prompt without background information.
- The appendix distinguishes prompt conditions according to whether scientists’ background information is provided.
- Table 5 presents the prompt with scientists’ background information.
A.2 Python libraries used in SciCode.
Figure 4 identifies the Python libraries used by problems in SciCode.
- Figure 4 concerns Python libraries used by SciCode problems.
- The figure summarizes library usage across problems in SciCode.
- Python library use is presented as a property of SciCode’s problems.
A.3.1 Example Main Problem
The example main problem asks for Chern numbers in the Haldane model on a hexagonal lattice, while Figure 5 presents its dependencies.
- The example asks for an array of Chern numbers for the Haldane model on a hexagonal lattice.
- The task sweeps the on-site energy-to-coupling ratio and phase values.
- Figure 5 presents the example general problem and its dependencies.
A.3.2 Example Subproblems
The example subproblems implement a Haldane-model workflow: construct the Hamiltonian, calculate Chern numbers, and sweep parameters to produce a phase diagram. The examples also document coarse-grid accuracy and verify expected parameter-independence and phase behavior.
- The first subproblem asks for a Haldane model Hamiltonian on a hexagonal lattice from wavevector, lattice, coupling, phase, and onsite-energy parameters.
- The second subproblem calculates the Chern number by discretizing the Brillouin zone using specified grid, lattice, coupling, phase, and onsite-energy parameters.
- The third subproblem sweeps m/t2 from -6 to 6 and phase from -π to π to create a two-dimensional Chern-number array.
- Rough k-space and parameter-sweep grids accelerate computation, while increasing grid sizes improves accuracy and produces closer agreement with the original Haldane phase diagram.
- At zero onsite energy, the Chern number is 1 for ϕ > 0 and -1 for ϕ < 0, while the phase diagram is independent of t1 and the ratio t2/t1.